There are children not receiving a single dose of any vaccine: from ‘data to policy’ in immunisation and health systems. Data quality and socio-economic determinants of unvaccination in low- and middle-income countries - Chapter 4: There is no gold standa

(1)

There are children not receiving a single dose of any vaccine: from ‘data to

policy’ in immunisation and health systems. Data quality and socio-economic

determinants of unvaccination in low- and middle-income countries

Bosch-Capblanch, X.

Publication date

2012

Link to publication

Citation for published version (APA):

Bosch-Capblanch, X. (2012). There are children not receiving a single dose of any vaccine:

from ‘data to policy’ in immunisation and health systems. Data quality and socio-economic

determinants of unvaccination in low- and middle-income countries. Rozenberg Publishers.

General rights

It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).

Disclaimer/Complaints regulations

If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.

(2)

Chapter 4. There is no gold standard

to consolidate discrepancies in

vaccination coverage figures

between data sources: are we

globalising or trivialising access to

data?

Xavier Bosch‐Capblanch. Status: being submitted.

(3)

Title

There is no universal ‘gold standard’ to consolidate discrepancies between data sources in vaccination coverage figures: are we globalising or trivialising access to data?

Xavier Bosch‐Capblanch1,2*, David W Brown3, Kaushik Banerjee4, Abdallah Bchir5, Tony Burton4 1 Swiss Tropical and Public Health Institute, Basel. 2 University of Basel, Basel. 3 United Nations Children Fund, New York. 4 Immunisation, Vaccines and Biologicals, World Health Organisation, Geneva. 5 The GAVI Alliance, Geneva. * E‐mail: x.bosch@unibas.ch

Abstract

Background Several data sources are used to estimate vaccination coverage: administrative data from countries, surveys and estimates based on the former two; but there are discrepancies in estimates between sources for the same country and year. The debate on the quality of data is based on the assumption that discrepancies should not exist and that surveys are the “gold standard”. We challenge these assumptions: the objective of this study is to describe the causes and magnitude of discrepancies in the third dose of diphtheria‐tetanus‐pertussis (DTP) coverage between different data sources. Methods Discrepancies were analysed using Bland & Altman limits‐of‐agreement (LOA) between two measures, including 228 country‐years which had data both from administrative and survey sources. Findings Recording pathways revealed that parameters to estimate coverage (e.g. children’s age, geographical scope, or denominator source s) are different in administrative and surveys estimates. Discrepancies showed a relative over‐estimate of administrative coverage when compared with two types of generic health surveys (LOAs < 0), and under‐estimates when compared with vaccination surveys (LOA > 0). Comparing surveys estimates in the 10 countries having had two surveys in the same year, differences in ranged from 0.005 up to 0.599. Interpretation Discrepancies due to ‘design’ issues have to be expected even if data accuracy could be perfect. What is an acceptable discrepancy, then? The general assumption that surveys are the ‘gold standard’ to adjust for typically flawed administrative coverage is not supported by the examples of good quality administrative data, data issues in surveys and discrepancies between different types of surveys. Cases with large discrepancies have to be individually

(4)

interpreted and reconciled on the light of additional local and global knowledge. In an era of increasingly global accessibility to data, careful judgement needs to take place in order to make it relevant for decision‐making.

Introduction

Each year since 2000, the United Nations Children's Fund (UNICEF) and the World Health Organization (WHO) have jointly reviewed, prepared and published estimates of national immunization coverage for selected vaccine preventable diseases (Joint Reporting Form process (JRF)) [1]. The main sources of empirical data on immunization coverage used in this process are administrative data based on reports from service providers (e.g. health centres staff, vaccination teams, private physicians) and surveys with items on children’s vaccination status. Administrative data report the number of vaccinations given during a certain period of time – usually one month – and recorded at service delivery points. These data are aggregated at district or regional levels and forwarded to the national level of the health system [2] where annual estimates are produced. Countries are also producing their own ‘official’ vaccination coverage estimates which may differ from the ‘administrative’ data because the latter is pitched by national authorities using additional information, such as reporting delays in the country or data from surveys. The ‘country official estimates’ are reported, alongside administrative figures, to WHO and UNICEF through the JRF. Vaccination data from the JRF process, surveys and other information, including local knowledge that may explain data changes (e.g. a new census), are then used to produce the joint WHO‐UNICEF estimates [2,3]. Household and community‐based surveys are common sources of immunization coverage data. In these surveys, immunization status is determined either by looking at immunization records (i.e. immunization cards or child health cards) maintained in the home, asking the child’s caregiver (i.e. caregiver recall), or both. The three main household survey sources are the Demographic and Health Survey (DHS) [4] , the UNICEF‐sponsored Multiple Indicators Cluster Survey (MICS) [5] and the Expanded Programme on Immunization (EPI) cluster survey [6]. In addition to its capabilities as an assessment and monitoring tool at the individual health service unit level, lot quality assurance sampling (e.g. LQAS[7]) can be used to assess coverage in small population areas. Both administrative and survey data have advantages and disadvantages. Administrative coverage data provide more timely information than surveys and are useful when surveys may not be feasible to do. These data are also central to programme management. Because of the more timely nature of the information, administrative data may reveal service delivery problems (e.g. vaccine shortage, poor session attendance) earlier on. However, coverage estimates based on administrative data are subject to numerator (number of children vaccinated) and denominator (number of children in the target population) biases [8]. In

(5)

contrast, survey data allow for estimating immunization coverage even if the size of the target population is unknown. However, survey methods and implementation can be also sources of bias when estimating coverage, although they are more difficult to study (e.g. inaccuracies and incompleteness of household health vaccination cards [9]). There has been much debate in recent years about the quality of vaccination coverage data to monitor immunisation programme performance and to inform decisions, especially in relation to performance based funding schemes. The debate has focused on the observed discrepancies between different data sources (namely, administrative and surveys) and approaches to describe and deal with those discrepancies have been proposed in the past [10,11]. Most of this debate is built on a series of assumptions; namely: (i) that discrepancies between different data sources should not exist or be minimal; (ii) that administrative data is generally of poor quality while surveys are the “gold standard”; (iii) that highly accurate estimates of coverage are always desirable to monitor performance; and (iv) that coverage from a single antigen (usually the third dose of diphtheria‐tetanus‐pertussis vaccine (DTP3)) is sufficient to monitor immunisation programme performance. Using publicly available administrative and survey data, we challenge the first two assumptions and critically review the last two. The objectives of this study are: 1. to describe the potential causes of discrepancies between data sources; 2. to analyse discrepancies between different data sources; 3. to propose a way forward for a more rational use of global vaccination data.

Methods

In order to describe the potential causes of discrepancies between data sources and establish whether discrepancies should be expected or not, we developed an algorithm that systematically describes the processes involved in producing, recording, aggregating and transmitting immunization coverage data, including both administrative sources and surveys, and that identifies possible errors for each data source. In particular, the issues addressed include: data flow, inclusion criteria of infants and children, time periods and geographical scope of vaccination coverage estimates. We used DTP3 coverage levels as the unit of analysis for discrepancies in country‐year dyads. The sources of DTP3 coverage data used in the study included administrative coverage, provided by WHO; countries official estimates and WHO‐UNICEF estimates [12]; and surveys: DHS [ 4], MICS [5] and EPI, the latter provided by WHO. Available data points for each type of source are shown in Table 1. The main comparisons where administrative sources versus country official estimates, versus WHO‐UNICEF estimates and versus surveys. In each pairwise comparison of different data sources, only those country‐year dyads with data from both sources could be included. Therefore, not all comparisons necessarily had the same country‐year dyads. Comparisons

(6)

that included surveys could only be described for countries where surveys had taken place, all of them being low‐ and middle income countries (LMIC). DTP3 coverage is calculated by dividing the total number of children with some evidence of having received DTP3 (either by card or caregiver’s recall) by the total number of children who reached their first birthday (less than 12 months of age). Throughout this study, coverage is expressed as a proportion, ranging from 0 to 1. Surveys included DHS, MICS and EPI. DTP3 coverage from surveys was reanalysed where microdata was available (microdata was not available for some DHS, MICS1, MICS4 and EPI surveys). Calculations of coverage were undertaken after harmonising vaccination variables across surveys [13] and took into account the multi‐stage cluster sampling design of these surveys. In surveys, vaccination status for each vaccine is documented either by the date of vaccination in the child vaccination or health card, by having a mark on the card (a code, usually ‘44’, is recorded in the data set), or by the caregiver recall when the child health card was not available. Missing, zero or higher than 1 (or 100%) DTP3 coverage values were excluded. As it is standard practice, children included in the analyses of surveys were the cohorts of children 12 to 23 months old at the time of interview, since those children would have had the opportunity to receive all routine immunisations scheduled for the first year of life, particularly DPT3. Because these children were actually vaccinated sometime in the twelve months prior to the surveys, surveys were compared with the administrative data from the previous calendar year (e.g. administrative data for Madagascar in the year 2003 was compared with the survey carried out in Madagascar in 2004). Discrepancies in coverage levels between data sources were described using Bland & Altman limits‐of‐agreement [14] between two measure methods, using the ‘concord’ Stata command, and were expressed as fractions of 1. Discrepancies were graphically represented showing paired differences against pair‐wise means. The concordance correlation coefficient was also estimated. Where DHS or MICS surveys were included in the analyses of discrepancies, sub‐ groups where done by phase of the surveys, as defined by the sponsors. All analyses were conducted using Stata 10.0 I/C[15].

Results

Are discrepancies between administrative sources and surveys expected? Figure 1 shows the different vaccination status of children, the pathways to record it and the possible recording errors. In the left side, the non‐vaccinated children are represented and in the right side the (correctly or incorrectly) vaccinated ones. If a child has not been vaccinated, his/her vaccination status can only be assessed in a survey which would retrieve the health card of the child. The vaccination card could erroneously show that the child received a correct or an incorrect dose (e.g. a dose beyond the age limits); or could correctly have a blank space for that vaccine. In that case, the caregiver would be asked and he/she could give

(7)

a correct or wrong answer, or no answer, as shown in the figure. If a child would have been vaccinated, this could similarly be captured by a survey, looking at the health card or asking the caregiver to recall. A vaccinated child would also have his/her vaccination status recorded in the vaccination delivery site, as not recorded (missing), as a correct dose or as an incorrect dose. Similarly, for the rest of the paths in the figure. Each one of these possibilities would produce different errors in the numerator and denominator of the vaccination coverage calculations. For example, a child incorrectly classified as vaccinated would increase the numerator; or a child with unknown vaccination status would reduce the denominator. Furthermore, the sources for the denominators are different in administrative data than in surveys: in the former, denominators are estimates of the proportion of under‐1s in the whole population; in the latter, are those under‐1s with known vaccination status included in the survey. Table 2 shows the differences in the parameters involved in calculating vaccination coverage between administrative sources and surveys. The table clearly suggests that there is no single parameter identical in administrative data and surveys and, therefore, discrepancies are unavoidable and expected even in the ideal scenario of perfectly accurate data. Errors in all these parameters, at reporting, recording or analyses levels, can occur (e.g. errors in estimating the age of children); which can be present in both administrative and survey sources. Therefore, there are two sources of discrepancies: the differences in parameters used in estimating coverage (discrepancies due to the ‘designs’) and the unavoidable errors that occur in any data collection and processing mechanism. Discrepancies in DTP3 vaccination coverage between sources A total of 88, 64 and 49 country‐year dyads were available for comparing administrative with DHS, MICS and EPI surveys, respectively. The availability of data for other comparison is detailed in Table 3. The LOA between administrative and other sources of vaccination coverage ranged from ‐ 0.099 (almost 10%) for DHS (i.e. DHS coverage was smaller than administrative coverage), to 0.033 in EPI surveys (EPI coverage was greater that administrative coverage). Discrepancies were less important when comparing administrative figures with country official estimates and WHO/UNICEF estimates, probably because the two latter estimates are the result of gauging administrative data using surveys (0.004 and ‐0.008, respectively). See Figure 2. Country official estimates showed a similar pattern, with maximum absolute differences when comparing with DHS (‐0.058) MICS (‐0.080) and EPI surveys (0.038). Discrepancies were less marked comparing country official estimates with WHO/UNICEF estimates (‐0.009), again probably because both adjust administrative estimates using survey data. WHO/UNICEF estimates were consistently lower than surveys (LOA below zero), except when comparing with the first phase of DHS (0.067) and with EPI surveys (0.020).

(8)

The correlation coefficient was generally high in comparisons which did not include surveys (e.g. administrative versus country official estimates, country official versus WHO/UNICEF estimates) with values ranging from 0.86 to 0.95, as would be expected given the fact that these are not independent measures; this is clearly visualised in Figure 3. It is worth noting that in all comparisons there is a great variability in LOA across country‐ dyads, and for the same country along several years, with cases showing differences much larger and much smaller than zero, in every comparison. Figure 4 illustrates one case of very different DTP3 coverage between administrative sources and three types of surveys (DHS, MICS and EPI surveys) along the years for a single country. The few cases where there were two surveys for the same country‐year dyad provided a unique opportunity to check the agreement between different surveys. These cases are shown in Table 4. Differences between survey sources ranged from 0.005 (Georgia1998 and Guinea 1991, DHS versus MICS and MICS versus EPI, respectively) up to 0.2 (Burkina Faso 2002) and 0.53 (Pakistan‐1990), both comparison being DHS versus EPI.

Discussion

We have compared DTP3 coverage from different sources across country‐years dyads. The preceding analyses suggest that discrepancies between data sources, particularly between administrative and survey data, should normally be expected due to the differences in all parameters used to calculate coverage with these methods. These differences are unavoidable and attributed to the different design of the methods to collect and analyse data. The question is then, to which extent observed discrepancies between administrative and survey data can be explained solely on the grounds of different designs? The dispersion of data points and differences do not seem to support the hypothesis of being only the designs what explains those discrepancies. There were cases of large differences in all comparisons and numerous ones with moderate discrepancies, in both directions. When administrative data is considered, discrepancies with surveys tend to be attributed to poor quality of administrative data. Some errors include, counting errors, errors or changes in the censuses or estimates of children living in a certain area or country, or missing and delayed reports [16]. Administrative coverage can be underestimated when vaccinations are not reported by lower administrative levels or part of the population either due to delays in reporting, absent reporting or lacking information on sub‐populations such as those served by the private sector, and therefore excluded from the data collection or reporting system (i.e. numerator smaller than it should be). Administrative coverage can also be overestimated when children vaccinated outside the target age group are erroneously included in the numerator. Estimates based on administrative data can also be biased by an inaccurate denominator, especially when outdated censuses and poor population projections are used [8]. Despite these issues, the quality of administrative data varies greatly from country to country, from year to year, and examples of good data quality, looking at the consistency of

(9)

the aggregated data thought different levels and at information system parameters, do exist as well [16]. On the other hand, surveys are not free from methodological and data management challenges. Some examples of issues encountered in these analyses include: the existence of duplicate vaccination variables, variables referring to data from health facilities and others referring to data retrieved at household level (in some DHS); the lack of properly labelled vaccination variables (e.g. in Kyrgyzstan‐2005); the inconsistent use of the labels (e.g. ‘DTP3’, ‘DTP3 or Hepatitis B’ and ‘DPT and Hepatitis B’); inconsistencies in the assessment of the vaccination status where caregiver recall[17,18] can underestimate[19,20] or overestimate[21] the vaccination status of children (and the factors associated with recall bias may differ from those favouring card retention[22]); or problems related to the field conditions when implementing the surveys, which remain undetected at the time of the analyses [9]. Both administrative and survey methods are subject to recording, computation and transcription errors as well as non‐compliance with established protocols due to poor training and supervision. Systematic and purposeful data fabrication is also possible challenges [2]. Furthermore, despite that there were great variability in the LOA between sources, there seemed to be a trend in the sense that DHS and MICS tended to underestimate coverage when compared with administrative data, while EPI surveys tended to overestimate it, suggesting the existence of ‘survey‐specific’ factors influencing the magnitude and direction of discrepancies. It was striking that large discrepancies between surveys could be observed in country‐dyads having more than two surveys. Issues that may explain why surveys provide different estimates for the same country‐year dyad could include: the wording of the questions, the coding of answers, algorithms for calculating results or the placement of the questions in the tool to enquire households. Therefore, when comparing administrative with surveys data, attributing the role of ‘gold standard’ to surveys is not always justified. If designs by themselves can only explain some of the discrepancy trends, then a case by case judgment is required to reconcile vaccination coverage data, which may depend on each country‐year dyad. What degree of discrepancies is acceptable, then? and how to deal with unacceptable degrees of discrepancies? We believe that the acceptable degree of discrepancies for a given country has to depend on (i) the coverage levels; (ii) past trends on coverage; and (iii) the decisional space of those using coverage data. For example, counties with low coverage (e.g. around 50%) do not probably need a lot of accuracy for considering taking action to improve programmes performance; or countries with larger but still suboptimal coverage may need much more accurate data from specific geographical areas only or to monitor progress. Decision‐makers of performance based funding schemes should consider those issues and inform their judgements using coverage data [23] but also local knowledge to interpret the discrepancies between data and to assess factors that may affect their accuracy, including

(10)

statistical precision of estimates, recording and reporting issues. Unless documented data flaws exist, or sources of bias are established, it does not seem reasonable to systematically give preference to one type of data source over another. Furthermore, a single indicator may be useful for some purposes (for example, DTP3 to monitor global trends) but not for others (e.g. equity, especially in countries where increases in DTP3 coverage may be achieved at the expense of reducing fully immunised children [24]). At country level, decisions cannot be taken on the grounds of a single coverage rate[25], but rather looking at other indicators and qualitative issues. Generic approaches of data reconciliation based on a single indicator do not seem appropriate for programme management or when critical decisions are to be derived from these data. It could even be argued that reporting discrepant figures is more informative than a single consolidated one. Good progress has been made in making data globally accessible [26]. One approach used to solve discrepancies, is to combine data from different sources using statistical or mathematical models [10, 11]. The advantages of this approach are that it can be applied systematically to any country‐year dyad, that they are replicable, assumptions are explicit and that can provide estimates of effective vaccine coverage [27]. However, this goes against the fact that discrepancies are not homogeneous in magnitude and direction across all countries and years; and secondly, statistical precision of reconciled figures may be too low [11] to make them appropriate for decision‐making at country and global levels. The approach of UNICEF/WHO is to use the JRF data (administrative and surveys) to obtain reconciled figures. The algorithm to produce those estimates has been criticised for its lack of transparency and subjectivity, both of which have been recently addressed [2]. We call for: ○ critically researching the assumption that the quality of administrative data is poor and establishing guidance on the accuracy needed for each type of decisions; ○ the international community and donors to rationalise and improve the fairness of performance‐based funding schemes by making transparent and evidence informed judgments which take into account global and local knowledge on programmatic and data quality issues; ○ rationalising and standardising the use of the increasingly available volume of data to make it suitable for decision‐making at country and global levels.

Acknowledgments

Funding sources

These analyses have been carried out in the context of several vaccination related projects and grants funded by the World Health Organisation and the GAVI Alliance

(11)

References

1 WHO. Immunization surveillance, assessment and monitoring. WHO/UNICEF Joint Reporting Process. http://www.who.int/immunization_monitoring/routine/joint_reporting/en/index.html. 2 Burton A, Monasch R, Lautenbach B, Gacic‐Dobo M, Neill M, Karimov R, Wolfson L, Jones G, Birmingham M. WHO and UNICEF estimates of national infant immunization coverage: methods and processes. Bulletin of the World Health Organization 2009; 87:535‐541. doi: 10.2471/BLT.08.053819. 3 WHO. Immunization surveillance, assessment and monitoring. Immunization coverage. WHO/UNICEF estimates of national immunization coverage. http://www.who.int/immunization_monitoring/routine/immunization_coverage/en/index4.html. 4 MEASURE DHS. Demographic and Health surveys. http://www.measuredhs.com/data/available‐ datasets.cfm. 5 UNICEF. Statistics and monitoring. http://www.unicef.org/statistics/index_24302.html. 6 WHO. Immunisation coverage cluster survey – Reference manual. WHO, Geneva 2005. http://www.who.int/vaccines‐documents/DocsPDF05/www767.pdf. 7 Valadez JJ, Brown LD, Vargas WV, Morley D. Using lot quality assurance sampling to assess measurements for growth monitoring in a developing country's primary health care system. Int J Epidemiol 1996; 25(2): 381‐7. 8 Brown DW, Feeney G, Burton AH. Raising Awareness Among Immunization Programme Managers to the Potential Bias Resulting from the Application of Fixed Factors to Obtain Target Population Size Estimates. Open Public Health Journal 2012; 5, 15‐18. Available online at http://www.benthamscience.com/open/tophj/articles/ V005/15TOPHJ.pdf 9 Ramakrishnan, R.; Venkata, Rao T.; Sundaramoorthy, L., and Joshua, Vasna. Magnitude of recall bias in the estimation of immunization coverage and its determinants. Indian Pediatrics 1999; 36: 881‐885. 10 Murray CJ, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet 2003; 362(9389): 1022‐7. 11 Lim SS, Stein DB, Charrow A, Murray CJ. Tracking progress towards universal childhood immunisation and the impact of global initiatives: a systematic analysis of three‐dose diphtheria, tetanus, and pertussis immunisation coverage. Lancet 2008; 372(9655): 2031‐46. 12 WHO. Immunization surveillance, assessment and monitoring. Data, statistics and graphics by subject.http://www.who.int/immunization_monitoring/data/data_subject/en/index.html. 13 Bosch‐Capblanch X. Harmonisation of variables names prior to conducting statistical analyses with multiple datasets: an automated approach. BMC Med Inform Decis Mak. 2011 May 19;11:33. 14 Bland JM, Altman DG 1986. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet I: 307‐310. 15 StataCorp (2007). Stata statistical software: release 10. College Station, TX: StataCorp LP. 16 Bosch‐Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy and quality of immunization information systems in forty‐one low income countries. Trop Med Int Health 2009; 14(1):2‐10. 17 Ramakrishnan R, Venkata Rao T, Sundaramoorthy L, Joshua V. Magnitude of recall bias in the estimation of immunization coverage and its determinants. Indian Pediatr 1999; 36: 881–85. 18 Gareaballah ET, Loevinsohn BP. The accuracy of mother’s reports about their children’s vaccination status. Bull World Health Organ 1989; 67: 669–74. 19 Langsten R, Hill K. The accuracy of mother’s reports of child vaccination: evidence from rural Egypt. Soc Sci Med 1998; 46:1205–12. 20 Valadez JJ, Weld LH. Maternal recall error of child vaccination status in a developing nation. Am J Public Health 1992; 82: 120–22. 21 George K, Victor S, Abel R. Reliability of mother as an informant with regard to immunisation. Indian J Pediatr 1990; 57: 588–90.

(12)

22 Suarez L, Simpson DM, Smith DR. Errors and correlates in parental recall of child immunizations: effects on vaccination coverage estimates. Pediatrics 1997; 99: 1–5.

23 Chee G, Hsi N, Carlson K, Chankova S, Taylor P. Evaluation of the First Five Years of GAVI Immunization Services Support Funding. Abt Associates Inc 2007.

24 Bosch-Capblanch X, Banerjee K, Burton A. Unvaccinated children in years of increasing coverage:

how many and who are they? Evidence from 96 low‐ and middle‐income countries. Trop Med and Int

Health 2012; 17: 697–710. 25 WHO. Vaccine introduction guidelines. Adding a vaccine to a national immunisation programme: decision and implementation. WHO – Immunization, Vaccines and Biologicals 2005. WHO/IVB/05.18. 26 Chandramohan D, Shibuya K, P Setel P et al., Should data from demographic surveillance systems be made more widely available to researchers? PLoS Med 5 (2008), p. e57. 27 Lessler J, Metcalf CJE, Grais RF, Luquero FJ, Cummings DAT, et al. (2011) Measuring the Performance of Vaccination Programs Using Cross‐Sectional Surveys: A Likelihood Framework and Retrospective Analysis. PLoS Med 8(10): e1001110. doi:10.1371/journal.pmed.1001110.

(13)

Tables and Figures

Table 1. Availability of DTP3 vaccination data by data source. Data source Earliest Year Most recent year Countries Country‐year dyads Administrative 1980 2006 189 1,668 Official estimates 1980 2010 193 5,972 WHO/UNICEF estimates 1980 2010 193 5,972 DHS 1984 2010 83 222 MICS 1991 2010 89 75 EPI 1979 2011 76 242

Table 2. Differences in vaccination coverage estimated by data sources.

Administrative sources Survey Children Older than 14 weeks and younger than 24 months in that calendar year At least 12 months of age and less than 24 months of age at the time of the survey Numerator Recorded just after administration of the vaccine in children attending vaccination sessions Drawn from a health card or recalled by caregiver for children in the survey Denominator Estimated proportion of under‐1s from the whole catchment population Children with known vaccination status included in the survey

Time period Calendar year Point estimate

Geographical area Country Sampled households in included clusters Data flow Aggregation towards central level Specific survey data management and cleaning

(14)

47

Table 3. Level of agreement and correlation between different data sources for DTP3 coverage.

Sources Years Level of agreement Correlation

A B Subgroup Start End Countries Dyads Diff. LL UL rho LL UL

Administrative Country official All 1980 2007 178 1583 0.004 -0.111 0.119 0.95 0.94 0.95 Administrative WHO/UNICEF All 1980 2007 178 1579 -0.008 -0.195 0.178 0.86 0.85 0.87 Administrative DHS All 1985 2007 56 88 -0.099 -0.402 0.204 0.49 0.34 0.61

Administrative DHS Version 1 1985 1985 1 1 Not estimable

Administrative DHS Version 2 1990 1991 2 2 Not estimable

Administrative DHS Version 3 1994 1997 6 6 -0.154 -0.401 0.092 0.41 -0.15 0.77

Administrative DHS Version 4 1999 2004 37 41 -0.076 -0.373 0.221 0.62 0.40 0.77

Administrative DHS Version 5 2003 2007 37 38 -0.113 -0.427 0.200 0.33 0.12 0.51

Administrative MICS All 1994 2007 64 91 -0.078 -0.446 0.290 0.64 0.51 0.74

Administrative MICS Version 1 1994 1995 4 4 -0.100 -0.669 0.469 0.17 -0.74 0.86

Administrative MICS Version 2 1999 2000 49 49 -0.049 -0.455 0.358 0.66 0.47 0.79

Administrative MICS Version 3 2004 2007 38 38 -0.113 -0.395 0.170 0.62 0.43 0.76

Administrative EPI All 1987 2007 37 49 0.033 -0.214 0.280 0.80 0.68 0.88 Country official WHO/UNICEF All 1980 2010 193 5021 -0.009 -0.177 0.160 0.93 0.93 0.94 Country official DHS All 1984 2010 81 212 -0.058 -0.388 0.272 0.63 0.54 0.70

Country official DHS Version 1 1984 1989 24 24 0.127 -0.300 0.554 0.37 0.04 0.63

Country official DHS Version 2 1989 1992 24 24 -0.120 -0.443 0.203 0.55 0.26 0.75

Country official MICS All 1991 2010 82 159 -0.080 -0.436 0.277 0.64 0.54 0.72

Country official MICS Version 1 1991 1997 43 49 -0.058 -0.401 0.285 0.64 0.45 0.78

Country official MICS Version 4 2009 2010 11 11 -0.136 -0.511 0.239 0.22 -0.10 0.49

Country official EPI All 1980 2010 73 202 0.038 -0.243 0.320 0.73 0.67 0.79

WHO/UNICEF DHS All 1984 2010 82 214 -0.033 -0.215 0.150 0.87 0.84 0.90 WHO/UNICEF DHS Version 1 1984 1989 23 23 0.067 -0.229 0.362 0.71 0.44 0.86 WHO/UNICEF DHS Version 2 1989 1992 24 24 -0.051 -0.205 0.103 0.89 0.77 0.95 WHO/UNICEF DHS Version 3 1992 1998 43 50 -0.030 -0.166 0.106 0.92 0.87 0.95 WHO/UNICEF DHS Version 4 1997 2004 46 57 -0.057 -0.194 0.080 0.89 0.83 0.93 WHO/UNICEF DHS Version 5 2003 2009 45 46 -0.049 -0.209 0.110 0.85 0.76 0.91 WHO/UNICEF DHS Version 6 2008 2010 13 14 -0.022 -0.202 0.158 0.58 0.11 0.84

WHO/UNICEF MICS All 1991 2010 84 163 -0.060 -0.327 0.207 0.78 0.71 0.83

WHO/UNICEF MICS Version 1 1991 1997 42 48 -0.021 -0.250 0.208 0.85 0.75 0.91

WHO/UNICEF EPI All 1980 2010 73 220 0.020 -0.154 0.193 0.90 0.88 0.93

(15)

Table 4. Country-year dyads with more than one survey, and absolute differences.

Country Year DHS MICS EPI Absolute difference Pakistan 1990 0.427 0.956 0.529 Burkina Faso 2002 0.570 0.770 0.200 Mali 1995 0.375 0.527 0.152 Burundi 1999 0.891 0.744 0.146 Rwanda 1999 0.860 0.810 0.050 Nigeria 2002 0.214 0.248 0.034 Niger 2005 0.418 0.393 0.025 Bangladesh 1999 0.721 0.744 0.023 Georgia 1998 0.805 0.800 0.005 Guinea 191 0.365 0.370 0.005

(16)

49

Figure 1. Data and information pathways of administrative and surveys vaccination data.

7 NUMERATOR Child immunised? Correct dose? Card record No Yes Yes As correct dose As incorrect dose Not recorded Caretaker recall Does not know Card record As correct dose As incorrect dose Not recorded Caretaker recall As immunised Does not know Card record As correct dose As incorrect dose Not recorded Caretaker recall Does not know

HU record Not recorded HU record Not recorded No Administrative coverage Estimates (official and WHO)

As immunised immunisedAs As not immunised As not immunised _ As not immunised As correct dose As correct dose Aggregation As incorrect dose As incorrect dose Denominator sources Survey coverage HU: health unit (point of vaccine delivery). Denominator sources: censuses (official or informal lists, updated or estimated). Yellow squares indicate data which does no enter in the calculation of vaccination coverage. Bluish shadows includes data items for calculating (administrative and survey) coverage. Green shapes indicate correct recording; and orange shadowed shapes incorrect recording.

(17)

Figure 2. Differences between administrative and (a) DHS and (b) EPI surveys. -. 4 -. 2 .2 .4 0 Di ffe re nc e o f DHS a n d Ad m in is tra tiv e .2 .4 .6 .8 1

Mean of DHS and Administrative

observed average agreement 95% limits of agreement

a_DHS

y=0 is line of perfect average agreement

-.2 -. 1 .1 .2 .3 0 Di ffe re nc e o f EP I (f ro m WHO ) - Ca rd o r r e ca ll a n d A d m in is tra tiv e .2 .4 .6 .8 1

Mean of EPI (from WHO) - Card or recall and Administrative

observed average agreement 95% limits of agreement

a_swE

y=0 is line of perfect average agreement

Note that in graphic (a) the line of the mean difference lies below zero, and in graphic (b) above zero.

(18)

51

Figure 3. Correlation between administrative and (a) WHO/UNICEF estimates and (b) DHS. 0 .2 .4 .6 .8 1 WH O es tim ate 0 .2 .4 .6 .8 1 Administrative

reduced major axis line of perfect concordance a_w 0 .2 .4 .6 .8 1 DH S .2 .4 .6 .8 1 Administrative

reduced major axis line of perfect concordance a_DHS

(19)

Figure 4. Coverage from different data sources in Ethiopia. .2 .3 .4 .5 .6 .7 1995 2000 2005 2010 Years DHS MICS EPI Administrative