{'fraction_non_alphanumeric': 0.03256182953330344, 'fraction_numerical': 0.017419199061777794, 'mean_word_length': 5.248275862068965, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Vaccination timeliness and delay in low- and middle-income countries: a systematic review of the literature, 2007-2017

ABSTRACT Background: Traditional measurements of vaccine coverage at specific ages can mask poor vaccine timeliness. However, optimal measurement of timing is unclear due to variations in countries’ recommended vaccination schedules and lack of a commonly accepted standard for “timeliness”. We conducted a systematic review of literature on vaccine timeliness and delay in low- and middle-income countries from 2007 to 2017. Methods: A search of articles published between January 1 2007 and December 31 2017, was performed in PubMed, EBSCOhost, and Embase. Results: 67 papers were included, of which 83% used a categorical measure of delay and 41% evaluated continuous delay. The most common age at assessment was 1 month, with earlier age benchmarks typically used with birth doses. Conclusions: Categorical definitions of vaccination timing vary widely, with benchmarks of delay varying from days to weeks to months. Use of a continuous measure of vaccine delay may be more informative and comparable.


Introduction
Vaccines have proven to be one of the most effective preventive interventions and remain one of public health's most successful means for controlling and eradicating serious and sometimes fatal diseases. 1 Since 1974, the World Health Organization (WHO) has promoted vaccines through the Expanded Program on Immunization (EPI), which includes recommendations for countries to publicly fund select vaccines. 2 The initial EPI-recommended pediatric vaccines were Bacillus Calmette-Guérin (BCG), oral polio (OPV), diphtheria-tetanus-pertussis (DTP), and measles vaccine, although the list has since expanded to include hepatitis B, Haemophilus influenzae type b (Hib), rubella, pneumococcal conjugate vaccine (PCV), and rotavirus vaccine. 2 Most countries have a National Immunization Technical Advisory Group (NITAG) which provides countryspecific recommendations, such as which vaccines are offered, whether combination vaccines are used, and when vaccines should be administered. 3 Widespread use of these vaccines has had a lasting impact on the occurrence of vaccine-preventable diseases. The incidence of most vaccine-preventable diseases in the US has declined >99% over the twentieth century. 4 Worldwide, we are on the verge of polio and measles eradication, due in part to the widespread adoption of EPI programs.
Vaccination program performance has typically been assessed through measurements of vaccine coverage: the proportion of individuals who have received a vaccine by a benchmark age (such as 5 years), regardless of the timing of administration. 5 However, vaccination coverage and timeliness of vaccine administration are related but separate issues, and high levels of coverage can sometimes mask low levels of timeliness. For example, an analysis of the Vaccine Safety Datalink, a consortium of 10 healthcare delivery organizations in the United States, shows that although vaccination coverage is high (>90%), 6,7 less than half of the children had received all vaccine doses on time. 6 Untimely vaccination could relate to access or affordability, or could instead be the result of the parent's not accepting the recommended vaccine schedule and being vaccine hesitant. 8,9 Studies measuring vaccine timeliness have become more frequent in the literature in recent years. In 1998, Bolton et al. explored issues with vaccination coverage estimates within Baltimore, Maryland, and recommended the use of ageappropriate indicators, 10 and a 2007 article 11 and 2009 editorial 12 both encouraged more regular measurement of vaccine timeliness. Subsequently, additional studies have been conducted focusing on low-and middle-income countries (LMICs). LMICs have a disproportionately high burden of vaccine-preventable diseases, which are often more serious in younger infants so that administration of vaccines at an early age (as scheduled) is especially important. 13 Additionally, children in LMICs are more likely to be non-and under-vaccinated and encounter significant barriers to vaccination access, magnifying the challenges faced in vaccination receipt and delivery. However, the optimal measure of timeliness remains unclear, resulting from both differences in schedules among different countries, but more so by confusion as to what constitutes "timeliness" relative to vaccination date recommendations. There currently is no agreed-upon definition of timeliness across international bodies.
This systematic review presents a summary of the literature on vaccine timeliness and delay in low-and middle-income countries between the years of 2007 and 2017, with a focus on timeliness definition, age at assessment, countries/setting of research, and analytic approach, among others. It is the first paper, to the author's knowledge, which seeks to evaluate variable ways of measuring and quantifying vaccine timing in LMICs. This manuscript serves to describe the findings of previously conducted studies and to provide recommendations for future studies measuring vaccination timing.

Methods
This systematic review followed guidelines from the Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA). 14

Search strategies
Searches were performed in 3 different electronic databases: PubMed/MEDLINE, EBSCOhost, and Embase. The search terms used were ("vaccination" or "immunization" or "immunisation" or "vaccine" or "EPI") and ("timely administration" or "timeliness" or "on time"). The searches were limited to papers published between January 1, 2007 and December 31, 2017.
The titles of all papers returned through the use of the search terms were initially screened for relevance. The abstracts of all remaining papers were then assessed with specific inclusion and exclusion criteria in mind. Manuscripts were included if they met the following inclusion criteria: (1) studies were conducted on data from LMICs, as defined by the World Bank; 15 (2) studies examined pediatric vaccines that were a part of the WHO EPI 2studies only featuring adolescent and adult vaccines (e.g. human papillomavirus vaccine (HPV)) or vaccines not on the EPI (e.g. influenza vaccine) were excluded; (3) studies had to calculate some measure of timelinessbroadly defined as a vaccination measure which was age-specific; (4) the study had to provide some analysis about vaccination timelinessstudies presenting only descriptive statistics were excluded; comparisons between groups with or without statistical testing was required and, (5) the study had to be a non-randomizedcontrolled trial exploring vaccination outcomes, as RCTs test an intervention to change vaccine administration rather than assess vaccination timeliness. Finally, non-English studies were excluded.

Vaccination timing definitions
We use the following definitions: vaccine uptake refers to any vaccine measurement, vaccine coverage is the vaccine uptake measured without regard to any timing measure, and vaccination timing is vaccine uptake with some measurement of time or date. Vaccination timing could refer to one of the three conditions: (1) up-to-date vaccination means an individual received a vaccine by a certain threshold age, (2) vaccination delay is the continuous measurement of vaccinationeither in the age of vaccination, or the time elapsed since the vaccine was recommended to be administered, and (3) vaccination timeliness refers to vaccination administered within a certain time since a recommended age of vaccination.

Study selection
NBM removed all duplicates and assessed all titles for relevance. Two reviewers (NBM/ALW) independently assessed all abstracts and full-text publications for eligibility using the eligibility criteria laid out. All disagreements were resolved by discussion between reviewers, and the tie-break of a third reviewer. In addition to assessment for relevance, data were also extracted independently and a quality assessment was conducted by the two reviewers (NBM/ALW). The following data were abstracted: location of study; study population; study design; sample size; study year; vaccines considered; definition of timeliness/vaccine delay; explanatory variables considered. Risk of bias was assessed using an adaptation of the Downs & Black checklist, 16 which had previously been used in a systematic review of vaccine uptake. 17 Items relating to interventions (Question 4,8,(13)(14)(15)19,20,23,24,and 26) were dropped as they were not relevant to our inclusion criteria. As a result of eliminating the aforementioned questions, the maximum quality score was an 18. Studies were considered to be good quality if they achieved a score of 14-19, Moderate quality if they achieved a score of 9-13, and low quality if they achieved a score of <9. Further information regarding the quality assessment can be found in the Supplementary Materials.

Study quality and synthesis of findings
The goal of this review was to examine how researchers define and analyze vaccine timeliness. We present a description of the current state of studies on vaccination timeliness, and discuss the temporal and geographical distribution, the study and analytical design, the definition of timeliness, and the inter-group comparisons of timeliness.

Ethical approval
As a systematic review of previously published manuscripts, this study was not under the purview of studies examined by the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board.

Overview of included papers
The search terms yielded 488 articles for screening from PubMed, 388 papers from EBSCOhost, of which 16 were duplicates from the PubMed search, and 636 papers from Embase, of which 271 were unique non-duplicates with either PubMed or EBSCOhost. After the title, abstract, and full paper screen, 67 papers were included in the systematic review.  Full details regarding paper selection are found in Figure 1.

Definition
Studies measuring timing were generally of two types: those using continuous vs. categorical measures, although some papers used both. Only one study 75 did not explicitly evaluate vaccine timing on either a continuous or categorical scale, but rather just measured whether children were up-to-date by a threshold date of 14 weeks. Of the 66 remaining studies, 27 (41%) papers used a continuous outcome such as median delay in days or weeks: for example, Hughes et al. graphed cumulative incidence curves of age at vaccination for BCG, pentavalent, and polio vaccine series. 39 More simply analyzed, Wagner et al. present the mean age at measles vaccination for a number of different demographic groups. 76 The majority of papers (55/66, 83%) used some categorical measure of timeliness. More than half of these papers (30/55, 55%) used variable definitions for vaccine timeliness based on the vaccine in question (i.e. different definitions for measles vs. DTP vs. HepB), whereas (21/55, 38%) used a fixed definition of timeliness for all considered vaccines. Only 7% of papers (4/ 55) used an interval definition between doses to determine timeliness.
The most common age benchmark used to assess timeliness was around 1 month. All but three (18/21, 86%) of papers which used a fixed definition employed this timeframe, operationalized as 1 month [9 papers], 4 weeks [3 papers], 28 days [3 papers], 4.3 weeks [1 paper] and within the same month as recommended [2 papers]. Additionally, there were a few papers which used an age benchmark for timely vaccination of around 1 month for all vaccines other than Hepatitis B vaccine, which had a timeliness definition of 1 day [2 papers]. Other age benchmarks used were 1 day (4/55, 7%), though 3 of these papers were combined with longer delay definitions for other vaccines, 1 week (3/55, 5%) or 2 weeks (6/55, 11%). Lower benchmarks (e.g. 1 day, 1 week) were typically used with birth doses (HepB, BCG, OPV0). Most papers did not explain why specific vaccination benchmarks were selected; those that did indicated that the recommendations derived from various sources, including manufacturer's instructions 77 and/or used the WHO definition of full vaccination according to scheduled vaccination dates, not definitions of timeliness. 52 Finally, 4 papers (7%) evaluated timeliness in terms of the interval which elapsed between doses rather than the date of vaccination itself, which would result in 'timely' vaccination calculations if the first vaccine in the series was initiated late but all subsequent vaccines were given within the recommended interval. 21,23,73,84 Country under study  Overall, the largest represented WHO region was the African region (29/67, 43%). The next most represented WHO region was the Western Pacific region (19/67, 28%) with 68% of these papers (13/19) focused on China. Considerably fewer studies were conducted in the Region of the Americas (10/67, 15%), the South East Asian Region (8/67, 12%), and the Eastern Mediterranean Region (8/67, 12%). Finally, only 3 studies (4%) were conducted in the European Region (Table 2). Studies used data from 79 different countries (Table 3), with the most common countries studied being China (13/67, 19%) and Kenya (8/67, 12%). Most studies (53/67, 79%) were not nationally representative and instead involved samples from a particular district or city, 10 (15%) used nationally representative data from a single country, and 4 (6%) studies used data from multiple countries. There was no discernible pattern of specific countries as more common sources of data over the study period from 2007 to 2017.

Antigens
The vaccines assessed are shown in Figure 3. The most common vaccine was BCG (38/67, 57%), followed by the measlescontaining vaccine (33/67, 49%), and OPV or IPV (19/67, 28%), though some studies examined multiple vaccines ( Table 1). Timeliness of DTP was assessed in 13 papers (19%) and the pentavalent vaccine was explored in 6 papers (9%). However, only 21 papers (31%) examined full vaccination (i.e. receipt of all doses of all recommended EPI vaccines) as a primary or secondary study outcome measure, with most of the surveyed papers instead addressing a specific vaccination or a combination of vaccinations, but less than full vaccination, as their study measures.

Study design
The studies varied in design, with most using cross-sectional surveys (50/67, 75%) including the Demographic Health Survey or DHS (6/67, 9%), Multiple Indicator Cluster Survey or MICS (1/67, 2%), with the remainder using other types of non-nationally representative cross-sectional surveys (43/67, 64%). A smaller number, 13/67 (19%) of the studies were cohort studies (including data from surveillance systems and immunization information systems), and 3/67 (5%) were randomized controlled trials of which two were cluster-randomized, although none of these used vaccination as either a randomized intervention or an outcome, just as secondary analysis. There was also 1 included case-control study.

Date source(s)
A total of 62 studies (93%) reported the source of information on vaccination dates used to calculate vaccine timeliness within their study. Most studies (49/62, 79%) used data from vaccination cards (or other types of health cards, social welfare cards, or vaccination certificates based upon the country in which the study was conducted); 13% (8/62) used data from immunization information systems, 5% (3/62) used data abstracted from clinic records, and two (3%) based the date of vaccine receipt off the date of study, i.e. when study or data collection occurred on the date of vaccination.

Age
There was a wide diversity of ages in the study population across the 67 papers. In the 57 studies with a clearly stated age

Discussion
Measures of age-specific aggregate coverage can obscure or conceal untimely vaccination. The measurement of vaccination timeliness is increasingly perceived as essential to an assessment of immunization system performance and estimates of vaccination coverage, and inclusion of timeliness measures should progressively become the standard in assessment of vaccine coverage in the future. In this systematic review of studies on vaccination timing in LMICs, we found a lack of agreement about what constituted a "timely" vaccination, which was compounded by different vaccination schedules in different countries. Altogether, there was significant disagreement about the definition of timely and delayed vaccinations, with some papers using continuous delay and some categorical timeliness measures. The included papers spanned broad geographic reach but were heavily skewed to the African region and China. The most commonly studied antigens were BCG, measles, and OPV/IPV. Predictors were highly variable, but maternal sociodemographic factors were commonly used. The vast majority of studies included were cross-sectional in study design, most of which used data from vaccination cards, though there was very little consensus among the age range of children to assess for timing. The diversity in operationalization of variables limits comparisons among studies. The significant variability in defining timing found in this review presents problems for comparing timely vaccinations across countries. Categorizing vaccination timeliness requires a consensus on what constitutes a timely vs. untimely vaccine. Unfortunately, there is often not clear guidance on this. For example, the manufacturer's instructions for PCV list a fourdose schedule which can start as early as 6 weeks. WHO, however, recommends a three-dose schedule, with several interpretations by region (6, 10, 14 weeks; 2, 4, 6 months; or the first two doses, followed by a booster between 9 and 15 months) 85 . Clearly, any country-specific evaluation of timeliness should follow guidelines from that country's Ministry of Health, but the definition of timeliness (e.g. 1 week vs 1 month after recommended vaccination date) has different implications when doses within a series are spaced by 4 weeks vs. 2 months.
This variability and the difficulty in comparing findings across studies is compounded by the uneven geographic distribution represented in this review. Almost half of the included papers were focused on the African WHO Region, and another third were from the Western Pacific, specifically China. These papers thus do not paint a representative picture of vaccination timing among LMICs in the European, Eastern Mediterranean, South East Asian, and American Region. In particular, timeliness studies in South and Southeast Asia are needed given the population concentration there and a large burden of vaccine-preventable diseases.
Overall, the most commonly investigated vaccines (measles, BCG, DTP, and polio vaccine) seemed to be relative to international concern about a given disease. For example, measles and polio are both subject to global eradication plans, 86,87 perhaps a reason for their high representation in timeliness studies. Vaccination of measles in a timely manner is particularly important given that the disease is highly contagious and earlier vaccination can reduce the number of susceptible persons in the population, 88 although vaccination at 8-11 months of age is also associated with reduced effectiveness 89 . DTP dose 3 is often used in international assessments of vaccination programs, 90 not only because the vaccine protects against three serious diseases, but also because the vaccine has been widely available for decades, and the third dose represents an immunization system's ability to revaccinate a child on multiple occasions. A large number of studies investigated BCG, even though the vaccine has seen declining use; more countries have stopped routine BCG vaccination programs since 1981, though most of these are high-income countries in western Europe. 91 Newer vaccines, like rotavirus, PCV, and Hib vaccine, understandably were less likely to be included; these vaccines were added to the WHO EPI in 2013, 2012, and 2006, respectively 2 . However, with the increasing complexity of the EPI schedule and new vaccines being regularly added in different countries, there is also a need for future timeliness studies to address a broader array of antigens, particularly those that are strictly age-limited like Rotavirus and Hib.
Beyond the antigens assessed, there was also considerable variability in the predictors explored as correlates of untimely or delayed vaccination. Gender, education, socio-economic status, urban/rural residence, maternal age, maternal occupation, and birth setting can be situation-specific and highly informative. Interestingly, religion, caste, tribe, and distance to the clinic are potentially important predictors which were very rarely represented in this review but should be considered for inclusion in timing studies depending on the countries involved. Future studies should examine the contribution of vaccine hesitancy to vaccination timeliness and other vaccination metrics.
Vaccination date sources were also surprisingly variable. While most studies used vaccination cards, restricting a study population to those who have vaccination cards can reduce the generalizability of findings as those who possess vaccination cards might not be representative of the general population as a whole. Additionally, vaccination cards can be prone to data recording errors. Immunization information systems (IIS) are preferred in terms of providing an objective and accurate source of vaccination data without depending on retention of a vaccination card, and also allows for following a child longitudinally. Finally, IISs can assist with timeliness determinations while making vaccination data available to the family and clinicians. However, most LMICs do not have functional IISs and large, nationally representative surveys like DHS will continue to be a critical source of information about vaccination coverage and timeliness and researchers using these data sources should include timeliness measures in their analysis, though most included studies did not use nationally representative datasets like DHS.
Finally, the age benchmarks used to include children in timeliness studies were exceedingly variable, which is another, albeit very important, reason why a continuous measure is preferred in addition to improving comparability across studies. Certain historically used age groups (e.g. 23 months) will continue to be important seminal markers of 'on-time' vaccination coverage, although the use of continuous measure allows for uninterrupted use of those while also permitting much greater flexibility in looking at vaccination coverage and timeliness over the childhood life span. While analytic approaches also varied, the inclusion of survival analysis techniques in a majority of papers points to the utility, again, of continuous measures of vaccination delay as an interpretable and comparable metric.

Weaknesses in current studies and potential solutions
Any study on vaccination timing will require information about birth and vaccination dates. These are common data fields in electronic registries, such as immunization information systems, and should be included on all vaccination cards or booklets held by families. However, the retention of these cards can be low. In one study from the Southern Nations, Nationalities, and Peoples' region of Ethiopia, 51% of respondents did not have a vaccination card; however, of those without a vaccination card, 62% previously had a cardbut were unable to find it for the interview. Accordingly, only 20% never had a vaccination card given to them. 92 Relying on parental recall of vaccination dates is also likely fraught with problems. A systematic review of the validity of vaccination cards vs parental recall found that coverage (not including any timing or date information) from recall was an  94 Future studies on vaccination timing in LMICs will likely continue to rely heavily on recall of vaccines administered which means that some participants will have vaccination date information, and others will not. The use of statistics that take into account both left and right censoringsuch as nonparametric Turnbull estimators or parametric-accelerated failure time modelswould permit greater use of vaccination data from those both with and without vaccination cards. Turnbull estimation has been used in recent papers from India. 68,95 The Turnbull estimator allows for both right censoring (e.g. the child is not vaccinated at the time of the survey) and left censoring (the child has been vaccinated, but with an unknown date that is at the very latest, the day of the interview). These statistics can be brought up in SAS using the lifereg procedure (SAS Institute, Inc., Cary, NC, USA) or in R using the survival and SurvRegCensCov packages (R Foundation for Statistical Computing, Vienna, Austria). These procedures can be thought of as an extension of typical methods of survival analysis (e.g. Cox regression), which only consider right censoring.
We also recommend that future studies employ precise methods to estimate months. Months can vary between 28 and 31 days, according to the Western calendar, and some countries use different systems of measurementfor example, the Ethiopian calendar consists of 30-day months, in addition to an intercalated month of 5 or 6 days. As a shortcut, many studies employ 30 days as an average. In one study, where "month" and "year" of vaccination, but not "day," was collected, vaccinations were considered timely if it was given in the month it was due based on the child's birth date. 24 However, this would underestimate timely vaccinations (relative to a 1-month benchmark). A better method is to identify the actual date that a vaccine administration would be untimely. For instance, a child born on February 14 2007, who should receive a certain vaccine within one month for it to be timely would need to be vaccinated by March 14. These dates can be identified through certain coding steps (for instance, using an intnx statement in a data step in SAS).
Vaccination timeliness can be considered as both continuous (e.g. mean number of days delayed, or average age at vaccination) and categorical measures (e.g. % with a timely vaccination). Due to differences between countries in vaccination schedules and a lack of concordance in the literature on what constitutes a "timely" dose, a continuous measure may be more appropriate. Cumulative incidence curves (or reverse Kaplan-Meier plots) can graphically depict this information in an easy-to-understand manner. If a categorization is desired, multiple groupings (e.g. both timeliness based on 1 week and timeliness based on 1 month) is preferable, in the absence of any country-specific guidance on what constitutes a timely dose.
Finally, certain statistical basics should always be followed. Studies with clustering of the sample should specify survey procedures in the regression analysis (for instance, using Taylor series estimation of standard errors). In crosssectional or cohort studies, multivariable regression with an outcome of the "prevalence" or "likelihood" of a timely vaccine can utilize a binomial or log-binomial regression, which would output a "risk ratio" or "prevalence ratio" which is more easily understood than an odds ratio. 96

Strengths and limitations
This study has several limitations. We limited ourselves to studies published in PubMed, EBSCOhost, and Embase (in addition to studies found through an abstract search), however, manuscripts on timeliness could have also been submitted to regional journals not indexed in international databases. We also limited our search to LMICs, as these are countries where timely administration of vaccines can be an especially important mechanism to reduce morbidity from infectious diseases, and it was beyond the purview of this review to examine differences in timeliness between highincome countries and LMICs.

Conclusions
Ensuring children have access to timely vaccinations is a critically important and cost-effective way to decrease the incidence of disease and reduce morbidity within a population. In this systematic review of 67 papers from LMICs, analysis of vaccination timing has become more common over time, and papers represented all 6 WHO regions. However, there is a lack of concordance in the literature on what constitutes a "timely" vaccinationwith studies variously using benchmarks of 1 day, 1 week, 2 weeks, 4 weeks, 1 month, etc. As such, providing information on the continuous spread of date of vaccination administration (e.g. graphing the cumulative incidence of vaccination by increasing age), or characterizing median delay in vaccine receipt from the suggested schedule might provide more valuable information than a categorization (e.g. stating the proportion vaccinated by a certain benchmark), although both can be presented within one study.