{'fraction_non_alphanumeric': 0.040746367496396714, 'fraction_numerical': 0.02368431303805851, 'mean_word_length': 5.159309021113244, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Identifying Children and Youth With Autism Spectrum Disorder in Electronic Medical Records: Examining Health System Utilization and Comorbidities

Autism spectrum disorder (ASD) is a neurodevelopmental disorder requiring significant health and educational resources for affected individuals. A reference standard for ASD was generated from an existing population‐based cohort of 10,000 children and youth aged 1–24 years who were randomly selected for chart abstraction from 29,256 patients from 119 family physicians. We developed and validated an algorithm to identify children and youth with ASD within an electronic medical record system (N = 80,237, aged 1–24 years) in order to examine the prevalence of comorbidities and quantify health system utilization within the cohort. We identified 1,062 children and youth with ASD representing a prevalence of 1.32%. Compared to individuals without ASD, those with ASD had a higher prevalence of asthma, were more likely to visit a specialist, undergo surgery, and be hospitalized for psychiatric reasons. Children and youth with ASD in Ontario have complex health system needs, illustrated through a significant burden of comorbidities and increased health system utilization. Lay Summary Our paper generates population‐based estimates of health system use by children and youth with ASD, who have a higher burden of comorbidities than the general population. We developed a case‐finding algorithm and applied it in electronic medical records to create a cohort of children and youth with ASD, thereby generating an important resource to further study the health care needs of individuals with ASD.


Introduction
Autism spectrum disorder (ASD) is a group of lifelong neurodevelopmental disorders (NDD) that present as complex phenotypes with a high degree of heterogeneity. The prevalence of ASD in 8-year-olds in the United States has been estimated to be about 1 in 59 [Baio et al., 2018]. In Canada, it was recently estimated that 1 in 66 children between the ages of 5 and 17 years has the disorder [Ofner et al., 2018]. In general, ASD is approximately fourtimes more likely to be diagnosed in males vs. females, varies based on geographical region, and is increasing over time; factors that are reflected in both the American [Baio et al., 2018] and Canadian estimates [Ofner et al., 2018].
The increased burden of comorbid conditions makes meeting the needs of these patients more complex, and corresponds to increased health system utilization compared to the general population [Dizitzer et al., 2020]. In fact, patterns of comorbidities have been shown to identify subgroups of ASD with distinct clinical trajectories. [Kohane, 2015] This makes understanding the co-occurrence of these conditions with ASD across different populations and geographical regions vitally important for health system planning. There remains a need for evidence-based recommendations to inform programs and services for individuals with ASD and their families. To generate the needed evidence, efforts are underway to develop different approaches to identify individuals with ASD at a population level, using information readily available in electronic medical records (EMRs) [Bush, Connelly, Perez, Barlow, & Chiang, 2017;Coleman et al., 2015;Lingren et al., 2016] and administrative health data [Coo, Ouellette-Kuntz, Brownell, Shooshtari, & Hanlon-Dearman, 2018;Dodds et al., 2009]. The objectives of this study were to (1) develop an algorithm for the identification of children and youth with ASD in an EMR system in Ontario, Canada and (2) use this algorithm to examine the prevalence of comorbid conditions in children and youth with ASD and to quantify their health system utilization compared to unaffected children and youth.

Study Design and Data Sources
In order to compare the prevalence of comorbidities and health system utilization between children and youth with and without ASD, we used a population-based cohort of children and youth in Ontario, Canada via family medicine EMR records from the Electronic Medical Record Primary Care (EMRPC) database (formerly known as EMRALD). EMRPC consists of all clinically relevant information for over 400,000 patients, from over 350 family physicians using PS Suite® EMR (formerly Practice Solutions). Physicians' participation in EMRPC is entirely voluntary and only requires that they have been using the EMR for at least 2 years [Tu, Widdifield, et al., 2015].
Within this cohort, we developed and validated an algorithm for ASD, identified the affected children and youth and subsequently linked the cohort using unique encoded identifiers to several health administrative databases held at ICES (formerly known as the Institute of Clinical Evaluative Sciences) (Appendix A). ICES is an independent, nonprofit research institute funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). As a prescribed entity under Ontario's privacy legislation, ICES is authorized to collect and use health care data for the purposes of health system analysis, evaluation, and decision support. Secure access to these data is governed by policies and procedures that are approved by the Information and Privacy Commissioner of Ontario.

Identifying ASD in EMRPC: Algorithm Development and Validation
A reference standard for ASD was generated from an existing population-based cohort of 10,000 children and youth aged 1-24 years who were randomly selected for chart abstraction from 29,256 patients from 119 family physicians; the details of this cohort have been described elsewhere [Hauck, Lau, Wing, Kurdyak, & Tu, 2017]. Briefly, trained nurse chart abstractors manually reviewed patients' EMR for diagnoses of ASD (including Asperger's Syndrome and pervasive developmental delay (PDD)), and other neurodevelopmental (e.g., ADHD) and mental health disorders (e.g., depression). A detailed abstraction manual was used to score each record as "definite" or "possible" for the presence of disease/condition, or if the disease was "ruled out." Abstractor-identified cases were confirmed by a family physician (KT). Overall, 112 individuals were identified as having a confirmed diagnosis of ASD (mean age 12.7 years, 1:3 ratio females to males) and 9,888 were determined to not have ASD (Fig. 1).
We used the 10,000 individuals in the chart abstraction cohort to develop and test algorithms for the identification of individuals with ASD in the larger EMRPC cohort (N = 80,237 children and youth ages 1-24 years [as of March 1st, 2016] in EMRPC). Established methods for algorithm development, testing and selection, routinely employed at ICES, [Ivers, Pylypenko, & Tu, 2011;Krysko, Ivers, Young, O'Connor, & Tu, 2015] were used to generate and evaluate case-finding algorithms for ASD. Potential algorithms included relevant case identification information from a keyword search of the cumulative patient profile (CPP) available in EMRPC (Table I), as well as Ontario Health Insurance Plan (OHIP) physician billing codes. The CPP is a field at the top of the EMR that summarizes key diagnoses, medications, and family histories. While there are specific diagnostic codes for ASD (e.g., ICD-9 299.x), there are no OHIP physician billing codes specific to ASD. Therefore, to capture physician billing related to ASD, the OHIP billing codes 299 "childhood psychoses e.g., autism," and 315 "specific delays in development" were used. These codes are commonly used by physicians to indicate a health services related to an ASD diagnosis. The tested algorithms varied with respect to the data sources included (e.g., CPP, OHIP billing), and the number and the frequency of codes required (e.g., 1-3 codes in 1-3 years). The final, optimal algorithm was selected to maximize both the positive predictive value (PPV) and sensitivity of the algorithm.

Measuring Prevalence of Comorbidities and Health System Utilization through Administrative Health Data
The full EMRPC study cohort (N = 80,237) was linked to administrative data using unique encoded identifiers and analyzed at ICES. The Registered Persons Database (RPDB) provided information on demographic variables (e.g., age, sex, neighborhood-level income quintile, and geographic location). Statistics Canada's Postal Code OM Conversion File (PCCF) links postal codes to census data and was used to determine individuals' neighborhood-level income quintile, and geographic location (rural/suburban/ urban). Neighborhood-level income quintile is a measure of socioeconomic status that divides the population into five neighborhood-level income groups (1-lowest, 5-highest).  Autism "?" "Autism" " not" "Asperger" " ? mild" "Asberger" " vs" "Autistic" "autistism" "Austistic" "Aspberger" "pervasive developmental disorder" "peervasive developmental disorder" "pdd" Geographic location was defined based on community size, where communities with <10,000 residents were defined as rural, communities with 10,000-99,999 residents were defined as suburban, and communities with >100,000 residents were defined as urban.
The Canadian Institute for Health Information Discharge Abstract Database (CIHI-DAD) and National Ambulatory Care Reporting System (CIHI-NACRS) provided information on diagnoses, procedures and other characteristics for hospital visits, community-based ambulatory care, outpatient clinics, day surgeries, and emergency department visits. The OHIP physician billing database was used to provide information on diagnoses and medical service billings by Ontario physicians including specialists (e.g., psychiatrists, pediatricians, neurologists). To determine the prevalence of comorbidities in individuals with and without ASD, we focused on conditions with existing validated case-finding algorithms for children and youth in Ontario administrative data (e.g., asthma [Gershon et al., 2009], diabetes [Guttmann et al., 2010]). Information on psychiatric comorbidities was based on definitions from the MHASEF report: The Mental Health of Children and Youth in Ontario: 2017 Scorecard [MHASEF Research Team, 2017]. A look-back window of 3 years (from the index date of March 1st, 2016) was used to capture recent health system utilization and a lifetime lookback from the index date was used to estimate prevalence of comorbid conditions (chronic and psychiatric).
This project was approved by the Research Ethics Board at the University of Toronto and Sunnybrook Health Sciences Centre, Toronto, Canada.

Statistical Analyses
To validate the identification of patients with ASD within the EMR, sensitivity, specificity, PPV, negative predictive value (NPV), and corresponding 95% confidence intervals (CI) were calculated for each tested algorithm using the identified reference standard.
To compare demographics, comorbidities, and health system utilization differences between individuals with and without ASD in the full cohort, the chi-squared test was used for categorical variables and analysis of variance was used for continuous variables. Modified Poisson regression models with robust variance estimators [Zou, 2004] were used to estimate prevalence ratios (PR) and 95% confidence intervals (CI) for the relationship between ASD and comorbidities and health system use. Results were considered significant at an alpha level of 0.05. This is demonstrated by a 95% CI that does not include unity (e.g., PR = 1.0, indicating the exposure and outcome are independent). While estimates could have been generated using logistic regression, because the outcomes are common, odds ratios would likely over-estimate the relationship between ASD and each outcome. Models were adjusted for age (continuous, years), sex, neighborhood-level income quintile (1 = lowest, 5 = highest; categorical), and geographic location (rural/suburban/urban; categorical). Table II lists the algorithms that were tested and compared for identifying individuals with and without ASD within the EMR. The optimal algorithm was a keyword search of the CPP with a sensitivity of 82.1% (95% CI 73.8, 88.7%), specificity of 100% (95% CI 99.9, 100%), PPV of 98.9% (95% CI 94.2, 100%), and NPV of 99.8% (95% CI 99.7, 99.9%).

Identifying ASD in EMRPC: Algorithm Development and Validation
OHIP billing codes, alone or in combination did not show good discrimination. At least one instance of the OHIP billing code 299 (childhood psychoses e.g., autism) ever ("Any OHIP billing code (299) ever") had a sensitivity of 28.6% (95% CI 20.4, 37.9) and specificity of 99.9% (95% CI 99.9, 100.0). Increasing the number of codes to two or three in 1 year, or the number of years in which these codes could occur (e.g., 2-4 years) did not improve the algorithm. Using both OHIP billing codes (299 and 315) also did not perform as well as information from the CPP alone (Table II, e.g., "Any OHIP billing code (299 or 315) ever"). Further, when OHIP billing codes were used in combination with the CPP search terms, little improvement was seen, with only slight increases in the sensitivity, regardless of the number of codes within a given time frame (two or three), the time frame considered (1-3 years) or the OHIP codes considered (299 alone, or in combination with 315) ( Table II). The CPP keyword search optimized both sensitivity and positive predictive value, while maintaining high specificity and was therefore selected as the final, optimal algorithm. Secondary review of records for individuals misclassified by the optimal algorithm (1 false-positive, 20 false negatives) found that all false negatives included some mention of ASD, but this information was captured in free text and consult notes and not in the CPP. Of the 20 individuals classified as false negatives, seven had a clear diagnosis of ASD in their medical record, while the remaining records were found to mention a possible ASD diagnosis, but these could not be confirmed. Further, nine of the false negatives were identified as having Asperger's Syndrome, a milder form of ASD. Notably, review of these medical records revealed complex medical histories including diagnoses of ADHD (N = 5 confirmed, and N = 8 with a possible ADHD diagnosis), features of anxiety and depression (N = 7) or other possible NDD (N = 7). The single false-positive was confirmed to have a clear diagnosis of ASD in the EMR, indicating a potential abstraction error. Chart review showed that this individual had multiple co-occurring conditions including ADHD, and anxiety/depression. Abbreviations: EMRPC, Electronic Medical Record Primary Care; ASD, autism spectrum disorder; CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value; CPP, cumulative patient profile (within the Electronic Medical Record). a The Cumulative Patient Profile (CPP) is a text entry field at the top of a patient's log within the Electronic Medical Record that summarizes key information. Here CPP refers to a keyword search within the CPP. OHIP billing code 299 refers to "childhood psychoses e.g., autism" and code 315 refers to "specific delays in development".
Applying the optimal algorithm, we identified 1,062 out of 80,237 children and youth between the ages 1-24 years as having a diagnosis of ASD (Table III). Of these, the median age was 12.6 years and the majority (78.3%) were male (ratio of females to males 1:4). This corresponds to an ASD prevalence of 1.32%, or 1:75 in 1-24 year olds and 1:56 in 5-17 year olds.

Measuring Prevalence of Comorbidities and Health System Utilization through Administrative Health Data
After adjusting for age, sex, income, and geographic location, compared to those without ASD, individuals with ASD had a significantly higher prevalence of asthma (PR = 1.36 95% CI 1.21, 1.53) and mood and affective b True positives refer to the number of individuals correctly identified by the algorithm as having ASD in the reference standard of 10,000 individuals (truly have ASD based on chart abstraction). c True negatives refer to the number of individuals correctly identified by the algorithm as not having ASD in the reference standard of 10,000 individuals (truly do not have ASD based on chart abstraction). d False negatives refer to the number of individuals incorrectly identified by the algorithm as not having ASD in the reference standard of 10,000 individuals (truly do have ASD based on chart abstraction). e False positives refer to the number of individuals incorrectly identified by the algorithm as having ASD in the reference standard of 10,000 individuals (truly do not have ASD based on chart abstraction). f Sensitivity refers to the algorithm's ability to accurately detect true positives. g Specificity refers to the algorithm's ability to accurately detect true negatives. h The positive predictive value (PPV) refers to the proportion of individuals classified by the algorithm as having ASD who truly have ASD. i The negative predictive value (NPV) refers to the proportion of individuals classified by the algorithm as not having ASD who truly do not have ASD. j The optimal algorithm selected to maximize both the sensitivity and the positive predictive value (PPV). Note. Percentages sum to 100 within column categories. Abbreviations: EMRPC, Electronic Medical Record Primary Care; ASD, autism spectrum disorder; N, number; SD, standard deviation. a ICES suppresses cells of 5 or less to protect the privacy interests of individuals and reduce the chance of re-identification. b N = 1,062 individuals identified with ASD in a population of N = 80,237 represents an ASD prevalence of 1.32%. c Income quintile is the neighborhood-level income quintile. d Geographic location is defined based on community size, where communities with <10,000 residents are defined as rural, communities with 10,000-99,999 residents are defined as suburban and communities with >100,000 residents are defined as urban.
In multivariable adjusted models, individuals with ASD were approximately five times more likely to visit a psychiatrist (PR = 4.73, 95% CI 4.20, 5.33) and over three times more likely to visit a neurologist (PR = 3.39, 95% CI 2.73, 4.23). Individuals with ASD were also twice as likely to see a pediatrician, ENT, or endocrinologist, compared to those without ASD (Table V).

Discussion
Our paper is among the first to generate population-based estimates of health system use by children and youth with ASD. We identified and validated an algorithm to identify a cohort of over 1000 children and youth with ASD in EMR data. Using this cohort, we demonstrated that these individuals have a significant burden of comorbidities (chronic and mental health conditions) and increased interactions with the health system, whether through specialist visits, surgeries or hospitalizations. Individuals with ASD have been shown to have a significant burden of comorbidities including seizure, psychiatric, gastrointestinal, autoimmune, endocrine, allergic disorders Doshi-Velez, Ge, & Kohane, 2014;Kohane et al., 2012;Weiss et al., 2018], as well as other neurodevelopmental disorders [Joshi et al., 2010]. This increased burden coincides with increased health system utilization [Cummings et al., 2016;Schlenz, Carpenter, Bradley, Charles, & Boan, 2015] that is sustained into adulthood [Cashin, Buckley, Trollor, & Lennox, 2018;Croen et al., 2015]. In Ontario administrative health data, Weiss et al. [2018] examined health services utilization in young adults with ASD (ages 18-24 years) with and without other developmental disability. They also found that compared to those without ASD, individuals with ASD had a significantly higher burden of comorbidities-both psychiatric and chronic conditions (e.g., diabetes, hypertension, asthma). They were also more likely to visit a pediatrician, psychiatrist, neurologist, and visit the emergency room for a psychiatric reason. These findings support those of the current study where instead we focused on children and youth between the ages of 1 and 24 years. Together, these studies add to the body of evidence finding that the burden of comorbidities and health system utilization begins early and is sustained throughout childhood and into early adulthood.
To examine the prevalence of comorbid conditions and health system utilization in children and youth with ASD, we first had to develop an algorithm to identify individuals with ASD within the EMR. In doing this, we identified a cohort of 1,062 children and youth with ASD, corresponding to an estimated prevalence of 1:56 for children aged 5-17 years of age. These results are in line with national estimates in Canada included in the Public Health Association of Canada (PHAC) 2018 Autism Surveillance Report [Ofner et al., 2018] which reported an overall prevalence of 1:66. While identifying the optimal algorithm, we found the keyword search of the CPP to be both the most sensitive and specific way of identifying individuals with ASD within EMR. OHIP billing codes, alone or in combination, with variable frequency and time-periods did not show good discrimination. Case-finding algorithms developed using physician billing and diagnostic codes typically work for physician-diagnosed conditions, with good sensitivity and specificity (e.g., asthma, type 2 diabetes, and hypertension [Gershon et al., 2009;Guttmann et al., 2010;K. Tu, Campbell, Chen, Cauch-Dudek, & McAlister, 2007]). In contrast to these physiciandiagnosed conditions, ASD is frequently identified and managed by other health professionals (e.g., psychologists) and using support services that are often not reflected through health system billings and therefore not captured within administrative health data. This has significant implications for the development of casefinding algorithms for ASD in administrative health data [Coo et al., 2018]. a ICES suppresses cells of 5 or less to protect the privacy interests of individuals and reduce the chance of re-identification. b Adjusted for age (years), sex, income (quintile, 1 = lowest), and geographic location (rural/suburban/urban). * Statistically significant P < 0.05.
This study has numerous strengths including the use of a reference standard obtained through detailed manual chart abstraction for algorithm validation. Linking this cohort to Ontario administrative health data is a major strength as it allows for the examination of physician billing information and patterns in health system utilization. We were able to do this in a large cohort (>1000 individuals with ASD), providing useful insights into the broad healthcare needs of young people with ASD.
There are also several limitations to this study. First, our algorithm had a sensitivity of 82% and consequently it may have missed some cases of ASD. However, these are likely to be milder cases of ASD. Although the EMRPC patient population tends to be more rural and have higher socioeconomic status relative to the broader Ontario population; the age and sex adjusted prevalence of chronic conditions and measures of comorbidity are similar to rostered patients (those that have a family physician) in Ontario. [Tu, Widdifield, et al., 2015] The generalizability of EMRPC is also limited as it only includes EMRs from family physicians/general practitioners, and so it does not include the medical records of children who exclusively see pediatricians. Notably, the majority of children in Ontario receive their primary care through family physicians [Guttman, Lam, Schultz, & Jaakkimainen, 2006], especially in the rural setting as specialists largely locate to urban centers [Tepper, Schultz, Rothwell, & Chan, 2005]. Finally, the characteristics of family physicians choosing to participate in EMRPC differ somewhat from those of all Ontario physicians. Specifically, a greater proportion of EMRPC physicians are female (55.5% vs. 44.8% in Ontario) and as reflected in the EMRPC patient population, a greater proportion live/practice in rural or suburban settings (11.4% rural and 22.4% suburban in EMRPC vs. 4.8% rural and 14.6% suburban in Ontario). EMRPC physicians are also younger (mean age 47 years, vs. 53 in Ontario) and consequently, had been practicing medicine for an average of 3.5 years fewer, relative to Ontario physicians at the time of data collection.
It is also important to note that we restricted our study to the examination of comorbidities with currently validated algorithms for case identification in children within Ontario provincial administrative data. While this is also a major strength of this analysis it means we were not able to examine other potentially important comorbidities (e.g., epilepsy, gastrointestinal issues). We were also not able to examine services administered by other health professionals (e.g., psychologists, social workers, occupational and physical therapists) not captured by physician billing codes in administrative health data, but who are known to play an important role in the diagnosis and care of individuals with ASD.
Overall, this study demonstrates that children and youth with ASD in Ontario have complex health system needs, illustrated through a significant burden of comorbidities and increased health system utilization. This work was accomplished by first developing and validating an ASD case-finding algorithm within EMR in Ontario, simultaneously generating an important resource to further study the health care needs of individuals with ASD. System-level planning in both the health and educational sectors is needed to address the sustained and complex needs of individuals with ASD. Future work should focus on examining health system use trajectories in these individuals across the life-course.