{'fraction_non_alphanumeric': 0.02980140052806796, 'fraction_numerical': 0.01175525198025485, 'mean_word_length': 5.145033860045147, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Autism Spectrum Disorder Parent Report for Outcome Monitoring: A Preliminary Report of Development and Clinical Utility.

OBJECTIVE
Children with autism spectrum disorder (ASD) face challenges across many functional domains. A tool that gathers relevant clinical information before visits, emphasizing symptoms that are likely to change over development and inform clinical interventions, could improve health care quality, allowing for more patient-centered and efficient care. This study evaluated the clinical utility and preliminary psychometrics of the ASD Parent Report for Outcome Monitoring (ASD-PROM), a web-based measure assessing competence in core features of ASD, along with the breadth of concerns and comorbidities that frequently co-occur with ASD.


METHODS
An interdisciplinary team drafted the ASD-PROM and made iterative revisions based on parent feedback. Parents of 62 children completed the ASD-PROM before their autism-specialty clinical visit, 53 completed the ASD-PROM twice, and 48 completed the Vineland Adaptive Behavior Scales, Second Edition (Vineland-II) concurrently. Parents (n = 25) and clinicians (n = 13) completed postvisit surveys to assess utility (phase 1). The ASD-PROM was then released for general clinical use (phase 2).


RESULTS
On a Likert scale (1 = very poorly, 10 = very well), parents found that ASD-PROM items described their child's abilities well (median = 8.0; interquartile range [IQR]: 7.0-9.5) and had a positive effect on care (median = 8.0; IQR: 7.0-10.0). Clinicians found the ASD-PROM effective in assessing parent-reported patient abilities (median = 9.0, IQR: 7.0-9.0) and felt the ASD-PROM helped make their care more patient-centered and efficient (both median = 8.0, IQR: 6.0-9.0). Two-week test-retest reliability was acceptable (0.95). ASD-PROM scores correlated positively with scores from similar domains on the Vineland-II (Pearson r 0.30-0.50, medium to large effects).


CONCLUSION
The ASD-PROM is a freely available tool to gather information on developmental and behavioral functioning in children with ASD before autism-specialty clinical visits. Clinical utility and preliminary psychometrics are promising, although limitations (including a low response rate during clinical use and a need for additional in-depth assessments and potential resulting modifications to the tool) remain to be addressed. Ultimately, the ASD-PROM may help promote patient-centered and efficient care for children across a wide range of ages and developmental levels.

developmental disorder that presents in early childhood and is characterized by impairments in social communication and social interaction accompanied by restrictive and repetitive behaviors. 1 Children with ASD and their families face many challenges and comorbidities that affect health and quality of life, including difficulties with sleep, feeding, toileting, safety awareness, maladaptive behaviors, and psychiatric comorbidities. [1][2][3][4][5][6][7] Clinical care for children with ASD requires periodic monitoring of developmental and behavioral progress to assess effectiveness of services and improve quality of care. 8 During visits, clinicians must address caregivers' primary concerns while also monitoring for the wide range of comorbid medical and behavioral conditions that frequently occur in individuals with ASD. Both of these goals must ideally be completed in a relatively short amount of time, given the time constraints allowed within a clinic visit and the challenges of lengthy visits for individuals with ASD. 9 Patient-reported outcome measures can track the health status and disease burden in an individual and monitor response to treatment. This can improve patient-centered aspects of medical care by helping families and clinicians to focus medical visits on primary areas of concern. 10 Patient-reported outcome measures can also decrease the time necessary for medical historytaking during a clinic visit, thus increasing the time available for counseling and care coordination. 11 In some cases, these benefits can occur because the measure itself addresses a patient's primary concerns; in other cases, it can occur because the measure monitors a child's status across a variety of domains, thus allowing the identification of "secondary concerns" that might substantially alter the clinician's plan of care but without requiring extensive in-person questioning about these concerns. Ultimately, such measures can improve health care quality, 12 patient-clinician communication, and patient satisfaction. 10 For use among children with ASD, an ideal measure would obtain information on developmental and behavioral function before clinical visits and address a variety of functional domains applicable to children with ASD across a wide range of ages, developmental stages, and functional levels. It would be freely available to maximize accessibility, even in low-resource settings. It would also be available electronically to allow for the use of adaptive test formats (computerized adaptive testing), to streamline the transfer of results into the electronic medical record, and to allow more efficient sharing of information. 13 Use of monitoring tools has been previously shown to improve efficiency when the tools are available in electronic format. 14,15 They also tend to have fewer missing values than paper forms. 15,16 Of particular importance, streamlined transfer of information into the medical record may reduce documentation demands on clinicians. These characteristics are of increasing importance in the face of a limited workforce of developmental behavioral clinicians and reports of clinician burnout. 17 There are numerous validated, reliable instruments with standardized scores that can help support diagnostic labels and requests for services. Some questionnaires focus on assessing behaviors and symptoms of specific disorders. For example, the Social Responsiveness Scale, Second Edition 18 and the Gilliam Autism Rating Scale, Third Edition 19 are commonly used to assess core behaviors associated with ASD. Similarly, self-report measures such as the Children's Depression Inventory, Second Edition 20 and Revised Children's Manifest Anxiety Scale, Second Edition, 21 respectively, assess symptoms of depression and anxiety. Numerous scales assess for sleep-related concerns. 22 Other measures are intended to screen for a wide range of psychological and behavioral issues (e.g., depression, anxiety, inattention, and hyperactivity). These include the Behavior Assessment System for Children, Third Edition (BASC-3) 23 and the Achenbach System of Empirically Based Assessment (including the Child Behavior Checklist, Youth Self-Report Form, and Teacher Report Form). 24 Still, others, such as the Vineland Adaptive Behavior Scales, Second Edition 25 and the Adaptive Behavior Assessment System, Third Edition, 26 assess daily functioning skills.
Such measures have numerous strengths: As part of a comprehensive evaluation, they can provide a relatively detailed picture of a child's strengths and weaknesses. However, they are not without their drawbacks. Most were developed as "paper-and-pencil" measures and some are not available electronically, although this is changing with the advent of new technologies. In this context, they are often static measures (rather than having questions adapted based on responses to other questions), and thus, raters often have to respond to items that may not be appropriate for the child's developmental level. In addition, although some measures are multifaceted (e.g., the BASC-3 includes adaptive functioning scales), in general these scales individually do not capture the full breadth of concerns that may need to be addressed at a monitoring visit for a child with ASD. Also, of note, most of these scales were initially developed for categorical diagnosis rather than focusing on symptoms that are likely to change over time in an individual.
A few questionnaires have been developed with the goal of monitoring symptoms that are likely to change over time among individuals with ASD. For example, the Autism Treatment Evaluation Checklist (ATEC) measures outcomes but does not differentiate developmental from behavioral concerns and does not have items that apply to children with higher developmental functioning. 27 In addition, the language in the ATEC focuses on skills children lack rather than new skills they have gained and thus does not meet a strength-based model of care. The Autism Behavior Inventory (ABI) and Autism Impact Measure also measure outcomes 28,29 but do not assess the breadth of medical comorbidities necessary for measuring outcomes of multidisciplinary ASD care. In addition, modification of the ABI and use in large research studies is prohibited, and only the paper-based version of the ABI is publicly available. The Pervasive Developmental Disorder Behavior Inventory 30 addresses both adaptive skills and maladaptive behaviors but is not freely available.
This study describes a process improvement initiative to develop a tool to gather targeted information from parents on developmental and behavioral functioning in children with ASD before autism-specialty clinical visits. This project involved the development of the ASD Parent Report for Outcome Monitoring (ASD-PROM) and initial evaluation of the clinical utility and preliminary psychometrics thereof. Our intent was neither to replace the many well-validated instruments frequently used by many clinicians to screen for or Vol. 42, No. 4, May 2021 provide in-depth characterizations of specific areas of function (particularly during diagnostic assessments) nor to supplant expert clinician history-taking that targets parents' primary concerns during visits. Instead, we aimed to develop a targeted questionnaire, specifically for children with ASD, that would address competence in features core to ASD (social communication and restricted/repetitive behaviors) along with the breadth of concerns and comorbidities that frequently co-occur with ASD, emphasizing symptoms that are likely to change over development and inform clinical interventions. Overall, the ASD-PROM aims to improve the quality of care delivered to patients with ASD by addressing the range of concerns that experienced clinicians wish to monitor most closely among children with ASD, helping clinicians to identify areas of need in advance of the visit and to target their attention and counseling to priority areas for each individual child and family.

METHODS
This project was performed over several phases. Supplemental Figure 1 (Supplemental Digital Content 2, http://links.lww.com/JDBP/A280) describes the sequence in which each step took place.

Pilot Phase: Survey Development
A multidisciplinary workgroup of professionals from a tertiary pediatric medical center (Boston Children's Hospital), including 6 developmental pediatricians, 3 neurologists, 4 psychologists, and 2 speech and language therapists initially drafted the Autism Spectrum Disorder Parent Report for Outcome Monitoring (ASD-PROM) through an iterative series of consensus discussions. Workgroup members were specifically chosen to capture a wide range of subspecialty expertise (in addition to autism), including sleep, feeding, toileting, language and augmentative communication, behavior management, quality improvement, and survey methodology. Items were developed to address core areas of development (communication, social, play, and adaptive skills) and behavioral functioning (restricted and repetitive behavior, attention, mood, sleep, and maladaptive behaviors) that are typically assessed in the clinical history and are important to the care of children with ASD. The questions were grouped according to 4 broad functional domains and 1 section on overall improvement, and items were included to assess skills for children from toddlerhood through adolescence. The tool was formatted with a developmentally progressive sequence based on crossdisciplinary clinical expertise. Responses used a 5-point Likert scale, indicating the frequency of each assessed developmental skill and behavior, within the 6 months before survey completion.
Parents (n 5 8) of children with ASD piloted the initial version of the ASD-PROM. The children ranged in age from 3.7 to 15.0 years (median 5 7.7, interquartile range: 5.2-11.7). Six of the 8 children were male (Table 1). In this pilot phase, parents independently completed the ASD-PROM in the presence of a researcher. Parents took 15 minutes on average to complete the ASD-PROM. The researcher then conducted a cognitive interview 31 with each parent regarding their impressions of the ASD-PROM. The interview addressed the clarity of the questions, whether the survey captured relevant information regarding the child's progress, and impressions about its family-friendliness. The ASD-PROM was iteratively revised after feedback was obtained from each parent interview.
Several significant changes were made to the ASD-PROM as a result of parent feedback during the pilot phase. Changes included rewording 33 questions for clarification purposes, adding 16 questions (many of which were added when parents pointed out that a single question in the pilot could have 2 different meanings, both of which the group considered important), and removing 12 questions that were found to be redundant or unnecessary to limit response burden and minimize parental discomfort (Supplemental Figure 2, Supplemental Digital Content 2, http://links.lww.com/JDBP/A280). Questions edited as a result of the pilot were distributed approximately evenly across domains of function rather than tending to be focused in a specific domain. A hierarchical progression was developed to avoid repetitive questions about a skill set beyond a child's developmental abilities, and skip logic based on the child's age (,6 years old, 6-11 years old, or $12 years old) was introduced to target age-appropriate skills. For example, if a parent reported that a child was nonverbal, the parent would not be asked about phrase speech, sentences, or conversations. In these cases in which branching logic excluded questions from the survey, the items were assumed to be negative responses for the purposes of analyses. For questions that addressed age-specific skills and behaviors, different question sets were included. For example, the parent of a 3-year-old would not be asked if the child can "demonstrate sportsmanship." Overall, by the end of the pilot phase, parents indicated that the tool evaluated the relevant key areas and provided perspective on how the child's functioning had changed over time. The final version of the ASD-PROM, with all questions included, consists of 128 items across 4 clinically defined domains: Communication and Social Skills (51 items); Adaptive Skills (3 items); Behavioral Functioning, including repetitive behavior, attention, mood, and maladaptive behavior (53 items); and Sleep (14 items). There was also a section on Overall Improvement (7 items).
The ASD-PROM was then programmed for administration through an online health survey system, TriVox Health (now Veta Health), in use throughout the medical center. This is a web-based disease monitoring and management system originally designed to facilitate the collection of parent-reported information to monitor outcomes of children with developmental behavioral disorders. Core features of this system include (1) automated, prescheduled e-mails to parents, with secure links to online questionnaires; (2) data storage in an Health Insurance Portability and Accountability Act of 1996 (HIPAA)-compliant database; (3) automated result notification e-mails to clinicians including alerts for safety concerns; and (4) natural language summary of results (i.e., a system-generated report template autopopulated with patient data) that can be easily viewed within the electronic medical record system and inserted into clinical notes.
The programming of the ASD-PROM in this system included a disclaimer reminding families that their clinician might not see the information until just before, during, or even after their child's visit, and they should call the doctor (or 911 as appropriate) for more urgent concerns. As a backup to this disclaimer, the ASD-PROM included 16 items that triggered automated alert  (7) 8 (6) Asian Responder education, n (%) 0.28 Did not complete high school Completed high school but not college 0 (0) 14 (23) 123 (22) Other diagnoses, n (%) Intellectual disability -17 (27) 87 (16)  algorithms to notify clinicians of safety concerns, including suicidal thinking, increased aggression, concerns for seizures, and hallucinations. These algorithms involved notifications to the patient's clinician (and the clinician's nursing pool, when applicable) through the electronic health record, in the same queue in which a notification would occur if a parent emailed or called about a medical concern. Previous studies have demonstrated that an embedded trigger algorithm leads to more timely identification of potentially adverse effects and more timely interventions. 32

Phase 1: Evaluation of Clinical Utility and Psychometric Properties
After initial development and piloting of the ASD-PROM and programming in the online health survey system, we initiated a multiphase study to evaluate clinical utility and psychometric properties of the ASD-PROM. For the purposes of this study, we define "clinical utility" as the usefulness of the ASD-PROM in patient care, including its benefit to patients and clinicians. Our institution's institutional review board provided approval for this study. This article will focus on clinical utility and preliminary psychometric properties necessary for determining potential clinical benefit. Future work will focus on research validity of the ASD-PROM, including factor analysis for item validity.
There were 2 groups of participants in phase 1: English-speaking parents of children who had upcoming autism-specialty follow-up visits scheduled (n 5 62) and their clinicians (n 5 19). Only parents of children whose clinicians consented to participating in the study were eligible to be included in phase 1 (48 clinicians certified as autism specialists at our hospital's Autism Spectrum Center were invited to participate in the study, and 33 consented, but only 19 of these ended up having a patient enroll in phase 1. Of these 19, 10 clinicians had more than 1 patient enrolled). Clinician consent was obtained using Research Electronic Data Capture (REDCap) tools hosted at our institution. REDCap is a secure, web-based application designed to support data capture for research studies. 33 Clinicians received training about how to access patient data in the online health survey program.
For recruitment purposes, patients were identified using a query of the hospital data warehouse for upcoming ASD visits, and their ASD diagnosis was verified through the medical record review. Enrollment was monitored to ensure inclusion of patients seen by a variety of specialist clinicians, including developmental behavioral pediatricians, neurologists, and psychologists. Identified families were contacted by mail before their upcoming clinic visit; those who did not return an optout postcard were then contacted through phone. Interested families were then registered in the online health survey system and provided consent for the study electronically.
After online consent (through the online health survey system), parents were asked to complete the ASD-PROM and answer a set of demographic questions. Sixty-two parents completed the initial ASD-PROM.
Test-Retest Reliability Fifty-three of the 62 enrolled parents completed a second ASD-PROM 2 weeks after completing the initial ASD-PROM to assess test-retest reliability.

Convergent Validity
Forty-eight of the 62 enrolled parents completed the Vineland Adaptive Behavior Scales, Second Edition Parent/Caregiver Rating Form (Vineland-II), 25 a parent report measure used to measure adaptive functioning in 4 domains that include Communication, Daily Living, Socialization, and Motor skills. The Vineland-II was mailed to parents, who were asked to complete all items regardless of age. The Vineland-II was used to evaluate convergent validity and the range of adaptive skills among children of participants.  Table 2.
Clinician Postvisit Surveys Clinicians also completed a postvisit survey about their experience with the ASD-PROM. Clinician surveys were completed through REDCap. Clinician surveys included questions about when clinicians reviewed the ASD-PROM results (before, during, or after the visit), and how much time they spent reviewing the results. Using a 1-to 10-point Likert scale (1 5 very poorly, 10 5 very well), the survey also asked about the usability of the ASD-PROM, including clinicians' perspectives on how well the ASD-PROM addressed the clinician's and family's questions at the visit, as well as perceived overall usefulness of the ASD-PROM at that particular visit. Thirtytwo clinician postvisit surveys were completed across 14 Table 2. Parent Perspectives on Usability and Utility of the ASD-PROM for Their Child's Autism Care (n 5 25)

Median (IQR)
How well or poorly did the questions of the ASD-PROM match your areas of concern at this visit? 8 (7-10) How well or poorly was the ASD-PROM survey able to describe your child's current abilities? clinicians. To reduce clinician study burden, each clinician completed up to a maximum of 5 postvisit surveys. These 14 clinicians included 12 medical physicians and 2 psychologists; 13 were female, 1 was male, 5 had completed training 0 to 5 years ago, and 9 had completed training .5 years ago. There were 17 children for whom both the parent and the clinician completed postvisit surveys. Questions from the clinician postvisit survey are listed in Table 3.

Clinician Global Survey
At the end of phase 1, each clinician who had at least 1 patient participate in the study was asked to complete a more comprehensive global survey, regarding their overall experience with the ASD-PROM. The goal of this survey was to obtain a more detailed assessment of clinician perspectives on ASD-PROM utility than the postvisit surveys provided and also to assess clinician perspectives on the ASD-PROM once they had had some time to become accustomed to its use. Thirty-three clinicians (encompassing all those who initially agreed to participate in the study) were sent a clinician global survey about their overall experience with the ASD-PROM at the end of the study period. Of these, 19 had at least 1 patient who had completed an ASD-PROM. Eleven of these 19 clinicians completed the survey, including 9 medical physicians and 2 psychologists; 9 were female, 2 were male, 4 had completed clinical training 0 to 5 years ago, and 7 had completed training .5 years ago. Questions from the clinician global survey are listed in Table 4.  Did not review 0 (0) Generally, when compared with typical established patient visits for patients with autism spectrum disorder when you did not use the ASD-PROM, how much time did you spend on each of the following?
Preparing for this visit overall (e.g., reviewing the medical record) Less time than usual 1 (9) Detecting concerning behaviors or conditions (e.g., depression and seizures) 7 (6-9) Making the visits more patient-centered 8 (6-9) Making me more efficient during the visits 8 (6-9) More efficiency in documentation 9 (5-9) Generally, how well or poorly did the ASD-PROM capture patients' profiles regarding their current abilities in the following domains Overall Improvement 8 (6)(7)(8) Generally, compared with gathering a history in your usual way without the ASD-PROM, how did using the ASD-PROM alter your efficiency during the visit?

Phase 2: Evaluating the Implementation of the ASD-PROM in the Clinic
After the first 20 parent and clinician postvisit surveys were reviewed and showed positive evaluations of the tool, the ASD-PROM was made available for general clinical use across the institution as part of a quality improvement initiative. Clinicians were educated about the ASD-PROM and encouraged, but not required, to incorporate its use in their ASD follow-up visits. To facilitate uptake, parents of patients (n 5 704) seen in specialty ASD clinics for follow-up visits were assigned to complete the ASD-PROM through the institution's online health survey system. A medical record review was conducted to characterize patient demographics for those who completed and those who did not complete the ASD-PROM, as a step toward assessing its feasibility for widescale implementation among a more diverse patient population. Data were collected in a REDCap database.
Analysis We performed all statistical analyses using SPSS (SPSS Inc, Chicago, IL). Continuous variables were summarized using median and interquartile range; categorical variables were summarized using frequencies and percentages. In phase 1, demographic correlations were calculated using Spearman's rho. To evaluate test-retest reliability, Pearson r correlations were calculated between pre (first ASD-PROM) and post (second ASD-PROM) ratings. Convergent validity between ASD-PROM and Vineland scores was measured using Cohen's kappa. In Phase 2, p values comparing completers and noncompleters are calculated using the Mann-Whitney U test for age and x 2 for all other categories. (In these analyses, x 2 does not include unknown/not reported values. Because the ASD-PROM development team was intended to be highly inclusive, with anyone at our institution with appropriate expertise and interest invited to join, some members of the development team also completed clinician surveys. Therefore, Mann-Whitney U tests were used to assess for differences in clinician global survey responses between members and nonmembers of the ASD-PROM development team. Accessibility Ensuring that the ASD-PROM would be open source, freely available, and modifiable by anyone who wished to use it was a high priority to the ASD-PROM development team. Therefore, a PDF of the ASD-PROM is attached to this manuscript as a supplemental file (Supplemental Digital Content 1, http://links.lww.com/JDBP/ A279). The ASD-PROM (with all skip patterns) is also available through REDCap and as part of TriVox Health (now Veta Health) and can be integrated into other systems in the future as needed.

Phase 1
The goal of phase 1 was to evaluate the clinical utility and psychometric properties of the Autism Spectrum Disorder Parent Report for Outcome Monitoring (ASD-PROM).
Demographics of the children are described in Table 1. The median age of children was 10 years (interquartile range [IQR]: 7.5-12.7, range 2-20). Vineland Adaptive Behavior Scales, Second Edition (Vineland-II) scores indicated a range of developmental function (Table 1).
Test-Retest Reliability Supplemental Figure 3 (Supplemental Digital Content 2, http://links.lww.com/JDBP/A280) provides scatterplots depicting the concordance rating between the 2 administrations in the 4 ASD-PROM functional domains: Communication and Social Skills, Adaptive Skills, Behavioral Functioning, and Sleep. Pearson r correlations between pre and post ratings were 0.95 overall (ranging between 0.81 and 0.97 for individual domains). These are all significantly different from zero, suggesting high consistency across ratings by parents on the ASD-PROM over time. Results indicated that the bias in the estimation of the correlation coefficients ranged between 20. 02 and 0.001, which is negligible. Thus, estimates of reliability are likely robust with the sample size of n 5 53 participants.
Because a high covariance between measurements at 2 time points could emerge when differences in the level (i.e., mean) are present, a series of paired samples t tests were conducted to verify that no change in the level was observed across time points. Results indicated that there was no significant difference in means between the 2 points across all 4 domains (p values ranged between 0.17 and 0.78), pointing again to high consistency in scores over time.  Table 2). There was no significant correlation between parent-reported usefulness of the ASD-PROM on care and patient age (n 5 25) or composite score on the Vineland (n 5 14).

Clinician Postvisit Survey
In surveys (n 5 32) completed after visits with a study participant, clinicians found the ASD-PROM to be useful for clinical care overall (median 5 7.0, IQR: 6.0-8.0; Table 3). There was no significant correlation between patient age and clinician-reported usefulness of the ASD-PROM. Clinicians spent a mode of 1 to 5 minutes reviewing the ASD-PROM. When looking at clinicians who reported reviewing the ASD-PROM before more than 1 clinic visit, time spent reviewing the ASD-PROM decreased across sequential visits for which a clinician completed a visit survey and reported reviewing the ASD-PROM (Spearman's rho; r s 5 20.43; p 5 0.02). By the second visit, no clinician reported spending more than 6 to 10 minutes reviewing the ASD-PROM. By the fourth visit, no clinician reported spending more than 1 to 5 minutes reviewing the ASD-PROM. Histograms demonstrating individual clinician responses to postvisit surveys are provided in Supplemental Figure 4 (Supplemental Digital Content 2, http://links.lww. com/JDBP/A280).
Clinician Global Survey Clinicians (n 5 11) indicated the ASD-PROM was effective in assessing parent-reported patient abilities (median 5 9.0, IQR: 7.0-9.0) and increasing patient-centered and efficient care (median 5 8.0, IQR: 6.0-9.0 for both). Overall, 55% of clinicians reported spending increased time counseling families and 91% felt the ASD-PROM either did not increase or shortened the length of the visit (Table 4). Five clinicians who completed the survey were also part of the ASD-PROM development team; Mann-Whitney U tests revealed no significant differences between members and nonmembers of the ASD-PROM development team for any of the questions on the Clinician Global Survey.

Phase 2
The goal of phase 2 was to evaluate the implementation of the ASD-PROM in clinical practice, particularly regarding completion rates.

ASD-PROM Completion
To assess completion rates with clinical use of the ASD-PROM, we evaluated ASD-PROM completion rates among 704 parents of individual patients who were sent the ASD-PROM through the online health monitoring system over a 2-year period beginning in January 2016, as part of standard clinical care. Among this group, 20% (n 5 142) completed at least 1 ASD-PROM survey. An additional 5% (n 5 35) started a survey but never completed it. Parent and child demographics for parents who did and did not complete surveys are in Table 1. In general, the median age of children whose parent completed the ASD-PROM was slightly younger than the median age of children whose parents did not complete the ASD-PROM. Parents who completed the ASD-PROM were also more likely to speak English as a primary language in the home than parents of noncompleters. There was no difference in completion rates based on a child's sex, race, ethnicity, maternal education, primary insurance type, or whether the visit billing diagnosis included comorbid intellectual disability and/or global developmental delay.

DISCUSSION
The Autism Spectrum Disorder Parent Report for Outcome Monitoring (ASD-PROM) was developed as part of a process improvement initiative, with the aim of creating a standardized tool to monitor developmental skills and behavioral function with a strength-focused approach. We evaluated the ASD-PROM among children aged 2 to 20 years and found that it was rated as useful by both parents of children with ASD and their autismspecialty clinicians. Preliminary psychometrics indicate good test-retest reliability. There was a significant positive correlation with the Vineland-II, with a large effect size for the relationship between the ASD-PROM Communication and Social Skills domain and the Vineland Communication domain and a medium effect size for the relationship between the largest subscales on the ASD-PROM (Communication and Social Skills; Behavior) and the Vineland Adaptive Behavior Scales, Second Edition (Vineland-II) (Communication; Daily Living; and Social Skills). Parents felt the tool could accurately describe their child's functioning and was clinically quite useful, and the questions were relevant to their areas of concern. When the ASD-PROM was used, clinicians reported spending more time counseling families without adding extra time to the overall visit. This finding, in the context of parents reporting good utility of the ASD-PROM and good matching of ASD-PROM questions to their areas of concern, may indicate that the ASD-PROM allowed visits to be more focused on patient/family priorities. It was outside the scope of this study to determine whether increased counseling time led to measurable clinical benefits. Future studies could evaluate these impacts on patient-clinician communication and quality of care further.
Importantly, the ASD-PROM is not intended to replace more in-depth, well-validated measures targeting specific developmental, behavioral, or medical domains. It is also not intended to replace the judgment of an expert clinician in taking a history that addresses a parent's primary concerns about their child or addressing the highly individualized and nuanced aspects of a concern that may determine treatment or referral decisions. Instead, it aims to streamline the substantial portion of a visit that would otherwise be spent gathering and documenting information on what we describe in the introduction as "secondary concerns." These are concerns the family has indeed noticed but may not mention during the visit (either because they are forgotten or because they are not at the forefront of a family's list of concerns) but that are necessary for assessing a child's abilities across a variety of domains, and that might substantially alter the clinician's plan of care. Responses on the clinician surveys largely reflect this goal. For example, clinician survey items addressing how well the ASD-PROM addresses family and clinician questions for the visit received only moderate scores (median 7 of 10), whereas items addressing how well the ASD-PROM captured patient abilities, reminded parents to ask questions they otherwise would not have thought of, made the visits more patient-centered, and improved efficiency scored higher (medians 8-9 of 10; parent survey n 5 25). In addition, individual clinician responses to postvisit surveys were often variable across surveys, suggesting that clinicians found that the ASD-PROM varied among patient visits in terms of how well it addressed the family's and clinician's questions and in terms of overall usefulness. Further use and future studies may be helpful in assessing patterns of clinical utility and thus determining the patient population for whom the ASD-PROM is most useful.
We implemented the ASD-PROM in a web-based system with the goal of allowing families to complete the ASD-PROM remotely, allowing clinicians to review the results at a time that is convenient for them, and allowing documentation of parent-reported responses in a format suitable for inclusion in clinical notes. Previous studies have suggested that transitioning from paper to electronic health records initially increases documentation time but then may ultimately improve workflow as clinicians become more familiar with a new system. 34 This is consistent with our finding that clinician time spent reviewing the ASD-PROM tended to decrease over the first few visits, as clinicians presumably became more accustomed to (and thus efficient at) reviewing the ASD-PROM. Alternatively, it is possible that clinician time to review ASD-PROM decreased for other reasons (e.g., the clinician became bored of reviewing the ASD-PROM or did not find a more prolonged review to be useful); this could be further evaluated with targeted questions asking clinicians about this in future studies. Regarding documentation efficiency, it is notable that the clinicians who found the ASD-PROM to have the most positive effect on documentation efficiency were the same clinicians who used the autopopulated ASD-PROM report template in their clinical notes most often. This could be because other clinicians did not know about the ability to autopopulate the template, and additional education would thus likely improve clinician efficiency. Alternatively, it is possible that clinicians were aware of the template but had not yet developed a workflow for using the template for documentation purposes or preferred to summarize this in their own way.
Low completion rates were a concern not only in phase 2 (clinical use) but also in phase 1 (evaluation of clinical utility). In our study, clinician response rates were typically greater than 50%, which is within the historical range of clinician survey response rates (30%-50%). 35 Parent response rates tended to dwindle over the course of the study, with each subsequent measure requested (second ASD-PROM, Vineland-II, and postvisit survey) having a progressively lower response rate. This may reflect the burden of multiple study measures distributed across multiple days. It is worth considering the possibility that this response bias led to inflated scores on satisfaction surveys (i.e., with those parents and clinicians least satisfied with the ASD-PROM being more likely to drop out). On the other hand, in clinical use, there is no requirement that parents and clinicians use the ASD-PROM if they dislike it; therefore, those who choose to use the ASD-PROM may (as expected) be those who find it to provide the most benefit.
One potential limitation to the abovementioned findings is that because we initially tested the ASD-PROM in a specialty care clinic for patients with ASD, results may not generalize to all patients with ASD and their families. Participation bias may have also influenced the response to postvisit clinical utility survey questions. Review of study participants' demographics suggests that they were more likely to be White, non-Hispanic and to have private insurance than the broader clinic population. This may have been due in part to the limitation to parents who were English speaking because the ASD-PROM is currently in English alone. Future directions include translating the ASD-PROM into other languages and evaluating the utility of the ASD-PROM in a more diverse patient population. Response bias may have influenced which participants and clinicians chose to answer the postvisit and global survey questions. Finally, when the ASD-PROM was released for general clinical use, the overall response rate was only 20%. Efforts to educate families about the potential benefits of completing the ASD-PROM and clinical supports such as automated completion reminders might increase participation and engagement.
The primary barrier to the use of the ASD-PROM in a clinical setting is likely to be the low completion rate by parents. In our sample of 704 families sent the ASD-PROM for clinical purposes, 75% of parents did not open the survey. This is similar to completion rates for other electronic surveys. 16 The low completion rate may be a result of several factors, including individual family or clinician characteristics. Families without athome computer/Internet or smartphone access may also have difficulty completing the ASD-PROM. 32 In some cases, clinicians may not have provided explicit enough instruction to families about how to access the ASD-PROM and its relevance to their child's care. Survey system barriers may have also played a role in the low completion rate, such as the requirement for new users to create a secured account within the survey system before accessing the ASD-PROM. The survey system also sent surveys as being from "TriVox Health" rather than from the clinician; future changes that list the clinician as the sender might increase completion rates. In addition, 35 parents (5% of those who received a clinical survey) opened the survey but did not complete it. Although this could occur for multiple reasons, this finding raises concern about whether the length of the ASD-PROM is a barrier to completion. In the future, creating a shortened version of the ASD-PROM using statistically guided techniques will improve utility and feasibility for families to complete the ASD-PROM on a regular basis, without sacrificing information most relevant to a child's medical and developmental care. However, the rates of parents opening the survey are much lower than the rates of parents opening the survey but not completing it. This suggests that over and above concerns regarding survey length, barriers to accessing the survey itself may need to be addressed in future research or quality improvement initiatives. Particularly among families caring for a child with ASD, competing demands may lead to lack of time for survey completion.
Importantly, the quality improvement initiative outlined in the current study is only a first step toward maximizing the potential clinical benefits of the ASD-PROM. A series of stepwise improvements and further analyses are necessary. To begin, by introducing the ASD-PROM here as an open-source tool, we aim to provide opportunities for clinicians, researchers, and quality improvement specialists from a variety of institutions and subspecialties to consider how its use might be better adapted to the spectrum of concerns that most frequently occur among children with ASD. This may include adding additional questions, removing questions for which the benefit is not sufficient to outweigh the additional time burden that question adds, and/or creating a short form with the goal of decreasing survey fatigue and thus increasing completion rates. It may also involve regrouping the questions in a manner that is statistically guided while still clinically informed, using additional psychometric analyses. Such regrouping will also add for additional validation of the numerous domains currently addressed in the ASD-PROM (beyond the first-pass test of convergent validity with the Vineland-II that is presented here). Focused assessment of the performance of the ASD-PROM for specific age or developmental ranges, with consideration of adjustments that specifically capture norms for these ranges, would also be of benefit. In addition, ongoing use will allow the assessment of how well the ASD-PROM captures or predicts changes that occur longitudinally across development.
In summary, the ASD-PROM has the potential to address a critical gap in the clinical care for children with ASD. It allows the collection of qualitative data from families of children with ASD across a variety of functional domains and levels, resulting in improved clinical care from the standpoint of parents and clinicians. It is freely available for use and modification as needed in the clinical setting. For the ASD-PROM to ultimately achieve its potential as a tool that makes inperson visits with clinicians more efficient and responsive to parents' needs, additional work is needed to validate its subscales with rigorous psychometrics, validate its utility as a measure of prospective outcomes that change over time, and evaluate its feasibility in expanded clinical, research, and quality improvement settings.