{'fraction_non_alphanumeric': 0.031587473002159826, 'fraction_numerical': 0.018898488120950324, 'mean_word_length': 5.097530864197531, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Quality and readability of online information on idiopathic subglottic stenosis

Abstract Objective Idiopathic subglottic stenosis (ISS) is a chronic condition characterized by disease recurrence and multiple surgeries. These frustrated patients may utilize the internet to research their condition. The aim of this study was to determine the quality and readability of online ISS information. Methods “Idiopathic subglottic stenosis” was entered into Google. The first 50 websites that met inclusion criteria were extracted. The DISCERN instrument, Flesch Reading Ease Score (FRES), and Flesch‐Kincaid Grade Level (FKGL) assessed the quality and readability, respectively. Means, SDs, Pearson correlation coefficients, and two‐tailed Student's t‐test were calculated. Results The 50 websites consisted of 17 patient‐targeted and 33 professional‐targeted websites, plus 30 major and 20 minor websites. The overall DISCERN, FRES, and FKGL scores were 2.81 ± 0.99, 27.75 ± 15.27, and 13.65 ± 2.79, respectively (mean ± SD). Patient‐targeted websites had significantly lower quality (DISCERN [P < .00]) but were easier to read (lower FKGL [P < .00], higher FRES [P < .00]) than professional‐targeted websites. Minor websites had a significantly lower quality (DISCERN [P < 0.00]) but were easier to read (lower FKGL [P < .00], higher FRES [P < .00]) than major websites. There was a positive correlation between overall quality and difficulty in readability. Conclusion The quality of online ISS information was suboptimal. Resources were too difficult to comprehend and readability scores were above AMA and NIH recommendations. Improved online information is required to properly educate this patient population. Level of Evidence Level 4.


| INTRODUCTION
Idiopathic subglottic stenosis (ISS) is a rare disease that comprises 20% of subglottic stenosis cases and affects 1:400 000 persons per year. 1,2 ISS tends to affect perimenopausal Caucasian women, however, its etiology is currently unknown. [3][4][5] The leading hypothesis depicts ISS as aberrant mucosal inflammation secondary to a triggering event. 6 These triggers could include abnormal estrogen and progesterone signaling, gastroesophageal reflux disease, severe coughing, and mycobacterium species. [6][7][8][9] As the etiology of ISS is unknown, treatments only address the stenosis and not the source of the disease. This result in disease recurrence and multiple surgeries. 10 Since ISS is a chronic and frustrating condition, these patients often do their own research. Patients may consult the internet to better understand their health condition, and for some conditions such as breast cancer, this has significantly increased over time. 11,12 In 2013, 35% of U.S. adults indicated that they have utilized the internet to diagnose themselves. 11 In 2018, 80% of people on social media platforms used them to gather health information. 13 Unfortunately, online resources tend to be of low quality and are written at an advanced readability level. [14][15][16][17] Readability is defined as the reading level a person must have to comprehend written information. 18 With 43 million Americans having low literacy levels, it is essential to follow the readability recommendations of the American Medical Association (AMA) and the National Institutes of Health (NIH) to write health information at a fourth to sixth grade level. 19,20 An appropriate readability level is required for patients to improve health literacy, which is defined as having the ability to read, understand, and communicate important medical and health information. 21 Health literacy is a stronger predictor of health outcomes than a person's age, race, income, and education level. 22 Previous research has shown that low health literacy can lead to suboptimal adherence to preoperative medications and poor comprehension of discharge instructions. 23 Unfortunately, low health literacy has been documented in otolaryngology, as over one-third of patients undergoing a total laryngectomy met the criteria for low health literacy. 24 ISS is a chronic, relapsing condition, where patients have the luxury of time to do their own research. Consequently, the goal of this study was to evaluate the quality and readability of online patient education information on ISS. The hypothesis was that websites are written at a higher reading level than recommended and that the quality of information is suboptimal.

| MATERIALS AND METHODS
This project was exempt by the Research Ethics Board of the University of British Columbia because the study relies exclusively on information that is publicly available. 25 An unconstrained Google search of "idiopathic subglottic stenosis" was done to simulate a patient search. 26 This search was conducted on June 17, 2020 in Toronto, Canada using a non-academic Wi-Fi server to represent patients without academic credentials. The first 50 websites were included from the preliminary search. Websites were excluded and replaced if they were irrelevant sites, duplicate sites, broken links, advertisements, non-text media (eg, videos and PowerPoint presentations), or focused on pediatric subglottic stenosis. Websites not written in English were excluded. The top 50 websites that met inclusion criteria were categorized as patient targeted vs professional targeted, and major vs minor. Patient-targeted websites are written in layman's terms to provide patients with education and support. These included clinic pamphlets, patient stories, blogs, and so forth. Professional-targeted websites are written to communicate with professionals and included peer-reviewed journal articles, treatment guidelines, and medical specialty-specific newsletters. When peer-reviewed journal article abstracts were encountered but the full-articles required payment for access, only the abstract was analyzed to better replicate a patient search. Major websites included resources from established academic institutions, articles published in peer-reviewed journals, and from large national organizations. Minor websites included all remaining websites such as blogs and social media posts. Previous literature has shown that 95% of traffic on Google is on the first page of 10 hits. 27 As a result, a subgroup analysis comparing the top 10 hits with the total 50 hits was conducted.
This data collection, categorization, and subsequent quality and readability evaluation and data analysis were completed by a medical student at the University of British Columbia. This medical student was trained with the DISCERN instrument through the use of general instructions, a quick reference guide, and a list of terms. 28 The medical

| Readability assessment: FRES and FKGL
Readability was assessed using the Flesch Reading Ease Score (FRES), and the Flesch-Kincaid Grade Level (FKGL) score. These readability tests assigned their scores using the number of syllables in words and the number of words in sentences. 36 The FRES is an index that rates the text on a 100-point scale, where a higher score indicates that the text is easier to understand. 36 In contrast, the FKGL rates the text using U.S. grade levels, such that a higher score indicates that the text is more difficult to read. 36 These scores were calculated by copying the website text (omitting references) into Microsoft Word 2019 for Mac and using the spelling and grammar tool. A criticism of the FRES and FKGL scores is that both can be inflated by a medical term with many syllables. 37 To address this concern, "Idiopathic subglottic stenosis" was replaced in the text by ISS   and scores from the first 10 websites vs all 50 were compared using a two-tailed unpaired Students' t-test assuming unequal variances. An a priori significance level (P < .05) was used.

| RESULTS
The search yielded about 123 000 results. The first 50 websites were included, but upon further review, seven were excluded due to non-text media, broken links, irrelevance, and website duplication. These websites were replaced by the next websites in the original search list.  Figure 1). This indicates that search results were of low quality and required high reading levels. To ensure that the readability scores were not inflated by a medical term with many syllables, 37  and a significantly lower FRES value (P = 1.12 Â 10 À5 ; Table 3, Figure 1).
In subgroup analysis, the first 10 websites were analyzed and These scores were not significantly different from scores for all 50 websites (Table 5).
Titles and hyperlinks for the 50 included websites are listed in Table 6.

| DISCUSSION
With the advent of the internet came an immense amount of information that is accessible to patients diagnosed with various diseases, including rare conditions such as ISS. The internet, however, is not F I G U R E 1 A, DISCERN website quality scores for 10 websites, 50 websites, and all patient-targeted, professional-targeted, major and minor websites. B, FRES readability scores for websites with and without the ISS term, and for 10 websites, 50 websites, and all patient-targeted, professional-targeted, major and minor websites. C, FKGL readability scores for websites with and without the ISS term, and for 10 websites, 50 websites, and all patient-targeted, professional-targeted, major and minor websites. FRES: Flesch Reading Ease Score; ISS, Idiopathic subglottic stenosis. *P < .05 when comparing patient targeted to professional targeted websites. **P < .05 when comparing major to minor websites completely peer-reviewed by medical experts and no one regulates the information available. This study is the first to systematically evaluate the quality and readability of online information for ISS. Our results confirmed the hypothesis that online information on ISS is written at a higher reading level (college freshman level) than is recommended for the average patient and that the quality of information is suboptimal.
In terms of readability, the FRES and FKGL scores for these ISS websites did not meet the AMA or NIH readability recommendations of 60 to 70, and fourth to sixth grade levels respectively. 17,19,36,38 Included in these scores were patient education materials from hospitals and academic institutions which also exceeded the recommendations. 17 This study's results are supported by a cross-sectional analysis of 502 major articles from top U.S. children's hospitals, pediatric otolaryngology fellowships and the American Academy of Otolaryngology-Head and Neck Surgery. This analysis calculated a mean readability grade level of the 10th grade, which was lower than our mean FKGL, but still exceeded the AMA and NIH recommendations. 17 One of the better online resources for ISS was the North American National Airway Collaborative "ISS-A Rough Guide for Beginners" which was of moderate quality and had a seventh grade readability level. 39 These readability concerns may be less of an issue for the ISS community as a recent study in JAMA determined that 88% of ISS patients had at least some college education. 3 However, the quality of these ISS websites remains a concern. The mean quality of all included websites was determined to be low to moderate (2.81 ± 0.99) by the DISCERN instrument. This is in agreement with multiple studies that used DISCERN to evaluate the quality of websites focused on thyroplasty, vocal fold nodules, in-office vocal fold injections, and the treatment of swallowing disorders. [14][15][16]40 When the readability and quality scores were correlated, a weak positive correlation with FKGL and a weak negative correlation with FRES emerged. This means that as the reading difficulty of a website decreased, its quality decreased as well. Minor websites were significantly easier to read and of significantly lower quality than major websites. Likewise, patient-targeted websites were significantly easier to read and of significantly lower quality than professional-targeted websites. Ensuring that patient education information is both easy to read and of high quality can be challenging. DISCERN has been evaluated against several other tools [51][52][53] and was found to be easy to use and widely accepted in other medical specialities. 52 Three previous peer-reviewed publications of the senior author also used these three instruments. [14][15][16] Google was chosen as the search engine as it owns about 91% of the search engine market share. 54  authors as the tool was designed to provide objectivity to quality ratings such that a non-expert may be able to assess online health information. This methodology was also used in three peer-reviewed manuscripts published by the senior author and in publications on various medical topics including second trimester ultrasound examination, gender-affirming hormone treatments, and rhinoplasty. [14][15][16][62][63][64] However, to reduce rating bias, it may have been more ideal to have two independent reviewers extract data with the DISCERN tool. The assessor was not blinded to the category of the website, which could introduce bias in the DISCERN assessment. DISCERN cannot be used to assess the scientific quality or accuracy of a publication because this would involve fact checking against other sources. 28 It is also unable to score audiovisual materials. DISCERN was created to assess treatment choices. 28 Our search term, ISS, was not treatment focused; instead, it searched for all information relevant to ISS. Therefore, the DISCERN score may be low due to Section 2 of the instrument being non-applicable. Similarly, the FKGL and FRES readability assessments have limitations. For instance, they omit communication factors, like audiovisuals, that contribute to clarity and comprehension. 46 This study excluded non-English websites. To accommodate our multilingual patient population, material in different languages should be considered. Google Translate can also be used to translate websites into different languages.
These limitations should be addressed in future studies.

| CONCLUSION
ISS is a chronic, relapsing condition that is frustrating for patients. As a result, patients may be motivated to research their health condition on the Internet. Unfortunately, the Internet is not completely peer reviewed by medical experts and the information is not regulated. The available online information on ISS was of low quality and written at a reading level that surpasses the average patient. When patients look for higher quality information, they are unlikely to understand it because reading level increases with quality. It is essential that otolaryngologists inform their patients of this insufficiency and advocate for improved online patient education resources.

FUNDING AND CONFLICT OF INTEREST
The authors have no funding or conflicts of interest to disclose.