{'fraction_non_alphanumeric': 0.028412712867680538, 'fraction_numerical': 0.01967033967762499, 'mean_word_length': 5.057363485934914, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Readability assessment of American Shoulder and Elbow Surgeons patient brochures with suggestions for improvement

Background Many Americans have limited literacy skills, and the National Institutes of Health (NIH) suggests patient educational material be written below the 8th grade level. Many orthopedic organizations provide print material for patients, but whether these documents are written at an appropriate reading level is not clear. This study assessed the readability of patient education brochures provided by the American Shoulder and Elbow Surgeons (ASES). Materials and Methods In May 2017, 6 ASES patient education brochures were analyzed using readability software. The reading level was calculated for each brochure using 9 different tests. The mean reading level for each article was compared with the NIH-recommended 8th grade level using 2-tailed, 1-sample t tests assuming unequal variances. Results For each of the 9 tests, the mean reading level was higher than the NIH-recommended 8th grade (test, grade level): Automated Readability Index, 14.1 (P < .05); Coleman-Liau, 14.2 (P < .05); New Dale-Chall, 13.2 (P < .05); Flesch-Kincaid, 13.7 (P < .05); FORCAST, 11.8 (P < .05); Fry, 15.8 (P < .05); Gunning Fog, 16.5 (P < .05); Raygor Estimate, 15.4 (P < .05); and Simple Measure of Gobbledygook (SMOG), 15.1 (P < .05). Conclusions The ASES patient education brochures are written well above the NIH-recommended 8th grade reading level. These findings are similar to other investigations concerning orthopedic patient education material. Supplementary brochures and websites could be a useful source of information, particularly for patients who are deterred from asking questions in the office. Printed material designed for patient education should be edited to a more reasonable reading level. Further review of patient education materials is warranted.

1. Arthritis and Total Shoulder Replacement 2. Arthroscopy of the Shoulder and Elbow 3. Rehabilitation of the Shoulder 4. Rotator Cuff Tendonitis and Tears 5. Tennis Elbow 6. The Unstable Shoulder All brochures were written in English. The content of each was copied as plain text into an individual Microsoft Word document (Microsoft Corp., Redmond, WA, USA). In accordance with the readability software guidelines, figures, figure legends, copyright notices, disclaimers, acknowledgements, citations, and references in the brochures were excluded.
The Word documents were analyzed using Readability Studio Professional Edition 2015 (Oleander Software, Ltd., Vandalia, OH, USA). Within the program, 9 different readability scores were calculated for each article. These formulas were: Each of these tests has been used for analyzing patient education materials. 14,27,29,36 The tests are based on the sample text's syllables, words, and sentences (Table I), but each formula weights these components differently. Some formulas have additional components. The Dale-Chall formula contains a factor of "unfamiliar words," which Dale and Chall chose based on assessment of 4th grade Americans. 4 The Fry and Raygor reading levels are determined by plotting the calculated score to an accompanying readability graph. This graphing was performed automatically by the readability software. The Flesch-Kincaid Grade level, 20 SMOG, 18 Fog, 18 Fry, 10 and Dale-Chall 4 tests have been individually validated.
The more commonly used readability tests, the Flesch-Kincaid and Automated Readability Index, were originally developed for the United States Navy in 1975 for analyzing technical manuals; the Flesch-Kincaid is now the United States Military Standard. 16 The Fog Index was validated using McCall-Crabbs' Test Lessons in reading and was developed to predict the grade level of individuals who could correctly answer 90% of the passage's comprehension questions. 18 The SMOG index was also validated using Test Lessons in reading, except it was based on answering 100% of the questions correctly, so the grade levels predicted by SMOG are sometimes higher. 9 Although there is no gold standard, a review by Friedman and Hoffman-Goetz 9 noted that many of these formulas correlate strongly with each other, and using more than one to increase validity is recommended. 1 Additional information regarding commonly used readability tests is presented in a thorough review of the topic by Badarudeen and Sabarwhal. 1 Reading level was calculated for each article by averaging the 9 readability scores. The mean reading level for each article was compared with an 8th grade level using 2-tailed, 1-sample t tests assuming unequal variances. Normality of the sample distribution was tested with the Shapiro-Wilk test. All statistical tests were performed using R 3.4.0 software 28 (R Foundation for Statistical Computing, Vienna, Austria) with RStudio 1.0.153 software 30 (RStudio Inc., Boston, MA, USA). An α level of 0.05 was determined.
Recommended reading levels range from 1 to 5 grades below the intended audience. 3,22,23 Most Americans have completed 10 to 13 years of schooling, 17 so the 8th grade was chosen as a conservative measure in accordance with the NIH recommendations. 34

Results
The results confirmed the hypothesis that the reading level of the brochures would be higher than the 8th grade. The readability scores of all 6 brochures were calculated with each of the 9 readability formulas, except for the "Rehabilitation of the Shoulder" brochure. The Fry and Raygor scores could not be calculated for this brochure because it contained too many words with high syllable and character counts.

Discussion
A significant portion of the adult population in America has limited literacy skills, 38 and the NIH recommends that patient materials be written at or below the 8th grade level. 34 The purpose of this study was to determine whether the ASES patient education brochures meet the NIH recommendations. The results of the study demonstrate that the brochures are written well above the recommended 8th grade level and range in difficulty from a grade level of 13.4 to 15.3. This is at college level and is likely higher than the average patient's reading comprehension. For clinicians that supply these brochures, it may be useful to go over the material with patients during their office visit.
This study does have several limitations. The reading grade formulas do not score visual information. Each of the patient brochures included images that could reduce the reading level in practice. In addition, the formulas are calculated based on syllable and character counts per word, sentence, and paragraph. This is not an ideal method for capturing language complexity, particularly in medical literature. For example, the word "individual" has more syllables and characters than the word "septic," even though patients are less likely to understand the latter. However, this study used several validated measures of readability, and 100% of the tests demonstrated that the reading level was higher than the 8th grade for each article.
Other studies of orthopedic patient information have found similar results, with reading grade scores ranging from 8.8 to 11.5. Most patients read at a level 4 grades below the highest grade completed. 26 More than one-third of patients presenting in an emergency department setting have inadequate or marginal health literacy and cannot understand discharge, follow-up, or medication instructions. 39 The National Center for Education Statistics  conducted a detailed report on adult literacy, which found surprising results. Nearly one-quarter of all Americans are in the lowest literacy proficiency level, and most of these individuals do not see themselves as "at risk" for low literacy. 17 Therefore, asking patients whether they understand their diagnosis or probing patients for questions is not likely to identify patients with low proficiency. Patients with low literacy often feel shameful and are less likely to ask questions or admit when they do not understand something, 15,21,24 so effective print material can be valuable. In support of the value provided by print material, a recent study by Madkouri et al 19 found that a presurgical information sheet was more informative than oral information given by the surgeon.
The results of readability studies have suggested several areas for improving orthopedic material for patients. The breadth of the problem is unclear. A recent study of common orthopedic patientreported outcomes by Perez et al 25 surprisingly found that more than 90% were at or below the 8th grade level, and all of the recently developed orthopedic-related Patient-Reported Outcomes Measurement Information System (PROMIS) measures were at or below the more stringent 6th grade level. However, a study by Roberts et al 29 compared American Academy of Orthopaedic Surgeons readability scores from 2008 to 2014 and found that the mean reading level decreased from 10.4 to 9.3. Approximately 84% of the articles were still written above the 8th grade reading level, indicating that little progress has been made regarding educational material.
There are several ways to continue improving written communication with patients. Some readability software programs provide a list of commonly used words with suggestions for replacement (Table III). The United States Department of Health and Human Services recommends using short sentences, defining technical terms, and supplementing instructions with visual material. Studies have shown that individuals with limited literacy skills prefer information with short words and sentences. 37 In addition, information should be organized so that the most important points are repeated and stand out. A test for assessing patient comprehension of oral communication was validated in 2015 by Giudici et al 12 in a group of 21 patients before surgery. For improving oral communication, patients could be asked open-ended questions; further, the teach-back method is effective for evaluating a patient's understanding. 13

Conclusion
The ASES patient education brochures are written well above the NIH recommended 8th grade reading level. Discussing the brochure material with patients while they are in the office may be useful. The findings of this study are similar to other investigations concerning orthopedic patient education material. Supplementary brochures and websites could be a useful source of information, particularly for patients who are deterred from asking questions in the office. Printed material designed for patient education should be edited to a more reasonable reading level. Further review of patient education materials is warranted, and future studies should be done to assess whether progress has been made.

Disclaimer
The authors, their immediate families, and any research foundations with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.