Article Text


  1. Osman H Ahmed1,
  2. S John Sullivan1,
  3. Anthony G Schneiders1,
  4. Paul R McCrory2
  1. 1Centre for Physiotherapy Research, School of Physiotherapy, University of Otago, Dunedin, New Zealand
  1. Correspondence to Professor S John Sullivan, Centre for Physiotherapy Research, University of Otago, PO Box 56, Dunedin 9054, New Zealand; sjohn.sullivan{at}






View Full Text

Statistics from

Recent advances in concussion knowledge1 ,2 have been widely disseminated and referenced in the scientific literature.3 Recommendations from the 2008 Concussion in Sport (CIS) statement2 highlight the importance of knowledge transfer,4 which is defined in the health setting as 'the process of exposing new knowledge to the frontline healthcare provider'.5 Traditional methods of transferring knowledge face-to-face and through leaflet-based approaches6,,8 are now being supplemented by websites which provide concussion information and advice.9 ,10

A major factor in the shift to online knowledge transfer is the continuing expansion of the internet into everyday life, with recent studies suggesting as many as 61% of American adults regularly look online for health information.11 The emerging role, and growing acceptance, of the internet in sports medicine has led to the creation of numerous websites which offer advice on injury management,12 ,13 injury prevention14 ,15 and training/rehabilitation programmes.16 ,17 Websites for many medical conditions18,,20 have been evaluated and critiqued in the literature, however to date, there are relatively few studies that have examined the overall quality of sports medicine websites.21 A recent study by Starman et al (2010)22 demonstrated 'a poor overall quality' of orthopaedic sports medicine websites, and recommended patient access and use for only well-known websites from reputable sources.

Online health information has been shown to contain inaccuracies and inconsistencies,23 ,24 indicating a need to evaluate the standard of the information provided.25 ,26 Three factors crucial to the integrity of a website are the information quality, the content of the information and its readability. The widely-used benchmark for information quality on health websites is the HONcode,27 created and maintained by the Health on the Net (HON) Foundation to provide a recognised ethical standard. Web publishers can apply to have the HON Foundation evaluate their website, and websites that comply with the HONcode principles are permitted to display the HONcode validation certificate.28 ,29 Currently, there is no generic tool for evaluating the content of all health websites. Several condition-specific instruments have been created,18 ,19 however, there is no tool which specifically assesses online brain injury or concussion information. The third identified factor relevant to the integrity of a website is its readability, and this is commonly assessed using readability utilities such as the Flesch Reading Ease Score (FRES) and Flesch–Kincaid Grade Level (FKGL).30 These tools have been primarily utilised for the printed word (including brain injury information leaflets),31 and have also been used to assess the readability of healthcare websites.32 ,33

Numerous organisations, both government and private, provide websites which deliver educational advice and information relating to concussive brain injury.34,,37 The importance of ensuring the provision of accurate online information is not disputed, however to date, there has been no investigation of the standard of this information in the field of concussion education and management. The purpose of this study was to evaluate the information quality, content and readability of a range of selected websites providing information on/related to concussion or sports concussion in particular.


A four-stage sequential strategy was undertaken, incorporating both a rigorous non-exhaustive systematic search and expert knowledge to ensure a comprehensive list of relevant websites for examination. This study utilised information that is freely available in the public domain, and ethical approval was not required or sought.

Search strategy

Stage 1: Google Search

The World Wide Web was systematically searched for websites relating to concussion using the search terms 'concussion' and 'sports concussion' in order to identify the websites that members of the public would most typically locate. The search was undertaken on 6 September 2010 using Google, the most-used search engine on the web.38 To improve the sensitivity of the search, Google sites from major English-speaking countries were included: (USA), (UK), (New Zealand), (Canada), (South Africa) and (Australia). For pragmatic reasons, the first 10 hits identified from each of these Google websites (ie, those shown on the first search page) were retrieved, combined in a spread sheet and duplicates removed manually.

Stage 2: Expert Knowledge

The websites retrieved from stage 1 ('Google Search') were supplemented by websites suggested by members of the research team in consultation with international colleagues. This included websites that are well-known providers of health information.

Stage 3: International Sport Federations

A sample of websites hosted by international sporting federations including, but not limited to, those federations sponsoring the international CIS meetings were included. The websites of selected international sporting federations whose sports are known to have a high incidence of concussion/head injury were also included.

Stage 4: Industry Sample

Websites which offered concussion-related services, primarily neuropsychological testing services and products, were identified by the research team from their prior knowledge.

Inclusion criteria for all four stages of the search strategy required that each website be in English and freely available to the public (ie, not require any membership or registration). Websites were excluded from the study if they were irrelevant to the topic of concussion/sports concussion, or were commercially sponsored links or news stories. As this study focused on written information only, links to video (eg, YouTube) or audio (eg, podcasts) were excluded.

Information quality (HONcode)

The HONcode is the most established and widely-used instrument for evaluating health information on the internet, and has been awarded to 7300 certified websites on more than 10 million pages in 102 countries.27 The HON Foundation awards the HONcode, a quality standard, to websites which follow the HONcode principles (listed in table 1) with the HONcode logo displayed on the pages of these websites. For the purposes of this study, information quality was designated by the presence of the HONcode logo on the website.

Table 1

HONcode principles*

Content assessment (CONcheck)

Due to the lack of a validated concussion evaluation instrument, a custom-designed 11-point checklist (a concussion information checklist called 'CONcheck') was created by the research team. Face validity of CONcheck was established by the input from a member (PM) of the group who drafted the 2008 CIS Statement.2 CONcheck was developed via a reiterative process of extracting core material from the 2008 CIS Statement,3 and refining this into a series of short statements which reflected the key messages(see table 2). CONcheck was created with a focus towards the general public searching for concussion information. A consequence of this is that some specialised return-to-play information from the 2008 CIS Statement2 targeted towards the management of elite athletes was intentionally omitted from CONcheck.

Table 2

CONcheck items

CONcheck was reviewed and modified by the research team until a final version was agreed upon, with external input sought (from WM, one of the authors of the 200139 and 20041 Summary/Agreement Statements and the 2008 Consensus Statement)2 and a simple descriptive coding scheme developed. Each of the items was scored as follows: 'Present and Complete' (2), 'Present but incomplete' (1), 'Absent' (0) or 'Inaccurate' (*). For each CONcheck item, a website was coded as 'Inaccurate' (*) if there was any information present which was contrary to that item of CONcheck (see examples in table 5). Those items classified as 'Inaccurate' were excluded when calculating the total CONcheck score. These results were tabulated to provide a total score, with the maximum score being 22.

Readability assessment

An essential feature of any website is its ability to communicate to the reader, and a key aspect of this is the capacity to deliver information at a level where it is understood. The FRES and FKGL30 are among the most widely accepted readability tools.31 The FRES uses the length of sentences and the number of polysyllabic words to determine scores from 0 to 100.32 A score of under 30 represents text that is very difficult to read, a standard score is between 61 and 70, while a score of over 91 would indicate that the text is very easy to read.31 The FKGL utilises the mean sentence and word length to calculate reading level. FKGL scores up to grade 12 (final year of high school) correspond to US educational school grades, with scores higher than 12 indicative of college level and domain-specific experts.32

The inbuilt readability statistics feature of Microsoft Word 200732 was used to determine the FRES and FKGL for each website. Approximately 1 A4 page (400–500 words) of text was selected at random from each of the websites identified and analysed.


Prior to the content assessment, the PI (OA) and two research colleagues (SA and HPL) reviewed the CONcheck a priori rules and discussed coding procedures. These three researchers were well informed of the content of the 2008 CIS Statement2 upon which CONcheck was based and each researcher had several years of experience in working with concussed sports persons and in the field of sports concussion research. Pilot testing was undertaken (by the PI) on two websites relating to concussion which did not appear in the first 10 Google hits in stage 1 of the search strategy. In the same 24-h period, (00:00–24:00 7 September 2010) each researcher independently reviewed each website using CONcheck.

Hyperlinks within the body of text that navigated to information within each website were followed and included in the analysis; however, hyperlinks leading to external websites were not followed. It was permitted to follow hyperlinks which navigated to a PDF within the same website, and the information from these PDFs was included for analysis. Once all researchers had completed CONcheck, results were pooled and where differences in the scoring existed, agreement was reached via consensus. It was noted if the information was contained on a PDF within a website, and also if the website made reference to either of the CIS Group statements.1 ,2

Statistical analysis

Descriptive statistics (mean±SD) were computed for all variables (overall CONcheck score, FRE and FKGL) and frequencies were determined for each CONcheck item. Independent t tests were performed to analyse for differences between those websites that obtained the HONcode and those that did not. An a priori level of significance was set at p≤0.05 for all analysis. Descriptive statistics were calculated using SPSS (V.16.0; SPSS, Chicago, Illinois, USA). Where inaccurate information was identified on a website, quotes were extracted to provide examples of this.


Implementation of the search strategy resulted in the identification of 43 websites (figure 1). The majority (30/43 or 70%) of websites were not HON certified (table 3). PDFs containing concussion-related information were found on 10 websites (23%), and five of these PDFs were either the 2008 CIS consensus statement2 or the 2004 CIS summary and agreement statement.1 Of these five websites referring to the 2004/2008 statements, only one was HON certified.

Figure 1

Search strategy and identification of websites.

Table 3

Website details, readability and HONcode

CONcheck items 2 and 3 scored highest (ie, had the highest frequency of '2-Present and Complete' scores) with 74% (N=32) and 72% (N=31) of websites, respectively (table 4). Those items of the CONcheck where websites scored the weakest (ie, had the highest frequency of '0-Absent' scores) were items 4, 9 and 11. Six websites (14%) achieved the maximum CONcheck score (22 out of a possible 22) and seven websites (16%) obtained the lowest possible score (zero). Readability scores ranged from 16.3 to 77.4 (mean=46.95, SD=14.23) for FRE and from 6 to 17.8 (mean=11.17, SD=2.72) for FKGL.

Table 4

CONcheck and readability summary

No significant differences were seen when comparing websites meeting the HONcode criteria (N=13) to those websites not meeting the HONcode criteria (N=30). Websites without the HONcode had slightly higher CONcheck scores than those with the HONcode (12.3±7.9 compared to 10.8±7.2, t41=0.571, p=0.571), and were also marginally easier to read as determined by the FRE (48.1±15.3 without HONcode, 44.3±11.5 with HONcode, t41=0.808, p=0.424) and the FKGL (10.9±2.9 without HONcode, 11.8±2.4 with HONcode, t41=−0.964, p=0.341).

Examples of incorrect information are displayed in table 5 along with extracted supporting quotes. The CONcheck item which was most commonly incorrect (by six websites) was item 5, 'Consultation with a doctor is essential after suspected concussion'.

Table 5

Examples of incorrect information


This initial evaluation of online concussion information shows a wide variation in the overall quality of websites. The HONcode was selected as the measure of information quality in this study to provide an ethical standard in line with the HON Principles (table 1), and the presence of the HONcode allows the reader to be sure of the 'reliability and credibility' of the information they are reading.27 The absence of the HONcode in more than two-thirds of the websites in this study is similar to that in a recent review of orthopaedic sports medicine websites.22 There are several possible reasons for the lack of wider HON certification. Websites may: not be aware of the HONcode and its potential value, have unsuccessfully applied for HONcode certification, or elected not to apply for the HONcode as they do not perceive it as important or may feel their site would not get accredited. It is important to note that while the HONcode requires websites to state when they were last updated, this does not indicate the currency of the information content. Information relating to return-to-play best practice following concussion has evolved over recent years and websites need to reflect such changes by ensuring that information provided is in line with current concussion knowledge.

The CONcheck instrument used in this study was created to assess which websites provide concussion information in accordance with the key aspects of the 2008 CIS consensus statement, which is currently considered the 'gold standard' for concussion information.3 All websites that contained links to PDFs containing the CIS meeting statements registered the maximum possible CONcheck score, however, other websites had a total absence of concussion information. With the exception of the International Ice Hockey Federation43 and postdata collection the International Rugby Board,40 sporting federation websites were lacking in specific concussion information. In a practical sense, this may not be problematic as individuals may seek concussion information from more medically-focused websites. It can be argued, however, as these sporting organisations are contributing members towards the CIS group and figureheads of their respective sports, they could be expected to be more proactive in providing concussion education resources to both their members and the medical personnel responsible for their care.

Literature from the US has shown that a significant proportion of the general public struggle with comprehending healthcare information in the printed format,41 with the recommendation made that health information should be targeted towards a FKGL of around 6.42 The websites included in this study had a mean FRE of 46.95 and FKGL of 11.17 suggesting that existing online concussion information might not be at a level where the majority of the population would be able to fully comprehend and use it. In contrast, a random sample of concussion-related 'tweets' (short posts placed on the microblogging site Twitter)43 analysed by our research team had an FKGL of 2.9 and an FRE of 85.6, suggesting that information via this medium or language level, albeit short, could be more comprehensible to the majority of the population.

As well as being readable, it is critical that the knowledge transferred is accurate. The high proportion (17/43 or 40%) of websites in this study which contained elements of incorrect information, although lower than other healthcare websites previously evaluated,40 suggests that there is potential for the mismanagement of concussion through erroneous or inaccurate advice. The CONcheck item which was most frequently incorrect related to the need to seek consultation with a doctor after concussion. Seeking medical consultation postconcussion is accepted as an essential aspect of concussion management2 ,3 and therefore it was of concern that many of the websites neglected to offer this key information.

The evolution of online health information encourages the use of innovative methods in the transfer of concussion knowledge.44 ,45 The 2008 CIS Statement (developed for physicians, athletic trainers and sports medicine professionals among others)2 was widely disseminated through the academic literature3 but this information may not necessarily filter down to grassroots level, namely to concussed players and their team-mates and those who care for them. There is a clear need to supply critical information such as the 2008 CIS Statement2 to the widest audience in as clear terms as possible, and in a format whereby individuals and their families can understand it. Future versions of the consensus statement should consider a 'plain language' summary statement targeted towards the general public to maintain knowledge transfer fidelity, in a format which can be easily posted and marketed. Recent studies have demonstrated the emerging role of Social Networking Sites (SNSs) such as Facebook46 and Twitter47 in relation to concussion information, and these SNSs have the potential to significantly facilitate knowledge transfer.

Due to the large number of medical/health websites, a selective and non-exhaustive website identification approach was taken in this study which meant that some relevant websites (eg, snowsports websites) may not have been captured. A consensus model added strength to the study.

Further studies are required to continue to validate the CONcheck to ensure that it becomes a recognised tool for use by healthcare providers, who may be looking to direct consumers (patients and players) towards online concussion information. Due to the importance of ensuring accurate online information, similar studies should be conducted in selected areas of sports medicine to evaluate the overall quality of information on the internet for the most common sporting injuries/conditions (eg, anterior cruciate ligament injury, Achilles tendinitis etc).


This study has shown that the overall quality of online concussion information is varied. Many of the websites identified by the search strategy were missing key facts relating to concussion management and were not written at a level where the majority of people would be able to understand them. Although the knowledge base behind current concussion management practices is sound, steps need to be taken to ensure that the information is disseminated in a manner which will be accessible to all parties. SNSs (including Facebook and Twitter) have the potential to play a role in the future of concussion knowledge transfer.


The authors thank Sridhar Alla and Hopin Lee for undertaking the CONcheck coding, and Professor Willem Meeuwisse for his input in the development of CONcheck.


View Abstract


  • Funding This study was facilitated by funding from the International Rugby Board.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions
