{'fraction_non_alphanumeric': 0.02830058123443122, 'fraction_numerical': 0.008407140880154996, 'mean_word_length': 5.146076972145439, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Systematic review of knowledge translation strategies in the allied health professions

Background Knowledge translation (KT) aims to close the research-practice gap in order to realize and maximize the benefits of research within the practice setting. Previous studies have investigated KT strategies in nursing and medicine; however, the present study is the first systematic review of the effectiveness of a variety of KT interventions in five allied health disciplines: dietetics, occupational therapy, pharmacy, physiotherapy, and speech-language pathology. Methods A health research librarian developed and implemented search strategies in eight electronic databases (MEDLINE, CINAHL, ERIC, PASCAL, EMBASE, IPA, Scopus, CENTRAL) using language (English) and date restrictions (1985 to March 2010). Other relevant sources were manually searched. Two reviewers independently screened the titles and abstracts, reviewed full-text articles, performed data extraction, and performed quality assessment. Within each profession, evidence tables were created, grouping and analyzing data by research design, KT strategy, targeted behaviour, and primary outcome. The published descriptions of the KT interventions were compared to the Workgroup for Intervention Development and Evaluation Research (WIDER) Recommendations to Improve the Reporting of the Content of Behaviour Change Interventions. Results A total of 2,638 articles were located and the titles and abstracts were screened. Of those, 1,172 full-text articles were reviewed and subsequently 32 studies were included in the systematic review. A variety of single (n = 15) and multiple (n = 17) KT interventions were identified, with educational meetings being the predominant KT strategy (n = 11). The majority of primary outcomes were identified as professional/process outcomes (n = 25); however, patient outcomes (n = 4), economic outcomes (n = 2), and multiple primary outcomes (n = 1) were also represented. Generally, the studies were of low methodological quality. Outcome reporting bias was common and precluded clear determination of intervention effectiveness. In the majority of studies, the interventions demonstrated mixed effects on primary outcomes, and only four studies demonstrated statistically significant, positive effects on primary outcomes. None of the studies satisfied the four WIDER Recommendations. Conclusions Across five allied health professions, equivocal results, low methodological quality, and outcome reporting bias limited our ability to recommend one KT strategy over another. Further research employing the WIDER Recommendations is needed to inform the development and implementation of effective KT interventions in allied health.


Introduction
Systematic reviews, comparative effectiveness reviews, and technology assessments aim to review and synthesize the relevant scientific literature as a basis for decision making [1]. Given the wide range of topics and outcomes addressed in this type of work, researchers frequently include nonrandomized studies to provide a detailed picture of the current knowledge and limitations of a given intervention [2e5]. Specifically, nonrandomized designs may increase the evidence base regarding long-term outcomes and safety. Nonrandomized studies may also be used to identify current limitations in evidence, recommend the types of studies that would provide stronger evidence, and guide future research [5,6].
In the context of such reviews, the appropriate classification of studies according to their design is important to guide (1) decisions around inclusion; (2) the assessment of methodological quality or risk of bias; (3) the combining

What is new?
We modified a tool for the classification of study designs in the context of systematic reviews, comparative effectiveness reviews, and technology assessments focusing on interventions and exposures.
Testing showed moderate agreement (k 5 0.45) among six testers and low accuracy against a predetermined reference standard.
We found that systematic testing and refinement enhanced the reliability of the tool.
The classification of study designs is challenging and may be because of shortcomings of the classification tool, inadequate and inconsistent study reporting, and variation in tester characteristics.
Application of such a tool should be accompanied by adequate training, pilot testing, and documented context-specific decision rules.
Additional testing and refinement by other groups using different samples is encouraged and will enhance the utility of the tool.
of study results in a narrative synthesis or by statistical pooling; (4) the interpretation of findings; and (5) grading the body of evidence. Although there are a number of tools in existence to classify study designs, only one has undergone rigorous testing. Furlan et al. [7] tested a ''traditional taxonomy'' in the area of interventions for low back pain and found low reliability among reviewers despite detailed instructions and definitions. Furlan et al. [7] made a number of recommendations for future research, including developing a more comprehensive taxonomy in terms of the scope of study designs and testing the taxonomy in different fields of research.
The goal of this project was to identify a tool that could be used within the context of systematic reviews and other synthesis work focusing on interventions and exposures, and assist with the classification of study designs. The primary objectives were to (1) identify classification tools that are currently used by systematic reviewers and other researchers to classify studies according to design; (2) select the best classification tool for modification and evaluation; (3) develop instructions, including an algorithm and decision rules, for application of the modified tool to studies of interventions and exposures; and (4) test the tool and accompanying instructions for concurrent validity and interrater reliability.

Identification of classification tools
A sample of classification tools was compiled by contacting representatives from all the Evidence-based Practice Centers (EPCs) funded through the US Agency for Healthcare Research and Quality, other relevant organizations, and individuals with expertise in this area. Individuals were contacted by e-mail and asked to identify any taxonomies, guidelines, or other systems used to classify study designs.

Selection of classification tool for testing
The five authors ranked the tools using the following predefined criteria: (1) ease of use (i.e., contains a logic that users can readily follow); (2) unique classification for each study design (no overlap); (3) unambiguous nomenclature and decision rules/definitions (if applicable); (4) comprehensiveness in terms of range of study designs; (5) potentially allows for identification of threats to validity; and (6) developed by a well-established organization.

Development of the classification tool for testing
The selected tool was modified to incorporate relevant elements of other tools and tested through an iterative process in which the tool was applied to a sample of studies. Decisions to modify the scheme were based on the collective experience of the authors. A glossary of study design definitions and related concepts was developed to accompany the tool. The tool and glossary document were reviewed by all authors before formal testing.
While the tool was being developed, the EPCs were requested to provide examples of intervention or exposure studies, where the assignment of study design had been problematic. Two authors (L.H., K.B.), who were not involved in producing the reference classification, selected 30 sample studies from the pool of 71 studies (Appendix A). Studies were selected to cover a range of topic areas and cover all of the key decision nodes within the algorithm to ensure adequate testing of the tool. For most designs (all but twodcase-control and crosssectional) at least two sample studies were included. Additional studies were included for the designs that we felt to be more commonly encountered in systematic reviews of therapeutic interventions and exposures (e.g., before-after studies, controlled before-after studies, and cohort studies). The selection of studies was based on the design determined by two of the authors, which was not consistent in all cases with the final reference standard classification. According to the reference standard classification, all decision nodes and all but two designs (prospective cohort, nested case-control) in the algorithm were represented. A list of all 71 studies is available from the corresponding author.

Testing and development of the reference classification
Six individuals from the University of Alberta EPC used the tool to classify the designs of the 30 sample studies with minimal additional instruction or direction; no specific training or pilot testing was conducted ( Table 1). The number of testers was based on previous work in this area [7] and published guidelines for reliability studies [8e10]. Testers were told that it would take approximately 5e10 hours to categorize the 30 studies and were asked to complete the assignment over a 2-week period. The study design names in the tool and glossary were masked, and letter codes were used in their place in an attempt to have testers work through the flow diagram in a systematic fashion rather than relying on study design labels. The tool is presented in Appendix B; the glossary is available at http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/ ?pageaction5displayproduct&productid5604.
Concurrently, three authors (D.D., P.L.S., M.V.) independently applied the tool to the 30 test studies to develop the reference classification, that is, the ''true'' classification for each study. Disagreements were resolved by discussion and consensus.
Overall interrater reliability was calculated using Fleiss' kappa (k). Interrater reliability was calculated separately for the reference standard raters and the testers. As well, interrater reliability was calculated based on the level of formal training of the testers (completed relevant graduate training vs. currently enrolled in graduate training). Accuracy of the testers was measured against the reference classification and interpreted according to Landis and Koch: kappa ! 0, poor agreement; 0e0.2, slight; 0.21e0.40, fair; 0.41e0.60, moderate; 0.61e0.80, substantial; 0.81e1.0, almost perfect [11]. The time taken to classify the sample of studies and the time taken per study were also recorded, and mean times were calculated.
After the first round of testing, the tool was modified further based on the results of semistructured interviews with the testers to ensure the tool's usefulness and ease of use in the context of a systematic review (Appendix C). Six testers from the University of Alberta EPC participated in a second round of testing using a random sample of 15 studies from the 30 studies used for the first round of testing. Three of the testers had been involved in the first round of testing and three had not. The three testers who were involved in the first round received no feedback after the first round of testing and were not aware of the reference standard design classifications of the sample of studies used for testing. As in the first round of testing, no specific training or pilot testing was conducted. Four of the testers received the flow diagram with the study design labels (testers in round 1 indicated a preference for unmasked labels), whereas two testers received the flow diagram with letter codes masking the study design labels. The same analyses were conducted for the second round of testing. The tool was further refined after the second round of testing ( Fig. 1).

Identification of classification tools
We contacted 31 organizations or individuals to identify study design classification tools. Eleven organizations or individuals responded providing 23 potential tools. All authors reviewed the tools and 10 were considered relevant for the purposes of this project ( Table 2). Table 2 provides the results of the ranking and observations of the tools made during the process. The three top-ranked tools were those developed by the Cochrane Non-Randomized Studies Methods Group (NRSMG), the American Dietetic Association, and the RTI International-University of North Carolina at Chapel Hill Evidence-based Practice Center (RTI-UNC EPC). The three tools were all algorithms, that is, they provided a logical sequence of ''yes or no'' decisions to make when classifying studies. None of the algorithms covered the range of study designs that systematic reviewers might encounter when conducting evidence synthesis work, particularly designs in which there is no comparator group. Further, the study nomenclature was inconsistent among the algorithms. The Design Algorithm for Studies of Health Care Interventions, developed by the Cochrane NRSMG, was considered the preferred tool and was used as the basis for further development (note that this tool is no longer advocated by the NRSMG).

Reference classification
The initial agreement among the three reference classification raters was fair (k 5 0.33). The three reviewers agreed on the classification of seven studies (23%), two of the three agreed on the classification of 14 (47%), and there was no agreement on the classification of nine (30%). Disagreements occurred at most decision points in the algorithm except for  (1) ''were at least three measurements made before and after intervention/exposure?'' (2) ''was intervention/exposure data registered prior to disease?'' and (3) ''were both exposure/intervention and outcome assessed prospectively.'' The area that created the greatest uncertainty and disagreements for the reference standard raters was the decision node ''was there a single cohort?'' Specifically, it was often difficult to determine whether the two groups under study were derived from the same cohort, and the tool did not provide any criteria to make this decision. The initial decision node (''was there a comparison?'') was also a source of disagreement. Specifically, it was unclear whether or not to classify the study as having a comparison when subgroup analyses were performed within a single group. A third point of disagreement was determining when a study was an interrupted time series (i.e., measurements taken at a minimum of three time points before and three time points after the intervention). Although there is a precedent for this definition (http://www.epoc.cochrane.org/Files/Website/Reviewer% 20Resources/inttime.pdf), the number of required time points may not be universally accepted.

Tester characteristics
Six testers, with varying levels of training and experience in systematic reviews, tested the modified tool. The length of time they had worked at the University of Alberta EPC ranged from 9 months to 9 years. Three of the testers had obtained a master's degree in public health or epidemiology, and three testers were undertaking graduate level training in epidemiology or library and information sciences. The time taken to classify the 30 studies ranged from 7 to 9 hours with a mean of 8 hours overall and 16 minutes per study. Because the tool was new to the testers, this time reflects, in part, the process of familiarizing   (10) Not an algorithm per se, rather a description of study designs and design elements. Useful for controlled trials, does not seem as useful for cohort studies.
Abbreviations: NRSMG, Non-Randomized Studies Methods Group; EPHPP, Effective Public Health Practice Project; ADA, American Dietetic Association; NRS, nonrandomized study. a These documents were produced by the same group but at different times; the most recent approach to study design classification advocated by the Cochrane NRSMG is the second tool listed which appears in the current version of the Cochrane Handbook for Systematic Reviews of Interventions available at: www.cochrane-handbook.org. themselves with the flow diagram and the accompanying definitions.

Agreement
There were no studies for which all six testers agreed on the classification ( Table 3). Five of six testers agreed on the classification of seven studies, four agreed on five studies, three agreed on nine studies, two agreed on eight studies. The overall level of agreement was fair (k 5 0.26). The levels of agreement for testers who had completed versus those who were undertaking graduate level training were fair (k 5 0.38) and slight (k 5 0.17), respectively.
Disagreements occurred at all decision points in the algorithm; however, testers identified the determination of a single cohort as particularly problematic. The testers also said that certain terminology in the flow diagram was unclear (e.g., ''group'' vs. ''cohort'') and that uncertainty arose because of poor study reporting. There was some variation in the manner in which testers used the flow diagram (e.g., use of glossary, working forward vs. backward through the algorithm).

Accuracy of testers compared with reference classification
There were no studies for which all six testers agreed with the reference classification, and there was wide variation in the testers' accuracy of classification (Table 3).

Tester characteristics
Six staff members at the University of Alberta EPC, with varying levels of training and experience in systematic reviews, were involved in the second round of testing. Three of the testers had been involved in the first round of testing, and three of the testers had no previous involvement with the project or knowledge of the tool being tested. One tester had a PhD in Medicine, three testers had a master's degree in epidemiology, and two testers had undergraduate degrees in health sciences or related field and were undertaking graduate level training in epidemiology. The length of time the testers had worked with the University of Alberta EPC ranged from 2 months to 9 years. Four of the testers used a flow diagram that had the study design labels, whereas two of the testers used a flow diagram with letter codes. The time taken to classify the 15 studies ranged from 2.25 to 4 hours with means of 2.75 hours overall and 11 minutes per study.

Agreement
There were three studies for which all six testers agreed on the classification ( Table 3). Five of six testers agreed on two studies, four agreed on six studies, three agreed on two studies, and two agreed on two studies. The overall level of agreement was considered moderate (k 5 0.45). The degree of agreement for testers who had completed versus those undertaking graduate level training was moderate (k 5 0.45) and fair (k 5 0.39), respectively. The difference between individuals with different levels of training was less than observed during the phase 1 testing. The level of agreement was moderate for both those who had the flow diagram with study design labels (k 5 0.41) and for those with letter codes (k 5 0.55). The agreement for testers who had and had not completed the first round of testing was fair (k 5 0.36) and moderate (k 5 0.45), respectively. For the three testers who completed both rounds of testing, intrarater reliability was fair (k 5 0.33, k 5 0.34) and moderate (k 5 0.59).
The least common agreement occurred at four key decision nodes: whether the study was an experimental design (5 of 15 studies), whether there was a comparison (4 of 15 studies), whether the assessment of exposure and outcome was prospective or retrospective (3 of 15 studies), and whether the intervention or exposure and outcome data were gathered concurrently (2 of 15 studies).

Accuracy of testers compared with reference classification
There were three studies for which all six testers agreed with the reference classification, but there was wide variation in the testers' accuracy of classification (Table 3).

Discussion
We identified over 20 tools to classify study designs and selected one for modification and testing. The final version of the modified tool showed moderate agreement among six testers and low accuracy against the predetermined reference classification. The moderate level of agreement is consistent with a previous study testing a ''traditional taxonomy'' [7]. The level of agreement observed in these two studies brings to light concerns about lack of agreement when using the classification tools and even greater concern with a far less transparent process when no classification tool is used at all. There are numerous tools in existence and, to our knowledge, few have undergone testing either during or after development. However, our findings also demonstrate that it is possible to systematically test and modify a tool to yield more reliable results.
There are a variety of reasons for the moderate and fair levels of agreement and accuracy observed in our study. The results likely reflect issues with the tool itself, as well as attributes of the studies that were selected for testing. The studies used during testing were identified and selected because they had posed challenges for previous reviewers with respect to their design classification. Agreement might be better with a sample of studies that is more representative of the studies that would be included in a systematic review. Further, the sample of studies that we tested covered a wide range of topics. If the studies had been on the same topic, which would be the case in a systematic review, there may have been greater reliability. One of the main reasons that the selected studies were difficult to classify was poor reporting within the studies, which resulted in the need for testers to make assumptions (e.g., whether a study was prospective or retrospective). We also found classification challenging when there were discrepancies between the intent of the investigator and the conduct of the study, between the design and how data were analyzed, and between the investigators' initial plan and their study implementation.
Moderate agreement also resulted from shortcomings of the tool itself. Many of the decision points were problematic. For example, in one-third of the studies, there were discrepancies as to whether or not the study was truly ''experimental.'' Identifying ''quasi-experimental'' studies is challenging as it requires a decision about the degree of control that the investigator has over certain aspects of study design and execution. Such studies may not be considered either purely experimental (a ''trial'') or purely observational. This area of study design needs to be more clearly reflected in the tool. Clear guidelines are needed to interpret the extent of control that an investigator has. The practical repercussion of this uncertainty in study classification is that some ''quasi-experimental'' studies (e.g., before-after or controlled before-after studies) may incorrectly be classified as trials; hence, their internal validity may be exaggerated, and the results given too much weight in the context of a systematic review [12]. One design, that is, particularly problematic has been variously referred to as an ''uncontrolled trial'' or ''singlearm trial.'' This design is associated with risk of bias arising from the lack of a control or comparison group. Consequently, these should be considered ''before-after'' studies, and our tool was designed to channel them toward this classification.
Other decision nodes that yielded inconsistent results concerned whether there was a comparison and whether the data collection was prospective or retrospective. Several factors may have contributed to this inconsistency, including a lack of clarity within the questions posed in the algorithm, the testers' relevant background knowledge, the testers' experience or training, the inconsistent use of design terminology among the studies, and lack of detailed reporting. Testers who had completed relevant graduate level training had greater agreement with the reference standard than those who were undertaking graduate level training.
We observed a fair level of agreement among the reference classification raters as well. The three reference classification raters had substantial expertise in research methods and systematic reviews: all had doctoral level training in epidemiology or research design and 4e8 years experience in systematic reviews and related research. The low level of agreement among these raters may reflect the more general complexities of study designs and the challenge of including all design considerations into a single flow diagram.
Variability in classification of studies may also reflect differences in how individuals applied or worked through the algorithm. For example, some testers worked backward or back-tracked to classify the studies according to what they felt was the most appropriate description. The testers also used the glossary accompanying the tool to varying degrees.
The difficulties in interpreting study design labels and the consequent difficulties in reaching agreement in assigning these labels to individual studies are consistent with those of other researchers. These issues have led some authors to recommend that systematic reviewers focus on features of designs rather than on design labels when assessing studies for inclusion and evaluating potential risk of bias [13]. We endorse this approach and recommend that reviewers should be as explicit as possible about the design features that are being considered. However, this does not obviate the need for or the usefulness of design labels in describing studies being considered for inclusion in systematic reviews. The use of a classification tool, including an algorithm, may provide greater transparency and consistency to the process by closely examining the design features.

Implications for practice
The appropriate classification of studies by design (or by design features) is a critical step in a systematic review to guide the selection of studies, the assessment of the risk of bias, the analysis of study results, the interpretation of results, and the grading of the body of evidence. There is a clear need for consistent use of terminology and study design labels, as well as a clear understanding of the terminology used in a particular field by those undertaking a systematic review in that field. We believe that a tool such as the one developed and tested in this study could be useful to guide this process, although the application of the tool requires several considerations to optimize agreement and reliability among reviewers.
First, training in research methods, as well as in the use of the tool, is essential. Pilot testing the tool in the context of each review is highly recommended. Second, decision rules are needed for different fields of research or review topics. Specifically, there needs to be clear decisions around how to handle a lack of clarity in study reporting. For example, when the response to a question in the algorithm is unclear, one option is to assume that the condition was not met. Documentation of the decision rules will allow for consistency and transparency. Users of the algorithm need to use standardized definitions of study designs and design features. This is what we had intended with the accompanying glossary.

Future research
The tool developed and tested in this study serves as a basis for use in systematic reviews and further research. We made minor revisions after the final round of testing that merit further testing. Research is needed to evaluate the tool within the context of a real systematic review. We encourage further testing and refinement of the tool by other groups using different samples to enhance its utility. Further research may also offer insights into the differences in reliability among individuals with varied experience, training, and education. The tool may serve as a basis for future methods work on identifying risk of bias, interpreting findings, and grading the body of evidence.

Conclusions
We developed and tested a tool for the classification of study designs. The level of agreement among six testers was moderate, and the accuracy against a reference classification was low. There are a number of explanations for the observed reliability and accuracy, including shortcomings of the tool, inadequate reporting of the studies, and differences in tester characteristics. Application of such a tool in the context of a systematic review should be accompanied by adequate training, pilot testing, and documented decision rules. This study demonstrates that systematic testing and refinement enhances the reliability of the tool. At the study level, clear reporting, adherence to published reporting guidelines, and appropriate and consistent use of design terminology should be enforced.