{'fraction_non_alphanumeric': 0.02422360248447205, 'fraction_numerical': 0.004554865424430642, 'mean_word_length': 5.274025974025974, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Quality assessment of clinical practice guidelines: who watches the watchmen?

Guidance was once such a simple thing: provided by a valued few to those who sought to learn. Knowledge, experience and wisdom were rare commodities, the topics to be discussed limited. The world has turned. Information is now abundant, healthcare data are widespread. Gradually, it has become impossible to read, appraise and interpret all published material, even within subspecialty fields. In an attempt to deal with this issue, guidelines have increasingly found their way into clinical practice and regulatory decision-making. As defined by the Institute of Medicine, clinical guidelines are ‘systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances’. In other words, guidelines should safeguard users and patients from wrong decisions. Although research supports their ability to do just that, in reality, there have been many that were contradictory, of poor quality, impractical or simply not evidence based. It has caused clinicians to be wary of them. Indeed, they have brought forth a new dilemma: if the guideline declares who is right and who is wrong, how then do we assure that the guideline itself is right? In this month’s Journal, Hazlehurst et al. provide a comparative assessment of guidelines produced by five major professional societies in endocrinology. The authors assessed a basic aspect of guideline quality: the methodology by which individual guideline groups rated the level of evidence underpinning their recommendations. They found that of the 58 guidelines published between 1995 and now, only 29 explicitly mentioned that a grading system was used at all. Overall, only 6 5% of recommendations were based on ‘high level’ evidence. In addition, different organizations used different systems, with different criteria for ratings of quality of evidence, leading to potential confusion and misunderstanding in the clinicians using them. Clearly, a valid, standardized system that is applied consistently would be the ideal. So what is the problem? Simply, we need to distinguish between the grading of the evidence supporting a recommendation and the grading of the recommendation itself. Not all grading systems separate decisions regarding the quality of the evidence from the strength of the recommendations. Those failing to do so create confusion. In this respect, Hazlehurst et al. highlight the utility of the grading of recommendations, assessment, development and evaluation (GRADE) system, which is increasingly accepted as the preferred framework by organizations worldwide. The idea behind GRADE is that although recommendations may arise from large, rigorous randomized controlled trials that show consistent and impressive benefits, this need not be the case. If data on appreciable harms of a treatment come from suboptimal observational studies, the availability of a harmless alternative may be enough to prompt a strong recommendation against it. The evidence for benefit of a treatment may be statistically overwhelming, and yet clinically so small and at such cost, that panels would be reluctant to attach a strong statement in favour or against. This has important consequences as it places emphasis on ‘choice’. Presented with the facts, clinicians must still weigh the desirable and undesirable effects of alternative management strategies carefully. They must do so according to their own values and preferences and particularly according to those of their patients. So, when the evidence is of high quality and the advantages of an intervention clearly outweigh the disadvantages, this will lead to a strong recommendation based on high-quality evidence (1A). But even if the evidence is of high quality, if different clinicians or patients might choose different strategies based on their balancing of the pros and cons, this would lead to a weak recommendation (2A). A simple, transparent grading of the recommendation can indeed effectively convey this key information. Perhaps unsurprisingly, Hazlehurst et al. found that guidance in endocrinology often did not rely on ‘high’ quality evidence and that little has changed over the past 15 years. Alas, we face a catch-22 here: if for a certain topic there is a good systematic review containing several large, well performed randomized controlled trials (and hence, the level of evidence is high), it is unlikely that clinicians will need explicit guidance as they can mostly interpret the evidence by themselves. In addition, both the generation, collection and/or interpretation of evidence may be influenced by other drivers: funding bodies – often industry. Indeed, industry only invests in stud*See related paper on pages 183–190 of this issue.