{'fraction_non_alphanumeric': 0.032699366032699365, 'fraction_numerical': 0.003670337003670337, 'mean_word_length': 4.87843137254902, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Comment on: Heterogeneity in meta-analysis should be expected and appropriately quantified.

Higgins in his commentary on the study by Patsopoulos and coworkers outlines some of the limitations of I in assessing heterogeneity in a metaanalysis. These limitations have perhaps not been as explicitly explained before. They have particular implications for assessing heterogeneity in meta-analyses of cohort and database studies and this aspect is worthy of further comment. As pointed out by Higgins, I is not a point estimate of the between-study heterogeneity. It is the proportion of total variation in the point estimates that is attributable to between-study heterogeneity. The total variation is comprised of between-study variation and within-study variation. If the individual studies have large sample size, such that the within-study variation is small, then I could be large (say475%), even if the between-study variation is small. This is particularly a problem for meta-analyses of cohort and database studies, which typically have large sample sizes. The advantages I are that it does not depend on the number of studies in the meta-analysis or the effect measure used (e.g. odds ratio, risk ratio, hazard ratio, mean). However, it does depend on the size (precision) of the individual studies. It might be sensible to compare I across different meta-analyses, where the component randomised controlled trials (RCTs) all have about the same sample size. However, it is not sensible to compare I across meta-analyses of RCTs (sample size in the 100s) and meta-analyses of cohort or database studies (sample size in the 1000s). An alternative is to report the between-study variance ( ), calculated as part of a random-effects meta-analysis. The disadvantage of 2 is that it depends on the effect measure (e.g. odds ratio, risk ratio, hazard ratio); that is, it does not provide a standardized way of measuring heterogeneity. Meta-analyses of large observational studies (that address policy questions) are increasingly being used to support funding decisions. Statisticians should strive to display and communicate the results in a way that is useful to decision makers. In this context, little would be lost if quantitative measures of statistical heterogeneity (e.g. Q and its P-value, I, ) were not reported. As Higgins points out, any amount of heterogeneity is acceptable, provided pre-defined inclusion criteria are met. A more useful approach would be to focus attention on the forest plot. If one or more studies are outliers, then there is more uncertainty about the mean measure of effect across all studies. This uncertainty is not necessarily adequately depicted by the confidence interval from a random effects meta-analysis, so that judgement is needed when applying results (as in all areas of applied statistics). Increasingly complex analyses, which are mysterious and difficult for nonstatisticians to understand, do not necessarily shed light on the problem at hand.