{'fraction_non_alphanumeric': 0.019455252918287938, 'fraction_numerical': 0.005002779321845469, 'mean_word_length': 5.741573033707865, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multiverse Analysis: A Method to Determine Researcher Degrees of Freedom in Test Validation

ObjectiveIn psychometric validation there are a near limitless number of different approaches researchers can decide to take when choosing sample type and size, test metrics, and covariate measures. It is particularly important to factor in these different analytical pathways when validating tools intended for use in clinical populations. Here, we aimed to provide a tutorial paper showcasing the multiverse, or specification curve, technique to establish the robustness of analytical pathways on psychometric validation outcomes in an example test of executive function. MethodWe examined the impact of choices regarding sample groups, sample sizes, different types of metrics from a given test, covariate inclusions, and outlier removal on convergent validation correlations between new tests of executive function. Data were available for 88 neurologically healthy adults and 117 stroke survivors, a total of 6,660 different analyses were run. ResultsWe found that the type of sample group and sample size used for analyses impacted validation outcomes. Comparing test metrics within-category (e.g. accuracy vs accuracy) rather than between-category (e.g., accuracy vs time) significantly increased the observed correlation coefficients. Covariate inclusion and outlier removal choices did not impact the observed coefficients in our analyses. ConclusionThis tutorial paper showcases how authors can use the multiverse technique to evidence the robustness, or fragility, of their validation outcomes across many different analytical pathways. The example here highlights the importance of fully justifying and considering the analytical pathway to take in validation work on clinically relevant tools.