{'fraction_non_alphanumeric': 0.019643953345610803, 'fraction_numerical': 0.01780233271945979, 'mean_word_length': 5.626016260162602, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Combining Multiple Regression and CART to Understand Difficulty in Second Language Reading and Listening Comprehension Test Items

Identifying sources of item difficulty allows test developers to better define the constructs that they are testing by providing empirical evidence. Past research has explored item difficulty of reading and listening comprehension items using multiple regression and classification and regression tree (CART) analyses (e.g., Freedle & Kostin, 1993, 1996, 1999; Sheehan, 1997); however, few studies have combined these techniques so that practitioners can evaluate their relative contributions to our understanding. In this study, 214 computerized reading and listening comprehension items completed by 87 nonnative English speakers of varying ability levels were analyzed using a two-fold approach. First, item difficulty was modeled as a function of 12 text and item and text interaction predictor variables in a multiple linear regression model. Seven of the 12 variables in the model accounted for about 31% of the variance in item difficulty as measured by the adjusted R2. Second, the data were used to build multiple regression tree models using CART, a nonparametric technique that uncovers linear dependencies among predictor variables. Seven variables, some of which were not identified in the regression analysis, were relatively important across all trees. We found that synthesizing results from the 2 methodological perspectives provided a richer picture of the interrelations of variables that affect item difficulty, and provided some empirical support for our construct definitions.