{'fraction_non_alphanumeric': 0.0395070677781805, 'fraction_numerical': 0.01594780717651323, 'mean_word_length': 4.935483870967742, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Optimizing scoring formulas for yes/no vocabulary tests with linear models

Yes/No tests offer an expedient method of testing learners’ vocabulary knowledge, although a drawback of this method is that since the method is self-report, actual knowledge cannot be confirmed. “Pseudowords” have been used within such lists to test if learners are reporting knowledge of words they cannot possibly know, but it is unclear how to use this information to adjust scores. Although a variety of scoring formulas have been proposed in the literature, empirical research (e.g., Mochida & Harrington, 2006) has found little evidence of their efficacy. The authors propose that a standard least squares model (multiple regression), in which the counts of words reported known and counts of pseudowords reported known are added as separate predictor variables, can be used to generate scoring formulas that have substantially higher predictive power. This is demonstrated on pilot data, and limitations of the method and goals of future research are discussed. Background Yes/No checklists of English vocabulary knowledge (YN tests) can be used as a quick, psychometrically reliable self-report method of checking students’ knowledge of large numbers of words in a relatively short period of time (Meara & Buxton, 1987). However, an obvious pitfall of such an approach is that tests reliant on self-reporting may produce results that differ from reality. Indeed, recent research by Stubbe and Yokomitsu (2012) reveals that students who completed a YN test of English words and were then asked to provide Japanese definitions for words they had reported knowledge of could on average only provide correct definitions for half. In many cases, overestimation of vocabulary size was due to confusion of an unknown word (e.g., root) with a known, but untested word with similar orthography (e.g., route). Such a disparity between YN test scores and translation tests scores is not uncommon. Waring and Takaki (2003) reported a 70% drop on scores from a recognition test to a similar (L2 to L1) translation test. Pseudowords (e.g. “steck” or “noof”) are commonly used in YN tests to detect false reports (Read, 2000); if a student claims to know a word that does not actually exist, it calls their reports of knowledge on real words into question. SLA literature contains a number of different formulas for YN tests that use the number of pseudoword checks, called false alarms (FA), to adjust the proportion of words considered to be known. For example, Huibregtse, Admiraal and Meara (2002) studied the following four correction formulas: h f (Anderson & Freebody, 1983), cfg (correction for guessing, Meara & Buxton, 1987), Δm (Meara, 1992) and Isdt (Huibregtse, et al., 2002).