{'fraction_non_alphanumeric': 0.02947845804988662, 'fraction_numerical': 0.014965986394557823, 'mean_word_length': 5.89375, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Guest Editor's Introduction: Qualitative Inquiries of Participants' Experiences With Standard Setting

Shepard, Glaser, Linn, and Bohrnstedt (1993) suggested that participants may not understand their judgments when they engage in the Angoff method because of the complexity of the underlying tasks. The perceived cognitive complexity of the tasks involved in judgmental standard-setting methodologies also led to a reiterated characterization of the Angoff (1971) procedure as “fundamentally flawed” in an evaluation of the standard-setting methodology that was used for the National Assessment of Educational Progress (Pellegrino, Jones, & Mitchell, 1999). In response to these criticisms and their experience conducting standard setting, Mitzel, Lewis, Patz, and Green (2001) developed the bookmark method in an effort to reduce the cognitive complexity of the judgmental tasks. They suggest that by ordering the items or score points to provide information on relative difficulty and by reducing the number of judgments, the cognitive demand on the participants is minimized. However, an empirical study that compared the results of Angoff and bookmark methodologies using independent panels yielded similar cut score recommendations (Buckendahl, Smith, Impara, & Plake, 2002). Standard-setting researchers have generally spent more effort developing methods or variations of methods that seek to transform a policy-adopted conceptual definition of performance into a numerical value that relates to both the definition of performance and the content specifications of a given test. The challenge for all standard-setting methodologies is to effectively translate a participant’s mental model of the target examinee (e.g., barely proficient student) into judgments that communicate the participant’s recommendation of a value that characterizes the point of separation between one or more categories. Although researchers have speculated on the cognitive demand for participants, few studies have examined participants’ thought processes or experiences during these judgmental standard-setting tasks. APPLIED MEASUREMENT IN EDUCATION, 18(3), 219–221 Copyright © 2005, Lawrence Erlbaum Associates, Inc.