{'fraction_non_alphanumeric': 0.02216216216216216, 'fraction_numerical': 0.003243243243243243, 'mean_word_length': 5.202234636871508, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Growth , Precision , and CAT : An Examination of Gain Score Conditional SEM

Monitoring the growth of student learning is a critically important component of modern education. Such growth is typically monitored using gain scores representing differences between two testing occasions, such as prior to and following a year of instruction. The current paper examines the precision of gain scores, and hence their potential meaningfulness, within an item response theory (IRT) framework. The conditional standard error of measurement (CSEM) of the gain score, calculated as a simple function of the component scores from which the gain score is derived, is used to evaluate gain score precision under two different conditions. In the first condition, an example vertical scale is developed using state testing program data to demonstrate that large variation in measurement precision is to be expected when reporting gain scores. Several graphing techniques are utilized to depict visually how measurement precision varies across the distribution of proficiency. In the second condition, adaptive testing is demonstrated to substantially improve the measurement precision of gain scores in comparison to traditional, fixed testing. The potential benefits of adaptive testing for gain scores are discussed, as are the real-world limitations. Acknowledgements A version of this manuscript was presented at the 2008 Annual Meeting of the National Council on Measurement in Education in New York. The author wishes to thank Denny Way and Ye Tong for their helpful comments on an earlier version of this paper. The author also expresses thanks to Bob Dolan, who made a number of suggestions that substantially improved the paper. Growth, Precision, and CAT 3 Growth, Precision, and CAT: An Examination of Gain Score Conditional SEM Measurement of student growth in learning is an important topic for K-12 state testing programs, both in terms of school accountability as well as for reporting progress of individual students. Recently, Ho (2007) provided a brief status report of growth models in the field of educational measurement, citing a surging interest in growth modeling that is likely to continue given draft reauthorization proposals for the No Child Left Behind Act (NCLB) that prominently feature growth models. Despite this surge, however, relatively less attention has been given to whether reported measures of growth are precise enough to be meaningful and useful. For example, it has been acknowledged that tests built for school accountability under the NCLB status model are unlikely to provide ideal measures under a growth model (e.g., Steering Committee of the Delaware Statewide Academic Growth Assessment Pilot, 2007). Most nonadaptive proficiency tests (i.e., those consisting of fixed, predetermined content), such as those designed to assess students’ knowledge, skills, and abilities vis-à-vis grade-level academic standards, measure the central part of the proficiency distribution much more precisely than the extremes of the distribution. In many cases, the non-central regions will be measured poorly. As a consequence, growth scores for students scoring aboveand belowaverage will be less precise than for students whose scores are near average. Furthermore, when selecting a measure of student growth it is important to consider the inherent precision of the measure. A common way to represent growth for a particular student is to simply take the difference between the student’s current year score and their previous year’s score. Assuming the two scores are on the same scale, the difference can be thought of as the amount the student has “gained.” Although policy makers find gain scores appealing because the scores are easily understood by stakeholder Growth, Precision, and CAT 4 groups, psychometricians have long questioned the reliability of gain scores. The issue of score precision is important because even the most elaborately designed vertical scale or growth model will be useless if the scores reported from it lack precision. In this paper, the precision of the most basic of growth measures, the simple gain score, is examined within an item response theory (IRT) framework. Because comparatively little of the vast psychometric literature on gain scores has approached the subject from an IRT perspective, some background on IRT gain score precision is provided. To this end, we explain that the conditional standard of error of measurement (CSEM) of the gain score is calculated as a simple function of the individual CSEMs of the two component scores from which the gain score is derived. Next, the variation in measurement precision that can be anticipated when reporting gain scores is demonstrated by developing an example vertical scale (for equating test scores across multiple grade levels) using item parameter estimates from a state testing program. This variation, we argue, likely renders non-informative any vertical scales developed from conventional (non-adaptive) tests due to lack of score precision. Finally, in another demonstration using item parameters from a state testing program, gain scores from a computer adaptive test (CAT) are compared to those from a non-adaptive version. Discussion of the potential benefits of CAT and the likely limits of those benefits is provided. Although the main context for the paper is vertical scales in K-12 assessments, the methods employed to investigate gain scores apply to any situation where IRT-based gain scores are used. Study Design