{'fraction_non_alphanumeric': 0.02971737323358271, 'fraction_numerical': 0.016209476309226933, 'mean_word_length': 5.539402173913044, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Rasch Measurement Approach to Analyzing Differences in Pencil-and-Paper and Online Formats for Higher Education Course Evaluations

The purpose of this study was to utilize the Rasch model to identify differences and/or similarities between pencil-and-paper and online student course evaluations at a Midwestern I Research University. Pencil-and-paper course evaluations were collected from 55 advanced undergraduate and graduate students in fall 2008. Online course evaluations were collected from 49 advanced undergraduate and graduate students enrolled in fall 2009. In both cases, the students were enrolled in an introductory statistics course taught at the same day/time by the same instructor. A polytomous Rasch model was utilized to analyze the data with Winsteps. Results demonstrate pencil-and-paper and online course evaluations function analogously for categories and person endorsability, but differently for course and instructor items, as well as learning outcomes. Measuring differences in pencil-and-paper and online course evaluations A Rasch measurement approach to analyzing differences in pencil-and-paper and online formats for higher education course evaluations Since the development and proliferation of the internet during the past two decades, higher education institutions have been considering and/or using web based course evaluation systems in place of traditional paper and pencil collection methods. Such development and usage has spawned numerous studies examining the differences and similarities of online and paper and pencil delivery modes for surveys and evaluations within all domains, including higher education. Myriad studies of student satisfaction, student and faculty perceptions, response rates, response bias, frequency and length of student comments, timeliness, cost effectiveness, and the like have been conducted. However, most, if not all, have employed various Classical Test Theory (CTT) methods, such as t-tests, chi-square, ANOVA, and MANOVA, but not the Rasch model. A latent-trait IRT model, the Rasch model provides a means to not supply basic aggregate data for course evaluations, such as in CTT methods, but to supply information about how an evaluation’s respondents and items are functioning and, in turn, the reliability and validity of aggregate student satisfaction and teacher effectiveness results. Hence, this study will focus on identifying differences and/or similarities between a web based and traditional paper and pencil course evaluation via the Rasch model to determine if course evaluations may function differently online than on paper. Literature Review Higher education institutions have utilized paper based surveys for decades to evaluate teacher effectiveness and student satisfaction with courses. With the proliferation of the internet within the past decade, institutions are entertaining the notion of evaluating courses online. Measuring differences in pencil-and-paper and online course evaluations Hmieleski (2000), as noted in Hoffman (2003), found 98% of the nation’s 200 most wired institutions were still predominantly collecting data via paper and pencil based surveys. However, Hoffman (2003) found 10% of 500 randomly selected colleges and universities were primarily using the internet for collecting student evaluation data. Also, he discovered 56% of institutions were already using the Internet for evaluating online courses, 10% would be implementing an Internet ratings process in 2003, 18% were reviewing this as an option for online courses, and 16% decided not to use the Internet to evaluate online courses (Hoffman, 2003). Thus, it is evident America’s higher education institutions consideration of trading paper and pencil evaluations for online web links is rising. As with a rise in any new phenomena, differences between the old and the new (web based versus paper and pencil course evaluations) have been evaluated via various scholars from numerous angles. Several scholars have discovered potential response influences for course evaluations in general. For instance, Layne, DeCritoforo, & McGinty (1999) discovered academic area can affect evaluation results, while Liegle & McDonald (2005) noted pro-exam and high performing student respondents’ evaluated instructors higher than pre-exam and low performing student respondents. Student satisfaction with the web based delivery mode of course evaluations has also been continuously examined. For instance, Anderson, Cain, & Bird (2006), Kasiar, Schroeder, & Holsted (2002), Layne et al. (1999), and McGourty, Scoles, & Thorpe (2002) all found students are content and prefer the online course evaluations. McGourty, Scoles, & Thorpe (2002) also discovered students view a web based format as more convenient, since they can fill it out on