{'fraction_non_alphanumeric': 0.03993203058623619, 'fraction_numerical': 0.033984706881903144, 'mean_word_length': 5.029449423815621, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reflective Intentions: Philosophical Concepts of Intentionality

Towards Understanding Expert Coding of Student Disengagement in Online Learning Luc Paquette (paquette@tc.columbia.edu) Department of Human Development, Teachers College, Columbia University, New York, NY, 10027 Adriana M.J.A. de Carvalho (adrianac@cs.cmu.edu) Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, 15213 Ryan S. Baker (baker2@exchange.tc.columbia.edu) Department of Human Development, Teachers College, Columbia University, New York, NY, 10027 Abstract Gaming the system, a behavior where students disengage from a learning environment and attempt to succeed by exploiting properties of the system, has been shown to be associated with lower learning. Machine learned and knowledge engineered models have been created to identify gaming behaviors, but few efforts have been made to precisely identify how experts code gaming behaviors. In this paper, we used cognitive task analysis to elicit knowledge about how experts code students as gaming or not in Cognitive Tutor Algebra. We show how building a cognitive model of this process gave us insights about the behaviors gaming is composed of. Keywords: Gaming the system, Cognitive Tutor, cognitive modeling, expert coding. Introduction In recent years, there has been increasing awareness that students often disengage from interactive environments by gaming the system , defined as attempting to succeed in an educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly (Baker et al. 2008). Gaming the system has been studied in multiple learning systems (Baker et al. 2008; Baker et al. 2010; Baker, Mitrovic, & Mathews 2010; Beal, Qu, & Lee 2006; Johns & Woolf 2006; Muldner et al. 2011; Walonoski & Heffernan 2006a) and has been shown to be related to poorer learning outcomes (Cocea, Hershkovitz, & Baker 2009; Fancsali 2013; Pardos et al. 2013), boredom (Baker et al. 2010), and poorer long-term academic success (San Pedro et al. 2013). Models of gaming the system have been created using both knowledge engineering (Aleven et al. 2006; Beal, Qu, & Lee 2006; Gong et al. 2010; Johns, & Woolf 2006; Muldner et al. 2011; Walonoski and Heffernan 2006a) and machine learning (Baker et al. 2008; Baker, Mitrovic, & Mathews 2010; Walonoski, & Heffernan 2006a) approaches. Both approaches have been successful at allowing educational systems to intervene when students are gaming the system (Arroyo et al. 2007; Walonoski & Heffernan 2006b) and understanding the relationship between gaming and other constructs, but they do not implement a deep understanding of the construct of gaming the system and of the context of the student's actions as experts do when coding gaming behaviors. In particular, instead of studying how experts code gaming behaviors, knowledge engineered models have typically implemented simple versions of two types of behaviors related to gaming (Baker et al. 2008), help abuse (Aleven et al. 2006) and systematic guessing. Help abuse has mainly been modeled using behaviors that include copying the answer from a hint (Gong et al. 2010; Johns, & Woolf 2006; Muldner et al. 2011) and repeated help requests (Beal, Qu, & Lee 2006; Walonoski, & Heffernan 2006a), but few efforts have been made to understand how expert coders determine when those behaviors are instances of gaming. Struggling students will often look at multiple help messages and might sometime copy answers from hints as part of their learning process. As we will discuss, expert coders rely on the context in which those behaviors take place to determine if the student is gaming or not. Systematic guessing is usually defined as quickly answering after errors (Beal, Qu, & Lee 2006; Gong et al. 2010; Johns, & Woolf 2006; Muldner et al. 2011) and making successive errors (Walonoski, & Heffernan 2006a). Similarly to help abuse, many instances of those behaviors can be associated with normal use of the learning environment. For example some minor errors can be very quickly corrected, a behavior that looks similar to guessing. In those situations, expert coders will once again look at the context in which the actions were executed in order to determine if they are systematic guesses or not. The machine learning approach, on the other hand, captures part of the context of the student's gaming behaviors, but it is not easy to fully understand what context the models capture and to evaluate whether it corresponds to expert judgments. In other words, machine learning often makes the same predictions as experts, but may be doing so in different ways.