{'fraction_non_alphanumeric': 0.027984865193246125, 'fraction_numerical': 0.010268160777606825, 'mean_word_length': 5.305533580380564, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Student Success Prediction in MOOCs

Predictive models of student success in Massive Open Online Courses (MOOCs) are a critical component of effective content personalization and adaptive interventions. In this article we review the state of the art in predictive models of student success in MOOCs and present a dual categorization of MOOC research according to both predictors (features) and prediction (outcomes). We critically survey work across each category, providing data on the data source, feature extraction from raw data, statistical modeling, model evaluation, prediction architecture, experimental subpopulations, and prediction outcome. Such a review is particularly useful given the rapid expansion of predictive modeling research in MOOCs since the emergence of major MOOC platforms in 2012. This survey reveals several key methodological gaps, which include extensive filtering of experimental subpopulations, ineffective student model evaluation, and the use of experimental data which would be unavailable for real-world student success prediction and intervention, which is the ultimate goal of such models. Finally, we highlight opportunities for future research, which include temporal modeling and research bridging predictive and explanatory student models.


Introduction
In their short history to date, Massive Open Online Courses (MOOCs) have simultaneously generated enthusiasm, participation, and controversy from both traditional and novel participants in the educational landscape. Trying to understand and influence both enrollment and completions has led to efforts to generate effective student models which can predict student dropout, completion, and learning in MOOCs. Despite the extensive attention devoted to such work by several related research communities and by the popular media, little overall synthesis of this work has been performed. We believe that such a synthesis is necessary, now more than ever, for several reasons.
First, MOOC research is at a critical stage in its development. An abundance of research has explored the phenomenon of MOOC dropout from several perspectives since Pappano (2012) declared 2012 the "year of the MOOC", as shown in Figure 1 and we survey n = 86 such studies in this work. A clear synthesis of this research is necessary in order to explore where consensus has emerged across the research community, where there may be research gaps or unanswered questions, and what action needs to be taken as a result of both. If we fail to learn from the lessons of several years of MOOC analysis, learner experiences and outcomes will fail to improve and MOOCs may fail to deliver on their promise for millions of learners around the globe.
Second, there is a need to evaluate not only the findings of such research, but also its methodology -not only to survey the techniques most commonly used, but to understand whether gaps exist (we believe they do) and what needs to be done about them. In particular, this issue is relevant to predictive modeling of student success in MOOCs because of the diverse communities that its practitioners are drawn from: education and the learning sciences, computer science, statistics and machine learning, behavioral science, and psychology researchers each bring different methods to the field. A methodological survey allows scientists to ensure that their knowledge is constructed on a strong methodological foundation, and to strengthen it where appropriate. In particular, a survey of predictive modeling allows for (a) the sharing of feature extraction and modeling approaches known to be effective, while also encouraging exploration into under-researched methods, and (b) sharing of overarching experimental protocols, such as prediction architectures and statistical evaluation techniques, which affect the inferences such modeling experiments produce. In this work, we provide novel and detailed data about the state of predictive student modeling in MOOCs for researchers interested in both (a) and (b).
Third, we believe that a critical promise of student success prediction in MOOCs has not yet been delivered on: the use of these predictions to actively improve learner outcomes and experiences through operationalization of predictive models in MOOC platforms. We hope that this work can identify effective strategies for such tools to be used "in the wild" in active courses to achieve their oft-stated goal of impacting learner success in MOOCs. The implementation of live, real-time tools and personalized interventions stands to benefit from effective predictive modeling which can target and personalize interventions for those who need them most. A clear knowledge of the research consensus on effective predictive modeling methods in MOOCs will support the construction of such tools, effectively "closing the loop" of predictive modeling in MOOCs. We leave the development of the interventions base don these predictive models to future work.
In the work that follows, we address each of these three goals. In the remainder of this section, we provide the reader with a basic introduction to MOOCs and survey the state of the MOOC landscape to date. In Section 2, we dive deeper into the specific focus of this work by discussing student success prediction in MOOCs, introducing the task, the data typically available for its execution, and the basic procedure for the construction and evaluation of predictive models. Section 3 surveys prior work on predictive models of student success in MOOCs, including a detailed matrix of n = 86 previous works on this topic in Table 6. We synthesize the results of this survey in Section 5, highlighting (a) overall trends; (b) research gaps and unanswered questions suggested by the literature survey; (c) methodological issues raised by our extensive overview of the "state of the practice" in the field; and (d) opportunities for utilizing the current research consensus to impact the educational experience of MOOC learners. We conclude in Section 6. This work is part of a series on predictive models in MOOCs, and in future works we provide a discussion of techniques for model evaluation; replication of machine learned models in MOOCs; and novel techniques for prediction.

MOOCs: A Novel Educational Context
Massive Open Online Courses are enticing, in part, because they are so different from many other forms of education. However, exactly what a MOOC is is itself the subject of some debate. We do not seek to resolve this debate here; in this section, we detail several generally agreed-upon characteristics of MOOCs in order to build a working definition for use within this review. We take MOOCs to have the following attributes: Massive, Open and Online: By definition, these are the attributes most closely associated with MOOCs. MOOCs are massive in that they typically have far more students than even the largest traditional classroom courses. This would include, at minimum, hundreds of learners for very specialized courses to hundreds of thousands of learners for more general or popular courses. The instructional team tasked with supporting these learners typically consists of a very small staff; therefore the student-teacher ratio in these courses is far higher than in traditional higher education or e-learning courses. MOOCs are open to all learners, often being both public and free. The two largest English-language MOOC providers, Coursera and edX, initially offered all courses free of cost, though business model changes have seen more barriers to taking free courses over time (though both platforms still offer financial aid programs). The openness of MOOCs is perhaps what makes them most exciting by providing access to high-quality educational experiences for all learners around the globe. 1 Finally, MOOCs are online -they are digital, internet-based courses, not in-person courses. Course materials, assignments, and peers/instructors are all accessed on the World Wide Web via a computer or other device with a web browser or a dedicated platform-specific application.
Low-or No-Stakes: MOOCs are often "low-stakes" or "no-stakes" environments. Traditional higher education and e-learning courses are typically taken strictly for academic credit or other official certifications, often at a non-trivial financial cost to the participant. In contrast, MOOCs provide the option to simply take the course independent of any certification, credit, or degree program. This gives MOOCs a particularly unique set of course participants who sometimes have little or no investment in completing a course, making the task of student success prediction (and, consequently, the task of student support based on these predictions) particularly challenging. Under this definition, paid and for-credit online courses are typically not considered MOOCs. Many other low-or no-stakes learning environments exist -such as textbooks and tutorials, museums, and other offline and online resources -but these environments do not share the other features of MOOCs.
Asynchronous: The time scale for content consumption and participation in a MOOC tends towards the flexible, although the degree of this flexibility may vary. Many MOOCs are clearly divided into "modules," often by week, which are released to learners over time. These courses often have clearlydefined start and end dates, with successful completion being contingent upon learners meeting specified criteria by the course end date. Within these time windows, however, learners were typically free to browse content and complete assignments in any order and at any time. A fully asynchronous model has recently become more common in MOOCs, where learners have access to all content on demand after entering a course, and can complete content at their own pace. We note that this model has coincided with the transition to a subscription-based, as opposed to course-based, pricing model on certain MOOC platforms.
Heterogenous: As a direct consequence of many of these features, the popoulation of learners in MOOCs is heterogenous in terms of both demographics and intentions (Koller et al (2013); Chuang and Ho (2016)); even as course populations skew toward college-educated males from industrialized countries, these course populations are still far more diverse than any of the other educational contexts superficially similar to MOOCs Glass et al (2016). The backgrounds of learners vary significantly, from graduate-level educated learners who are employed full-time in the subject area of the course, to students without a high school diploma; these learners vary in gender, age, nationality, and intent. The majority of MOOC students are located outside the United States and hold a bachelor's degree (Chuang and Ho (2016)), and there is also evidence that teachers are well-represented in course populations ; Chuang and Ho (2016)). However, obtaining even basic demographic data on users is currently only available through on optional questionnaires with low response rates (Kizilcec and Halawa (2015); Whitehill et al (2015); DeBoer et al (2013)), so predictive models are often unable to utilize this data directly and instead need to draw directly on learner behavior, not demographics or reported intentions.
Together, these features of MOOCs define an educational environment that is sufficiently different from other well-studied environments -such as e-learning, on-campus higher education, or digital K-12 education -to justify the formation of a new and separate predictive modeling literature. As an illustrative example, consider a comparison of a "dropout" student (a noncompleter) in a MOOC versus any of the traditional contexts mentioned: we would reasonably expect very different factors to contribute to dropout, different subpopulations to be most likely to drop out, and for these learners to experiment different consequences of dropping out, in a MOOC compared to other educational contexts. Indeed, some practitioners have argued that the term "dropout" should be altogether replcated by the more appropriate "stopout" (Reich (2014)); others have argued for broader reconceptualization of traditional student success metrics in MOOCs (DeBoer et al (2014)). As we will discuss below, there are also very different data sources available in MOOCs compared to each of these other contexts (as one example, MOOCs collect rich, granular behavioral data at a level that is unavailable in almost any other context; furthermore, MOOCs are characterized by a lack of complete and reliable historical or demgraphic data; in contrast, institutional course providers typically lack any readily-available behavioral data but have rich historical, demographic, and co-curricular data).
Our goal in describing these features of MOOCs is not to argue for a particular conceptualization of MOOCs; it is simply intended to clearly describe the features of MOOCs used as the criteria for inclusion in the literature review in Section 3 below. Where studies are excluded, it is typically because they did not evaluate what we considered to be MOOCs, or did not meet other criteria discussed in Section 3. We note that select studies were still included despite not meeting individual components of this definition (for example, we do consider some work evaluating for-credit courses); in these cases we typically include such work either (a) for completeness due to the novelty or important contribution of the work, or (b) in order to err on the side of inclusion when the context of the course(s) under evaluation was not clear. We also note that the xMOOC phenomenon is not represented in our analysis, but this is in part because we were not able to identify any instances of xMOOCs being used with predictive student success models.

The State of the MOOC Landscape
As of 2017, an estimated 58 million students have registered for or participated in at least one MOOC (Shah (2016)). The five largest MOOC providers, according to self-reported enrollment numbers, are Coursera 2 , 23 million registered users; edX 3 , 10 million registered users; XuetangX 4 , 6 million registered users; FutureLearn 5 , 5.3 million registered users; and Udacity 6 , 4 million registered users (Shah (2016)). Enrollment continues to grow over time, but there is some indication that enrollments have begun to slow as platforms have transitioned to paid models and phased out various free certification options, and as the course population declines in size over repeated iterations of a course (Chuang and Ho (2016)).
These impressive enrollment figures mask a well-known issue with the MOOC experience: around 90% of students who enroll in a MOOC fail to complete it Jordan (2014). Given the lack of barriers to entry, massive course populations, and high student to teacher ratios in MOOCs, this may not be particularly surprising. As shown in Table 1, a majority of predictive modeling research in MOOCs has focused on dropout prediction. While the massive dropout rate may fail to account for student intentions -indeed, most students do not intend to complete MOOCs -the best data indicates that slightly more than half of students intend to achieve a certificate of completion in a MOOC, and around 30% of these respondents achieve certification in a typical MOOC Chuang and Ho (2016). This low completion rate even among intended completers is still cause for concern, and we believe effective predictive models can support several approaches to improving MOOC dropout rates.
As of 2017, MOOCs cover a variety of topics, with over 6,850 courses offered by more than 700 universities across these platforms (Shah (2016)). Coursera, for instance, offers more than 180 specializations (sequences of courses in a specific topic area, such as "Data Structures and Algorithms" or "Dynamic Public Speaking"). There are several full online graduate degrees offered on the platform, such as the Master of Business Administration iMBA program offered by the University of Illinois, Urbana-Champaign on Coursera. The edX platform offers pathways for learners into higher education, such that when a program (called a MicroMasters) is completed on the MOOC platform, learners are then provided with credit transfer if they subsequently enroll in a residential graduate program. The University of Arizona's Global Freshman Academy provides the opportunity for students to complete their entire freshman year online. Regardless of platform, format, and structure, Computer Science courses continue to be the most popular courses on the platform, with science, history, business, and health courses also popular (Chuang and Ho (2016); Shah (2016); Whitehill et al (2017)).

Student Success Prediction in MOOCs
Before surveying the vast body of prior work on student success prediction in MOOCs, in this section we seek to clearly define and motivate the task. This framing is essential to the discussion below and to the conclusions we draw from this review.

Defining Student Success
Student success can mean many different things in a MOOC, and several different approaches have been used to measure and predict student success in MOOCs, including completion, certification, overall course grades, and exam grades, shown in Table 1. The task of discussing student success in MOOCs is particularly challenging due to the fact that we typically apply language and metrics adopted from traditional educational settings -i.e., dropout, achievement, participation, enrollment -that can mean very different things in the context of a MOOC (DeBoer et al (2014)).
In the context of this work, we define student success to denote a broad class of metrics which measure course completion, engagement, learning, or future achievement related to the content of a MOOC. We believe that each of these broad categories suggests at least one kind of motivation participants in a MOOC might have for joining the course, but each alone is certainly 8 Josh Gardner, Christopher Brooks

Outcome Description Count Dropout
A student drops out if they do not "complete" a course. Often operationalized as whether a student continues participating in a course until the course concludes.

39
Stopout A student is a "stopout" if they stop engaging with the course prior to the end of the course Taylor et al (2014). Often, stopout is functionally equivalent to dropout, but merely emphasizes that we often cannot observe a students' intention to "drop out" and instead only observe whether they stop interacting with the course.

Certification
Certification is achieved when a student earns a certificate of accomplishment for the course. This typically consists of earning enough points on course assignments to meet some pre-determined threshold for the course (typically around 70%). 16

Final Exam Grade
The students' grade on the course final exam, a cumulative test typically administered near the end of the course which tests the full subject matter taught in the course. 4

Final Course Grade
The students' overall grade in the course, calculated based on the instructors' specification. Typically, course grades are a mix of one or more of the following: in-video quizzes, out-of-video quizzes, homework assignments, problem sets, human-graded assignments, and exams.
14 Pass/Fail A student typically passes a course if they meet or exceed an instructor-specified overall grade threshold; otherwise they fail.
15 Table 1 Common student success metrics for predictive models in MOOCs. Dropout/stopout prediction is the most common, but several learning outcomes (final exam grade, course grade, pass/fail) have also generated significant attention. inadequate to describe "success." We review work which presents the results of a predictive model of any type of student success according to this definition.
Having several potential metrics to describe student success in MOOCs is useful for several reasons: (a) it allows us to capture metrics related to the diverse goals MOOC learners have, such as course completion, certification, career advancement, or subject mastery (Koller et al (2013); Reich (2014)); (b) it reflects the lack of research consensus on how to measure student success in MOOCs (Perna et al (2014); DeBoer et al (2014)); and (c) it allows us to test the robustness of models by potentially checking their ability to predict multiple different outcomes. While (c) has been an uncommon approach to date, we believe that this is an important avenue for future work (for one example, see Fei and Yeung (2015)).
Several metrics are used to measure student success in the works surveyed below. A collection of the most common metrics used for student success prediction and the frequency with which they occur in our literature review is shown in Table 1. For an examination of alternative long-term metrics of student success, see Wang (2017).

Why Model Student Success in MOOCs?
Student success predictions are useful for a wide variety of tasks, and these models vary along three main dimensions relevant to these tasks (shown in Figure 2). We identify three main reasons for developing predictive models of student success: Personalized Support and Interventions: Identifying students likely to succeed (or not succeed) has the potential to improve the student experience by providing targeted and personalized interventions to those students predicted to need assistance. This is the stated motivation behind much of the work surveyed here, which often refer to these students as "at risk" learners (a term adopted from the broader educational literature). In particular, because of the massive student population in MOOCs relative to the size of the instructional support staff, clearly identifying struggling students is important to providing those students with targeted and timely support. Many of the "human" resources in MOOCs are quite scarce (i.e., instructor time), and predictive models can provide timely guidance on (a) identifiation of which students need these resources, and (b) intervention by predicting which resources can best support each at-risk student. While a teacher might be able to directly observe students in a traditional in-person higher education course, or even in a modestly sized e-learning course, such observation is not available to support MOOC instructors at scale, and predictive models can serve this purpose.Particularly when instructor time and resources are scarse, predictive models which can identify these students with high confidence and accuracy are required. Additionally, many interventions would be unnecessary or even detrimental to the learning of engaged or otherwise successful students.
We refer to the dimension along which model predictive performance varies as its accuracy; we discuss how to measure the quality of a model's predictions below. Here, it suffices to say that accuracy is critical to the delivery of personalized interventions. We refer to models which enable targeted and timely interventions for supporting student success as actionable, and we argue in Section 5.2 that there are problems with the actionability of most prior predictive modeling research in MOOCs due to experimental design.
A subset of personalized interventions which would be enabled my effective predictive models in MOOCs is adaptive content, where the presentation of course content is tailored to optimize for student success. Very little prior research has utilized adaptivity or true real-time intervention based on student success predictions in any form in MOOCs. Whitehill et al (2015) utilizes dropout prediction to optimize learner response to a post-course survey (this work optimizes for data collection, not learner success), and He et al (2015) describes a hypothetical intervention based on predicted dropout probabilities (but only implements the predictive model to support it, not the intervention itself), Pardos et al (2017) implements a real-time adaptive content model in an edX MOOC, but this is optimized for time-on-page, not student learning. Kotsiantis et al (2003) developed a web-based support tool utilizing a machine learning-based predictive model for a distance learning degree program of 354 students, a scale far smaller than most MOOCs. This dearth of research on both adaptive interventions and adaptive content supported by accurate, actionable models at scale is, at least in part, due to a lack of consensus on the most effective techniques for building predictive models in MOOCs, which we address through the current work.
Data Understanding: Predictive models can also be useful exploratory or explanatory tools that help understand the mechanisms behind the outcome of interest. Instead of strictly providing predictions to enable personalized interventions or adaptive content, predictive models can be tools to identify learner behaviors, learner attributes, and course attributes associated with success or failure in MOOCs. These insights can drive improvements to the content, pedagogy, and platform, and contribute to our understanding of the underlying factors influencing student success in these contexts. They also contribute more directly to theory by providing more than the strictly empirical data from predictive models. We describe this dimension of models as theory-building to highlight their usefulness in the formation of theories about these underlying factors. From this perspective, certain types of models are more useful than others: models with straightforward, interpretable parameters (such as linear or logistic regression models, which provide interpretable coefficients and p-values; and decision trees, which generate decision rules) are far more useful for human understanding of the underlying relationship than those with many complex and interacting parameters (such as a multilayer neural network). Unfortunately, the latter are usually (although not always) more effective in making predictions in practice, so there is often a tradeoff between interpretability and predictive performance. Recent advances in making more complex models interpretable suggest that this tradeoff may be reduced in the future (e.g. Baehrens et al (2010); Craven and Shavlik (1996); Ribeiro et al (2016)), but at present this "fidelity-interpretability tradeoff" is still a salient issue for predictive models in MOOCs (Nagrecha et al (2017)). This issue is further discussed in Section 5.3.2 below. Similar issues apply to the underlying features or predictors used for such models, which also often face a tradeoff between complexity and interpretability (compare, for example, the simple counting-based activity features described in Section 3.1.1 and the complex, higher-order n-gram activity features described in 4.2).

Data for Student Success Prediction in MOOCs
In this subsection, we briefly describe the raw data available for student success prediction in MOOCs, including the common formats, schema, and types of behaviors and metrics collected. We provide data on the use of each raw data source across works surveyed in Section 5.1. Student success prediction in MOOCs has attracted a great deal of enthusiasm in part because of the data available to researchers interested in studying MOOCs. Digital learning environments such as MOOCs provide rich, highgranularity data at a scale simply not available in traditional educational con-Actionability Accuracy Theory-building Fig. 2 Three salient dimensions of predictive models in MOOCs. Models vary along all three dimensions, but there is no strict trade-off between any dimensions. We synthesize the state of MOOC research with respect to these dimensions, and highlight methodological gaps needed to improve predictive student models, in Section 5.2.
texts. While this data varies slightly from platform to platform, because of the dominance of only a few large MOOC providers (most notably, Coursera and edX), the available datasets are remarkably consistent in practice. This is useful for several reasons: (a) enables the use of consistent feature extraction and modeling methods, even across platforms, which reduces both development and computation time; (b) it allows for direct replication of research across courses and even across platforms.
Common data generated by MOOC platforms are discussed below; the use of these data types across our literature survey is also shown in Figure 6 in Section 5.1 below.

Clickstream Exports
Clickstream exports, also called server logs or clickstream logs, are typically records of every interaction with the server which hosts the course platform in JavaScript Object Notation (JSON) format. These interactions include every request to the web server hosting the course content, including each mouse click, page view, video play/pause/skip, question submission, forum post, etc. The same metadata is recorded for each interaction, and from this record, we can build detailed datasets at several levels of aggregation. An example of entries from a clickstream log is shown in Figure 3; note the many detailed attributes recorded for each interaction. Clickstream exports are the most raw, high-granularity data available from MOOC platforms. However, this granularity also presents a challenge: raw clickstream data cannot be directly used as input for most predictive models; instead, "features" -attributes relevant to the outcome of interest -need to be manually extracted from the clickstream log. This is a labor-intensive process (we use the terms feature engineering and feature extraction interchangeably to refer to this process). Feature engineering appears more important to the effectiveness of predictive models than the statistical algorithm itself (see Section 3 for a more detailed discussion of the importance of feature engineering). Indeed, many of the works surveyed here introduce innovations only to the feature engineering method and adopt otherwise standard classification algorithms for predicting student success (i.e., Veeramachaneni et al (2014)).
Clickstream data also presents a challenge of scale. This data is often quite large (tens of gigabytes for a single course), due to its granular nature and the many individual interactions that take place over the duration of a MOOC. Any aggregation of individual user sessions or interactions requires manually parsing and aggregating data from the clickstream. Simply reading, processing, and extracting the features from such data can be computationally expensive.

Forum Posts
A defining feature of most MOOCs is a set of thread-based discussion fora used for various tasks, including interactions directly related to course content and more general community-building and discussion. Different platforms implement discussion fora differently 7 , but across every major platform, the text of forum posts and a variety of metadata and related interactions (such as upvotes for questions or answers) are typically collected in a relational database, accessed via Structured Query Language (SQL). As shown in Figure 6, forum post data is second only to clickstream data in terms of its use in predictive models of student success in MOOCs. This data is often used to extract (a) measures of engagement, by tracking users' forum viewing patterns; (b) measures of mastery, understanding, or affect, generated by applying natural language processing to the raw text of forum posts; and (c) social network data by assembling graphs where various connections in the fora constitute edges. An illustration of a threaded discussion post in a Coursera course is shown in Figure 4.

Assignments
Assignments are often used in MOOCs similar to the way they are used in residential or in-person courses, and data related to assignment submission is also often stored in a relational database. A variety of assignment types are used in MOOCs, including automatically graded assignments (such as multiple-choice assessments and programming tasks), manually-graded assignments (such as data analysis reports or essays, which can be graded by both course instructors or, more commonly, peers in the course), in-video questions, and interactive lab simulations. Assignment data is typically limited to metadata (i.e., open date, due date) and assignment-level or (less commonly) question-level data about submissions. As Figure 6 indicates, the use of as-signment features is less common, likely due to a combination of (a) the low number of users who complete assignments in MOOCs, as a proportion of total registrants or participants, and (b) the substantial variation across courses in the way assignments are used.

Course Metadata
Detailed information about the course and instructional materials are also typically recorded in MOOC platforms and retained for post-hoc analysis. This includes information about course modules, video lectures (length, title, module), and assignments (including quizzes, homework, essays, human-graded assignments, exams, etc.). Little research has actively explored the use of course metadata in predicting student success. The research which has evaluated such data, however, suggests that it may indeed impact factors such as learner persistence and engagement Evans et al (2016).

Learner Demographics
Most MOOC platforms also record information about learner demographics, when it is available. However, such information is typically collected via optional pre-and post-course surveys, which are subject to various response biases (Kizilcec and Halawa (2015)). While this information is potentially interesting, its limited availability (and bias in the data that is available) has limited the research on demographics in MOOCs to date to a small number of studies which we survey in 4.2; Hansen and Reich (2015) explored using external datasets and IP address-based geolocation to fetch additional demographic data, but not for predictive student modeling.

Literature Survey: Predictive Models of Student Success in MOOCs
In this section, we survey prior research on predictive models of student success. We intend this to be a relatively broad, inclusive literature survey, and therefore include most literature which involves (a) an application of predictive modeling of student success, where student success is broadly construed according to one or more of the metrics listed in Table 1; (b) doing so in the context of a MOOC, or in a context sufficiently similar or relevant to MOOC prediction as to be of interest to researchers interested in such models; (c) which meet basic standards for quality research, including peer-reviewed work which contains sufficient description of their methods as to provide insight into the data and feature engineering, modeling, and experimental results. When a work was considered borderline on one or more of these criteria, we generally erred on the side of inclusion if it made a novel or relevant contribution to the literature. The literature surveyed was drawn from several top conferences and journals in the fields of learning analytics and educational data mining,  Table 2 Feature types and outcomes used across works surveyed. When experiments considered predictive models with multiple feature types or outcomes, they were included in each category. "Academic" outcomes includes any of the following: pass/fail, final grade, assignment grade,exam grade. "Completion" includes all metrics of course completion, including certification and participation in final course module.
computer science, web usage mining, and education, but was also collected from other sources (online searches, citations from other works surveyed). The works below are grouped into broad conceptual categories based on the features, outcomes, and analysis of the experiment. Generally, the divisions between works are most closely identified by the features used as input for the predictive model; but these features align quite closely with the analysis in most cases (i.e., experiments which use activity-based features most often predict an activity-based outcome, dropout; experiments using performancebased features most often predict a performance-based academic outcome); these strong feature-outcome associations in works surveyed are shown in Table 3). We also note that the feature-outcome correlations are imperfect and there is significant overlap between many groups; where necessary, we mention relevant work in all applicable categories in our survey below.
This categorization is a novel contribution of the current work, and has not been previously applied to predictive modeling research in MOOCs, to the authors' knowledge. It demonstrates both well-researched areas and gaps or opportunities for future research. For example, Table 3 shows that only two works surveyed used performance-based features to predict course completion; further research in this area seems warranted.
Additionally, it has been noted in several works that in addition to being perhaps the most difficult, feature extraction is also one of the most critical tasks in predictive models of student success (Li et al (2016a); Robinson et al (2016); Nagrecha et al (2017)). For example, Li et al (2017), citing Zhou et al (2015), notes that "data preprocessing should be considered with more attention than learning algorithms". Sharkey and Sanders (2014) claims that feature extraction is "arguably the most important step in the process of developing a predictive model." Taylor et al (2014) state that "[w]e attribute success of our models to these variables (more than the models themselves)...any vague assumptions, quick and dirty data conditioning or preparation will create weak foundations for one's modeling and analyses," emphasizing their feature extraction methods over their modeling techniques despite fitting over 70,000 models in this experiment. The same authors argue in Veeramachaneni et al (2014) that "[h]uman intuition and insight defy complete automation and are integral part of the process" of predictive modeling in MOOCs; they find that the most predictive features are complex, often relational (requiring the linking of multiple data fields), and were discovered through expert knowledge of both context and content. Feature extraction is highlighted as one of the core components of the dropout prediction problem in Nagrecha et al (2017), which notes that "the electronic nature of MOOC instruction makes capturing signals of student engagement extremely challenging, giving rise to proxy measures for various use-cases" -that is, the extraction of signal (useful features) from the electronic records of a MOOC is a key task in the pipeline of predictive model-building. Therefore, we believed that an effective categorization scheme for this review should highlight feature extraction techniques and their underlying theoretical foundations. The association between many feature extraction methods and the outcomes they are used to predict further "brightens the lines" of this categorization in many cases (such as with performance-based models, which are overwhelmingly used to predict academic performance as shown in Table 3).
Each of the broad model categories considered below has something important to offer predictive modeling efforts, but there are likely different underlying factors driving the predictive performance of student success in each category, which makes the separate discussion necessary. Similar feature-based groupings have been used or suggested in other works (e.g. Whitehill et al (2015Whitehill et al ( , 2017; Li et al (2017); Liang et al (2016)).
Despite the current survey's emphasis on understanding predictive models of student success, we avoid categorizing the work surveyed based on their predictive results alone. This is because of large case-by-case variation in (a) the experimental subpopulations, which are different subgroups of different MOOC course populations, (b) the methodology and metrics for model evaluation, and (c) the outcome being predicted. These three factors are so divergent across the work surveyed that holding the performance of each experiment to the same standard would be more misleading than it would be useful. Indeed, some research has addressed this question directly, demonstrating how different prediction architectures, for example, can influence the results of predictive modeling experiments in MOOCs (Boyer and Veeramachaneni (2015); Brooks et al (2015a); Whitehill et al (2017)). We will discuss some of the methodological shortcomings that make conducting these comparisons so difficult in Section 5 below, including inconsistent experimental populations; ineffective model evaluation; unrealistic or impractical prediction architectures; inconsistent model performance metrics; and others. In future work, we present a sociotechnical platform designed to enable direct replication of predictive modeling results on the same MOOC datasets, which can ameliorate the issue of "apples-to-oranges" model comparison faced by readers to date.

Activity-Based Models
As shown in Figure 5, models utilizing activity-based features and outcomes are overwhemlingly the most common in the work surveyed. This is so for sev- Counts of works surveyed by feature type, which broadly represents the most common approaches in predictive models of student success in MOOCs. Activity-based feature sets are the most common, which primarily reflects the activity-based outcome (dropout) most commonly predicted in the works surveyed. Note that experiments considering multiple types of features were counted in all relevant categories.
eral reasons: first, as demonstrated in Figure ??, most of the works surveyed predict an engagement-based outcome related to dropout or course persistence. Activity features seem most appropriate for this type of prediction task (although more diverse feature sets may improve the quality and robustness of these models). Second, activity data are the most abundant and granular data available from MOOC platforms. Clickstream files (shown in Figure  3) provide detailed interaction-level data about users' engagement with the platform, and such granular data is simply not available for any of the other feature categories (collecting a similar level of granularity for these other feature types would require far more sophisticated data collection practices, such as affect detectors or other sensors, which are impractical at MOOC scale). Third, activity features appear to provide reasonable predictive performance even in non-activity-based prediction tasks, such as in grade prediction (e.g., ). Indeed, it is reasonable to expect behavior to be associated with non-behavioral outcomes (i.e., learning). However, we note that state-of-the-art predictive models generally combine feature types to achieve a complete, multidimensional view of learners (i.e., Taylor et al (2014)).
The level of sophistication of the activity-based features in the works surveyed varies substantially, ranging from simple counting-based features (e.g. Kloft et al (2014); Xing et al (2016)) to more complex features, including temporal indicators of increase/decrease Veeramachaneni et al (2014); Chen and Zhang (2017); Bote-Lorenzo and Gómez-Sánchez (2017), sequences Balakrishnan and Coetzee (2013); Fei and Yeung (2015), and latent variable models Sinha et al (2014a); Ramesh et al (2013Ramesh et al ( , 2014; Qiu et al (2016). Despite this variation, each of these typically uses the same underlying data source (clickstream, or a relational database consisting of extracted time-stamped clickstream events) and draws from a relatively small and consistent set of base features, including: page viewing, or visiting various course pages, such as video lecture viewing pages, assignment pages, or course progress pages; video interactions, such as play/pause/skip/change speed; forum posting or forum viewing (a more specific subset of page viewing which has received particular attention); assignment and quiz submissions, which can take a variety of forms depending on the course and which may include all assignment attempts and activity as well as the final submission.
The relative consistency of the underlying activity-based feature sets and the few categories into which they can be distilled is largely a reflection of the consistency of the affordances available across the dominant MOOC platforms, particularly edX and Coursera: page viewing, video viewing, forum posting, and assignment submission were, until the introduction of relatively recent features such as interactive programming exercises, some of the only activities available to users of the platform, and the only activities recorded in Coursera clickstreams Coursera (2013).
In this section, we first discuss research where the goal is truly predictive, i.e., learning a mapping from learner-level data to an outcome label in Section 3.1.1; we discuss explanatory work separately in Section 3.1.2. There is often a clear distinction between each type of research, primarily based on whether the experimenters are interested in evaluating the model predictions (in the case of predictive research) or the model parameters (explanatory research). We discuss progress toward the intersection between predictive and explanatory models for future research in Section 5.3.2.

Predictive Activity-Based Models
One of the earliest predictive activity-based models relevant to this survey was Ramos and Yudko (2008), who argue as early as 2008 in the title of their work that "Hits" (not "Discussion Posts") predict student success in online courses. This pre-MOOC study, conducted on an online university course, is notable for its finding that a simple count of page hits predicted student success (as measured by final course grade) better than either discussion posts or quiz scores, predicting between 7% and 26% of the variance in course grades. This finding has been reinforced in subsequent experiments evaluating behavioral features against other feature types in MOOCs (Crossley et al (2016); Gardner and Brooks (2017b)). Ramesh et al (2013) applied Probabilistic Soft Logic (PSL) to a set of activity-and natural language-based features to model student performance in another early MOOC experiment. Using an expert-generated latent variable approach in which engagement is "modeled as a complex interaction of behavioral, linguistic and social cues", this work demonstrates an early application of latent variable modeling, which continues to be used in predictive models of student success. However, this particular method presents a potential barrier to practical implementation by utilizing manually-specified features. Requiring human-generated PSL rules is problematic for predictive models of student success: (a) even experts do not appear to know the underlying factors important to student success in MOOCs, particularly in a new course or a different domain, and (b) learning these features is itself the goal of datadriven predictive modeling. Manually defining latent engagement categories, as Ramesh et al. do, prevents truly data-driven discovery of latent user profiles or engagement types. Furthermore, by restricting the model to a small set of 5-7 features, this approach limits experimenters from learning about broader feature sets and their relationship to student success. Ramesh et al (2014) expands on their approach by using the latent variable assignments from this PSL method as predictors in a survival model.
Another early activity-based modeling experiment is Balakrishnan and Coetzee (2013). This this work uses a relatively small set of activity-based features (Cumulative Percentage of Available Lecture Videos Watched, Number of Threads Viewed on the Forum, Number of Posts Made on the Forum, Number of Times the Course Progress Page was Checked), compiled over each week of the course, to construct Hidden Markov Models to predict dropout. The work reflects an early approach to utilizing the textittemporal nature of MOOC data (both by assembling a weekly feature set, and by using the HMM, which captures transition probabilities over time). A particularly novel aspect of this work is Balakrishnan and Koetzee's use of students' checking of their course progress page as an input feature. A challenge present in all predictive models of student success in MOOCs is accounting for students' diverse intentions (browsing, learning, completing, etc.). Balakrishnan and Koetzee introduce the course progress checking feature as an observable -and effective -proxy for an intention to complete: students who never check their course progress have a dropout rate of 20-40% at each week of the course, while students who check their progress four or more times have a dropout rate of less than 5% each week. This particular result suggests that finding observable proxies for student intentions is a tractable and useful problem for predictive models of student success in MOOCs. Kloft et al (2014) also provided a foundational early predictive model, utilizing a Support Vector Machine built on features extracted entirely from clickstream events. They find that high-level features related to activity (number of sessions, number of active days) were predictive of dropout during the first third of the course; measures more directly related to engagement with course content (wiki page views and homework submission page views) became more predictive in the middle third of the course; and general activity-related features (number of requests, number of page views) were most predictive during the final third of the course. Kloft et al. use principal component analysis to demonstrate that successively generating a wider feature space by concate-nating feature vectors for each subsequent week improves separability between dropout and non-dropout students by the final third of the course. This feature appending strategy has been widely adopted in predictive modeling in MOOCs (e.g. Xing et al (2016)), likely due to the nearly universal structuring of MOOCs into weekly modules. Kloft et al (2014) only report the accuracy of their method, but based on the data they provide, the model's predictions offer less than a 5% improvement on a majority-class prediction over the first 10 weeks of the course, when most dropouts occur (the challenge of evaluating this particular result using only accuracy highlights issues related to model evaluation and the lack of consistent metrics for reporting predictive results we discuss in Section 5). If predictive models are to be used to support early interventions in MOOCs, more accurate predictions are required.
Several works attempt to address this need for early predictions of student success in a MOOC. Jiang et al (2014b) offers a simple logistic regression classifier based only on week 1 behavior which effectively predicts certification in a MOOC offered to university students. This model uses only four predictors representing different aspects of student engagement in week 1 of the course (average quiz score, number of peer assessments completed, social network degree, and an indicator for being an incoming university student at the institution offering the course), again suggesting that a limited, but diverse, feature space can effectively predict MOOC student success. However, the fact that this model was trained and tested on only a single MOOC -one which may be particularly unique, because it was offered with an incentive (early enrollment in a university biology major) to current or prospective students at the hosting instutition -means that further replication is needed to determine the extent to which these results are generalizable. We highlight similar issues with comparing different experimental populations in MOOC research in Section 5.2.
Other work which attempts to perform more fine-grained dropout prediction with the intention of performing early intervention includes Xing et al (2016), which uses an ensemble of C4.5 tree and Bayesian Network models built on a set of counting-based engagement features and a PCA-based approach similar to Kloft et al (2014). Baker et al (2015) find that early access to course resources in an e-learning history course, including a course textbook and its integrated formative assessments, provides accurate predictions of success or failure within the first two weeks of the course. In a pair of works which use more sophisticated temporal features to aid in early dropout prediction, Ye and Biswas (2014) and Ye et al (2015) find that fine-grained features related to either (a) temporal engagement with lecture quizzes or (b) the quantity of engagement with lecture quizzes improve models, but that once either (a) or (b) is included, adding the other provides no further performance gains. This result may suggest a plateau to the effectiveness of the features they evaluate, or it may highlight the need for suffifiently complex modeling techniques to learn the complex patterns in rich, granular feature sets.
A practical issues with "early warning" systems is that their predictions can change dramatically during the early stages of a course as the model predicts based on only small amounts of data. He et al (2015) uses a smoothed logistic regression model trained from a previous offering of a MOOC to make calibrated predictions on a future offering which where fluctuation of predicted dropout probabilities over time is minimized. This smoothing provides stable predictions of at-risk students for early intervention, a useful property for realworld implementation which allows the students tagged as "at-risk" to remain relatively stable over time.
In contrast to the simple feature appending approach used by e.g. Kloft Kloft et al (2014), which shows variable (and only slight) improvement over weeks as data accumulates, more sophisticated temporal modeling approaches have demonstrated the ability to improve predictions more rapidly and consistently using engagement-based features. Brooks et al (2015a) examine how a higher-order time series method improves by exploring its incremental changes in performance with each additional day of MOOC data; they demonstrate rapid model improvement over the first three weeks of each MOOC evaluated. Fei and Yeung (2015) explore sequential models, including a Long Short-Term Memory neural network (LSTM), which takes sequences of weekly activity feature vectors as its input. Fei and Yeung demonstrate consistent improvement in these models' performance as additional data is collected over course weeks, particularly over the initial weeks of a course. This model is directly compared to several others, outperforming (1) an SVM (for reference to Kloft et al (2014); however, Fei and Yeung utilize a different SVM kernel); (2) two variants of Input-Output Hidden Markov Models (compare to Balakrishnan and Coetzee (2013), which uses a different HMM variant); and (3) (2016)). The use of an LSTM is a promising approach, but further replication across a larger sample of courses is needed. Fei and Yeung's work also demonstrates how challenging it can be to compare results across machine learned models when exact replication of experimental populations and method is not possible (for example, Fei and Yeung cannot compare their model on data from Balakrishnan and Coetzee (2013), nor can they perfectly reproduce the HMM model implementation from only the published description; we discuss this issue in Section 5.2), but their effort to provide these reference points is still useful. Additionally, Fei and Yeung (2015) implement their model using three different definitions of dropout, which demonstrates the challenges of comparing predictive models of student success and also suggests the robustness of their results. We find there to be a particular need for further work which compares multiple feature extraction and modeling techniques within a single experiment. Only through such work can we continue the scientific process of not simply identifying novel predictive modeling methods, but comparing their effectiveness in order to identify the most effective techniques. Wang and Chen (2016) evaluate a Nonlinear State Space Model (NSSM) in comparison to several other models, including an LSTM, and suggest that the NSSM achieves superior performance.
Sinha et al. Sinha et al (2014b) also evaluate a rich set of activity features, using n-gram activity features in combination with higher-order graphical fea-22 Josh Gardner, Christopher Brooks tures (which represent the richness, repitition, and activity/passivity of students' interaction sequences) about students' interaction sequences to predict dropout. They also conduct the useful comparison of whether using features from the current week only vs. a students' entire history improves performance, finding that the full history does not provide a significant improvement over current week only features Sinha et al (2014b) (this result conflicts with Xing et al (2016), which finds that historical features improve the quality and stability of predictions in a single course offered on Canvas), but Sinha uses a larger and perhaps more representative sample of MOOCs.
Another innovative approach to representing and modeling activity-based features is presented in Zafra and Ventura (2012), where a multi-instance genetic algorithm is used to model "bags" of instances representing information about each students' activity across various behavior and resource types. This algorithm is particularly unique in its ability to resolve missing-data issues with sparse features (such as forum posts) available only for a small subset of learners (Gardner and Brooks (2017c)); the multi-instance algorithm accepts bags of varying sizes to accommodate the unique subsets of activities displayed by each student. Zafra and Ventura's experiment is conducted in the context of a set of e-learning courses offered via Moodle, but the authors argue that this approach is scalable and that it would be particularly useful for large online courses due to the heterogeneous student behavior patterns in these courses.
In a pair of works utilizing the same underlying feature set, Halawa et al (2014) and Kizilcec and Halawa (2015) explore the use of learner activity features for predicting dropout in MOOCs. In the first of these works, Halawa et al (2014) use a simple thresholding model to explore the use of counting-based learner activity features to predict dropout, theorizing that both observable learner activity and dropout are driven by unobservable "persistence factors" which students possess to varying degrees. Halawa et al show that this model is able to spot risk signals at least 2 weeks before dropout for over 60% of the students in their experimental population (students who joined in the first 10 days of the course and have viewed at least one video), suggesting that early dropout prediction may be tractable for this group. Building off of this work, Kizilcec and Halawa (2015) apply this analysis to a sample of 20 MOOCs, utilizing the same feature set with a simple logistic regression model. These works reinforce the finding that activity-based features can be used to construct effective predictive models of student success in MOOCs, and may be far more important than sophisticated predictive algorithms.
In a series of works, Veeramachaneni et al (2014); Taylor et al (2014); Taylor et al (2014); and Boyer and Veeramachaneni (2015) further demonstrate both the utility of effective feature engineering and how robust feature sets can be combined with effective predictive modeling techniques to build performant student success predictors.
Veeramachaneni et al (2014) explore crowd-sourced feature extraction, leveraging members of a MOOC to apply their human expertise and domain knowledge to define behavioral features for stopout prediction. The authors find that these crowd-proposed features are more complex and have greater predictive performance than simpler author-proposed features for all four cohorts evaluated (passive collaborator, wiki contributor, forum contributor, and fully collaborative). This work utilizes a simple regularized logistic regression for the predictive model, again demonstrating that many effective predictive models of student success in MOOCs have relied on clever feature engineering, not sophisticated algorithms. We note that their regularization approach is common in MOOC research (see 5.1 for details) due to the large number of correlated predictors used with activity-based models. Taylor et al (2014) applies the feature set from Veeramachaneni et al (2014) to explore a space of over 70,000 models using a self-optimizing machine learning system. However, the consideration of such a massive model space on only a single cohort of students virtually guarantees at least some success in prediction due even to spurious results alone. Further validation and testing of the "best" models identified in this work are needed. In many ways, this work is an extreme example of a common approach in educational data mining where large model spaces are explored without utilizing effective statistical techniques for their evaluation, resulting in performance data whose significance and generalizability is difficult to interpret. 8 (2015) explore transfer learning using a subset of this feature set. This work is notable for its experimental treatment of how previous iterations of a MOOC can be used to predict on future iterations, which is how such models are used in practice. This setup demonstrates the challenges of actually deploying such predictive models, and provides some initial data on effective transfer architectures for doing so. While many of the experimental results are inconclusive, Boyer and Veeramachaneni demonstrate two particularly important findings.

Boyer and Veeramachaneni
First, Boyer and Veeramachaneni find that a posteriori models -built retrospectively using the labeled data from the target course itself, which is the dominant experimental architecture considered in the work surveyed -presents "an optimistic estimate," and that such models "struggle to achieve the same performance when transferred" (we discuss potential issues with a posteriori models, and their prevalence across the work reviewed, in Section 5.2.3). They conclude: "when developing stopout models for MOOCs for real time use, one must evaluate the performance of the model on successive offerings and report its performance" (emphasis from original) Boyer and Veeramachaneni (2015). This and other work (e.g. Brooks et al (2015a); Evans et al (2016); Whitehill et al (2017))suggests that there is a great deal of work to do in replicating, re-evaluating, and exploring the generalizability of previous stopout prediction work performed using this architecture; we expand on this need in Section 5.2. For one example of work which compares models evaluated both within courses and across courses, see Wang and Chen (2016), which presents evidence that the "penalty" for model transfer across courses might be minimal.
Second, Boyer and Veeramachaneni find that an in situ prediction architecture transfers well, achieving performance comparable to a model which considers a users' entire history (which is not actually possible to obtain during an in-progress course). In situ architectures consider data and proxy labels from the same course to train a model (rather than true labels of future stopout, which are not known at the time of training/prediction in this realistic formulation of the task). This finding presents a possible approach to resolve the problems with using a posteriori modeling in practice, and is supported by other future work Whitehill et al (2017).
In a different examination of model transfer, we surveyed two works (Vitiello et al (2017b); Cocea and Weibelzahl (2007)) which examine how models trained on one MOOC platform transfer to another. Both demonstrate that high-performing features are stable even for models trained across different MOOC platforms. This suggests that effective activity-based feature sets may transfer well across MOOC platforms (when the data they require is available from these platforms), but further research is required to verify this result.

Explanatory Activity-Based Models
In this section, we survey work on a second class of behavior-based predictive models: explanatory models of student success. 9 In these models, experimenters fit prediction models with the goal of inspecting the model and learning about the underlying data-generating mechanism, not evaluating the predictions. This work is often conducted in ways that allow for stronger conclusions about the underlying associations between the predictors and outcome of interest, even if these claims are not causal.
This distinction reflect the "two cultures" of statistical modeling introduced by Breiman (2001) and discussed in Section 5.3.2: Section 3.1.1 reflected the "predictive modeling culture" while the current section reflects the "data modeling culture" in MOOCs. Each "culture" makes a unique contribution to our understanding of predictive modeling in MOOCs and therefore deserves its own treatment. Explanatory modeling work supports understanding of the underlying factors related to student success, including the direction, magnitude, and statistical significance of their association with the response. Such work typically employs far simpler models, favoring interpretability over predictive accuracy, and therefore warrants separate (or at least qualified) consideration. In this section, we discuss the literature with an emphasis on their explanatory results, not predictive results, in an effort to highlight the most relevant and useful aspects of these works.
Perhaps one of the most comprehensive works in this category is Evans et al (2016). In this work, the authors examine a sample of 44 MOOCs and over 2 million learners, evaluating both student and course traits for association with engagement and persistence. Four findings are particularly relevant to student success prediction in MOOCs. First, early engagement (such as registering more than four weeks prior to course opening, or completing a pre-course survey) is the strongest predictor of completion. Second, the steep dropoff in engagement is "very strong and nearly universal" across the courses examined, which provides evidence supporting the implicit assumption of generalizability across courses in many other works. Third, the title of individual lectures are associated with differing levels of engagement, with titles containing the words "intro," "overview," and "welcome" having significantly higher rates of watching. Fourth, the first offering of a course has significantly higher rates of completion than subsequent offerings -an important finding with implications for the real-world deployment of models learned on data from previous courses. Stein and Allione (2014) also take an explanatory modeling approach to evaluate learner behavior in a microeconomics MOOC. Reinforcing several other results discussed above, Stein and Allione find that early engagement -completing a quiz or a peer assessment exercise in the first week of a nineweek course -is a significant predictor of persistence in the course. They also evaluate a range of demographic factors for students who completed a precourse survey, finding that self-reported motivation for taking the course is not a significant predictor of completion, but that age is (with both young and very old students more likely to disengage). Stein et al. argue that their results demonstrate that "the attrition pattern is not uniform among all enrollees, but rather there are distinct sub-groups of participants who reveal their type early on" (Stein and Allione (2014), p. 2), suggesting that demographic information may be relevant to future explanatory and predictive models of student success in MOOCs.

Discussion Forum and Text-Based Models
Threaded discussion fora are a prominent feature of every major MOOC platform and are widely used in most courses. Detailed analysis of the data from discussion fora provides the opportunity to study several dimensions of learner experience and engagement which are not detectable elsewhere. This includes a rich set of linguistic (measured by analysis of the textual content of forum posts), social (measured by the networks of posts and responses, or actions such as up/downvotes), and behavioral features not available purely from the evaluation of clickstream data. Gardner and Brooks (2017b) and Gardner and Brooks (2017c) argue that understanding the individual contributions that separate data sources make to predictive models is useful in determining whether scarce developer time ought to be dedicated to feature engineering, extraction, and modeling from those sources; this is particularly relevant to the complex data in discussion fora.
In this and all remaining sections, we discuss both predictive and explanatory models together, due to the more moderate size of the literature surveyed and the fact that much of this work utilizes data models which are highly interpretable, even when used for prediction.
A foundational series of forum-based predictive work is that of Rosé, Wen, Yang, and collaborators Rosé et al (2014); Wen et al (2014b); Yang et al (2015), and particularly Yang et al (2013). At its core, this series of work uses discussion forum data to identify the social environmental characteristics that are most conducive to persistence or sustained engagement in a MOOC. In Yang et al (2013), Yang et al. use forum post data to explore the predictiveness of three types of features for forum posters in a single MOOC: cohort (the week in which a user joined the course), forum post (threads started, post length, content length), and social network (several metrics, including centrality, degree, authority, etc.). Of 16 variables considered in a variety of model specifications, Yang et al. find only three that are significant predictors for these students: being a member of cohort 1 (joining in the first week of the course), writing forum posts that are longer than average, and having a higher than average authority score are all associated with a lower probability of dropout. Rosé et al (2014) adds subcommunity membership to their set of predictors; in this case, cohort 1 membership is still significant, but their finding on authority is reversed -with a "nearly 100% likelihood of dropout on the next time point for students who have an authority score on a week that is a standard deviation larger than average in comparison with students who have an average authority score" -with the addition of membership in a single subcommunity identified via a Mixed Membership Stochastic Blockmodel (MMSB). This analysis suggests that the social factors, and not the language, of discussion fora may be more effective predictors of dropout for students who post in the fora. Wen et al (2014b) and Yang et al (2015) respectively extend this work to evaluate the linguistic features of forum posts to identify the learner motivation and the degree of confusion they express. Wen et al (2014b) use forum posts to derive (a) cognitive engagement features from the presence of unigrams in post text, and (b) human-coded learner motivation features. They find that these are significant predictors of dropout, using the survival modeling approach implemented in their previous work. Yang et al (2015) examines confusion in the text of forum posts, finding that the influence of confusion varies across courses, and that different types of confusion are significant predictors of dropout in each of the two courses evaluated. Yang et al. attribute this to differences in the domain of these two courses.
The work presented above, collectively, represents an important example of how student data -in this case, forum post text -can be used to generate higher-order features that may measure important elements of learning. It also demonstrates the usefulness of effective feature engineering, even when applied with relatively simple statistical models, in predicting student success in MOOCs. For further work on motivation classification and how it can be used to predict certification and surrogate exam-takers, see Xu and Yang (2016). Robinson et al (2016) apply natural language processing to pre-course open-response questions on learners' anticipated utility of course material. Using unigram features improves dropout prediction over a demographics-only model for students intending to complete the course. A series of richer features from the Linguistic Inquiry and Word Count (LIWC) framework (Pennebaker et al (2015)) are not found to be significant predictors of dropout. However, the final model in this work achieves a relatively low AUC (59.8) despite being evaluated using a post hoc architecture (cross-validated testing using the same course on which the model was trained) and analyzing a subpopulation which is less than 5% of the students who registered for the course, and less than 7% of the students who engaged with the course during the first two weeks. This suggests that the model is not particuarly well fit to the data, even given the small subsample of the course population used; further research would help identify the extent to which these results generalize to other populations. We note in Section 5.3.2 that the lack of fit from using simple, but interpretable, data models is an argument in facor of more complex (but less interpretable) models; we expect future work to continue the trend of bridging this fidelityinterpretability gap.
Dowell et al (2015) explore discourse features generated from forum posts, which are able to account for 5% of the variance in learner final grades (in contrast, a model with discourse features and participant features explains 93% of this variance). For the most active students (the top quartile, based on count of posts), discourse features explain 23% of the variance in performance. The authors conclude that discourse features are most effective at predicting performance for the most active students. Considering that forum posters might already be considered the most active and engaged students in a MOOC, these results suggest that the predictive usefulness of discourse analysis might be limited to a small subpopulation of learners in many MOOCs.
Crossley et al. compare the predictiveness of clickstream-based activity features and natural language processing features in Crossley et al (2016). Crossley et al. find that the clickstream-based activity features they evaluate are the strongest predictors, but that NLP features were also predictive; the addition of clickstream-based activity features improves the performance over a linguistic-only model by about 10% when predicting course completion Crossley et al (2016). While the sample size in this experiment is small and only consists of the small subset of students who both posted in a forum and completed an assignment, it makes a useful and important contribution to the literature by systematically comparing two of the dominant and most useful feature sets (activity and forum features). Further exploration, including systematic, statistical evaluation of the predictive efficacy of each feature set across larger course populations, is needed to validate these results and explore the degree to which they generalize. For an example of such a comparison, see Gardner and Brooks (2017c). Tucker et al (2014) apply natural language processing to explore the correlation between students' sentiment in posts about specific assignments and their performance on those assignments in an art MOOC, finding a modest negative correlation. They also find a modest positive trend in forum post sentiment over the duration of the course. Adamopoulos (2013) presents an alternative approach to using text analysis to understand student success in MOOCs by conducting textual analysis of public student reviews of MOOCs. Adamopoulos suggests that student course completion is influenced by perceived course quality, course characteristics (topic, perceived difficulty), characteristics of the offering institution (e.g. university ranking or prestige), platform characteristics (i.e. usability), and student characteristics (i.e. gender). This work matches other examinations of factors affecting student dropout in e-learning and distance learning courses (i.e., Levy (2007); Ji-Hye Park and Hee Jun Choi (2009)).
We conclude this section with a brief note. One of the particular challenges of working with text and forum data in MOOCs is the relative sparsity of this data: as optional activities, forum posts (as well as up-and down-voting, preand post-course surveys, and other questionnaires) are only available for the subpopulations which elect to participate in them. In most cases, this is a fraction of the population; sometimes as little as 5% (see Table 5.2.1). Therefore, work which utilizes these data sources typically restricts its experimental population only to the small subpopulation of students for whom this data is available. While this can still lead to interesting and informative insights about this subgroup, we believe that work which excludes 95% of the participants in a course ought to be considered either exploratory or very limited in its scope. This observation applies to a great deal of MOOC research, as we discuss in Section 5 below, but it is particularly problematic (and is least often acknowledged) in the category of work evaluated in this subsection.

Social and Cognitive Models
Our survey identified a distinct body of work which measures or predicts social relationships and cognitive states in MOOC learners, often through the use of behavioral data (including forum posts), performance tasks, or questionnaires. The data, features, and outcomes of interest in this work were sufficiently different to make a separate category useful in this analysis.
A substantial corpus more directly evaluates discussion fora as social networks, constructing networks from fora where students are nodes and various reply relationships constitute edges. For example, Joksimović et al (2016) uses two sessions of a programming MOOC, offered in English and Spanish, respectively, to evaluate the relationship between social network ties and performance (specifically, non-completion vs. completion or completion with distinction). Students who achieved a certificate or distinction were more likely to interact with each other than with non-completers (in contrast, Jiang et al (2014a) find in a different set of MOOCs that learners tend to communicate with others in different performance group). Furthermore, Joksimović et al (2016) find that weighted degree centrality was a statistically significant predictor of completion with distinction in both courses, and a significant predictor of basic completion in the Spanish-language course, while closeness and betweenness centrality showed more variable and inconsistent effects across courses. They conclude that structural centrality in the network appears to be positively associated with course completion Joksimović et al (2016). The finding matches that of Russo and Koesten (2005), who also identified centrality as a statistically significant predictor of student performance in a small e-learning course. In a related work, Dowell et al (2015) evaluate how social centrality itself can be predicted by text discourse features, finding that discourse features explain about 10% of the variance in performance (compared to 92% explained with a model using discourse + participant features); this increased to 23% explained for the most active participants in the fora. Yang et al (2014) explore the use of a graph clustering method which utilizes a probabilistic version of a Mixed Membership Stochastic Blockmodel (MMSB). They apply a Latent Dirichlet Allocation model to the subcommunities, constructing probabilistic models of students' social network membership over the subcommunities. Membership in some subcommunities defined by the MMSB are significantly predictive of dropout, while others are not; the number of subcommunities that are significant predictors varies between two and four across their three-MOOC sample (the authors consider up to 20 subcommunities per course). Other work has identified social networks as effective predictors of student performance in traditional academic courses (Fire et al (2012); Gašević et al (2013)). Sinha et al (2014a) uses activity data to infer cognitive states by generating an "information processing index" for each student based on an expertgenerated taxonomy of user interaction sequences defining various behavioral actions (e.g. "clear concept," "slow watching," and "checkback reference") and weights which the authors manually assign to each action group. Again, however, using manually-defined features risks injecting experimenter bias into the model instead of generating truly data-driven features in this model. Dillon et al (2016) use self-reported emotional states to examine the relationship between emotions and activity type; co-occurring emotional states; and the relationship between emotions and dropout. Anxiety, confusion, frustration, and hope are each significantly correlated with dropout. Initial work by e.g.  (2014)) utilizing sentiment analysis seems to also suggest that information related to emotional states captured from learner-generated text can be useful in dropout prediction. Gütl et al (2014) also evaluate learner emotions by administering questionnaires during learning activities, finding no significant difference between the relative proportion of happiness vs. sadness, anxiety, and anger between completers and non-completers. Russo and Koesten (2005) explore whether network centrality and prestige can predict "affective learning" -how students feel about a course -in an e-learning course, but find that neither is a significant predictor. While affect has been studied in K-12 education and in digital cognitive tutoring environments (e.g. Pardos et al (2013)), there is little research on emotions in MOOCs; we believe that the use of information corresponding to emotional states represents a useful line of inquiry for future work. Xiao et al (2015) and Pham and Wang (2015) use heart rate tracking on mobile phones to conduct "Implicit Cognitive States Inference," whereby MOOC learners' cognitive states are predicted from mobile phone measurements. In particular, this system purports to detect learners' mind wandering and interest/confusion. This work is a proof-of-concept, but given the growth of both mobile devices for learning and the expansion of sensors and multimodal learning analytics, this work points to potential future directions for student models that measure learners directly (not simply their navigation or submission behavior) and respond to real-time physiological, emotional, or cognitive feedback. Greene et al (2015) explores students' perceived relevance of course material, commitment, and students' implicit theories of intelligence, as well as demographic indicators and information about students' prior experience with MOOCs. Self-reported commitment is one of the strongest predictors of dropout in this work, but students' implicit theories of intelligence are not strongly associated with dropout. They also find that intended hours spent on the MOOC is a significant predictor of exam scores, but that implicit theory of intelligence was also not a significant predictor of exam score. Demographic variables, including age, prior education, and prior experience with MOOCs, are also found to be significant predictors of both dropout and achievement. We note that the relationship between intention and student success is reinforced in other work by Balakrishnan and Coetzee (2013), which measured intention by students' views of course progress pages; Gütl et al (2014) finds a high level of self-reported motivation for both dropout and non-dropout students.
Agudo-Peregrina et al (2014) examines interactions in online courses, finding that student-student, student-teacher, and student-resource interactions are all significantly related to learner performance in online courses, while the same interactions are not significant predictors in courses with an in-person component. Whie this work is not conducted in MOOCs, it does demonstrate how broader elements of student engagement with MOOC resources -including other students and teachers -might take on special importance in digital learning environments.
More research on the impact of each of social networks and cognitive states in MOOCs is necessary: these factors appear to be an important factor in students' learning, but are challenging to measure with existing data and even harder to do so with relatively small, single-course samples. In particular, we note that no existing research evaluates offline social data, and previous research appears to be overly reliant on discussion fora and questionnaires as sources of social and cognitive data, respectively. The examination of novel data sources on social and cognitive factors stands to substantially influence the research consensus in this area.

Performance-and Learning-Based Models
While the formal purpose of a MOOC is, broadly construed, for participants to learn, the use of learning-based features and outcomes in predictive MOOC models has been surprisingly limited as shown in Table 3. Much of the work in this section draws upon methods derived from the broader psychometrics, learning analytics, and educational data mining communities, applying wellknown methods (such as Item Response Theory or Bayesian Knowledge Tracing) to MOOC data.
One category of learning-based predictive work in MOOCs attempts to predict learning-based outcomes, rather than activity-based outcomes such as dropout. Some previously-discussed work in this category includes (Brooks et (2014)); such work commonly focuses on pass/fail, final grade, assignment, or exam prediction (further data on the use of learning-based prediction outcomes is provided in Figure ??).
Ren et al (2016) explore the use of "personalized linear regression" for predicting student quiz and homework grades in , finding that this approach ouperforms KT-IDEM, an item-level variant of Bayesian Knowledge Tracing widely researched in intelligent tutoring systems, in predicting homework scores across two MOOCs.
In another work which applies pre-existing learning assessment to online courses, Garman (2010) administers a commonly-used reading comprehension test (the Cloze Test) to students taking an e-learning database course. Garman finds that reading comprehension is positively associated with exam performance and overall course grade, but finds no association with online openbook quizzes and projects. Garman argues that this is because online tasks are more under control of the student (taken independently, with fewer or no timing constraints), while exams and course assignments occurred in an inperson environment with time constraints. While this study is administered in an e-learning course, these findings are relevant to MOOCs given the degree to which MOOC participants are expected to read and comprehend substantial amounts of text independently. Wojciechowski and Palmer (2005) also finds significant relationships between student reading comprehension (as measured by ASSET scores and ACT English scores) in university e-learning courses. Kennedy et al (2015) evaluate how prior knowledge and prior problemsolving abilities predict student performance in a discrete optimization MOOC with relatively high prior knowledge requirements. They Prior knowledge variables alone account for 83% of the variance in students' performance in this MOOC. The relationship between prior knowledge and student performance is well-documented in traditional education research, but is largely unexplored in MOOCs, despite the potential presence of many more students who lack prerequisite prior knowledge in MOOCs relative to traditional higher education courses; further research on both data collection (i.e., methods for efficiently measuring learners' prior knowledge) and on the impact of prior knowledge on learner outcomes is a useful avenue for future research. Time-on-task and task engagement are also student performance concepts which have been applied extensively to educational contexts outside of MOOCs. Champaign et al (2014) evaluate how learner time dedicated to various tasks (assignment problems, assessments, e-text, checkpoint questions) correlates with their learning gain and skill improvement in two engineering MOOCs. They find negative correlations between time spent on a variety of instructional resources and both skill level and skill increase (i.e., improvement in students' individual rate of learning), using assessments calibrated according to Item Response Theory. Champaign et al. find these results "obviously discouraging", but their evaluation is based on bivariate correlations; they note that the observed association is likely due to struggling students spending more time working with learning activities. A more fine-grained analysis is needed to determine whether the results are instead indicative of other behavior, such as productive struggle. This work certainly suggests that further evaluation is necessary to measure whether students are truly learning in MOOCs (as opposed to high-skill students succeeding, while low-skill students drop out) and what types of resources and affordances best support learning (Champaign et al (2014)). Cocea and Weibelzahl (2007) also address the task of evaluating students' engagement with content and explore the task of student engagement prediction in online tutoring programs; this work demonstrates that accurate engagement predictions (based on expert-rated engagement) can be made using relatively simple activity features extracted from log files. Koedinger et al (2015) examine the impact of using interactive educational resources in MOOCs versus using passive informational resources (videos, text) available in many MOOCs. They find that learners using more interactive resources learn significantly more than those who read more text or watch more videos, estimating the impact of a 1-standard deviation increase in interactive resource use to be more than six times that of a 1-standard deviation increase in watching or reading. However, they find that the use of interactive resources was not a significant predictor of dropout, with quiz scores and quiz participation instead being significant predictors. This suggests that while these resources may indeed assist students in learning more, this may not translate directly into course completion. This work highlights the importance of evaluating results along multiple outcome dimensions in MOOCs.
DeBoer and Breslow (2014) find that time spent on homework and labs predicts higher achievement on assignments, while time spent on the discussion board or book is less predictive or not statistically significant. Additionally, time on the ungraded in-video quiz problems between lecture videos is found to be more predictive of achievement than time on lecture videos themselves. Ashenafi et al (2015) and Ashenafi et al (2016) examine models for student grade prediction which only use peer evaluation; these models are applied in traditional courses with web-based components but the authors argue that their findings are also applicable to MOOC contexts. Peer assessment is used extensively in MOOCs (Jordan (2015)) and its predictive capacity is largely unexplored.
Brinton and Chiang (2015) explore using clickstream data to build models of whether learners are Correct on First Attempt (CFA) in answering questions in a MOOC. After building models to predict CFA, CFA predictions are used as features in a model to predict student quiz performance. Brinton and Chiang demonstrate potential performance gains from this approach, suggesting that not only effective feature engineering, but also the predictions of intermediate predictive models, can improve predictions of student success in MOOCs.  extends this work with a sequence-based input approach. Sinha and Cassell (2015) also explores sequence-based approaches to student learning, instead modeling the outcome as a sequence, predicting sequences of student grades using Conditional Random Fields. Kotsiantis et al (2010) apply various incremental algorithms to student performance prediction, finding that an ensemble of incrementally-trained predictive models can achieve improved final exam pass/fail predictions over the base learners. While this model is applied to a single higher education distance learning course, it demonstrates a successful application of a technique -incremental model training, requiring only a single pass through large datasets -which may be particularly useful with the massive datasets in MOOCs. Further exploration of these techniques stands to make real-time training and prediction more tractable. Sanchez-Santillan et al (2016) also explores the use of incremental interaction classifiers using Moodle course data.

Other Models: Demographics, Questionnaires, and Higher-Order Features
In this section, we explore various predictive modeling experiments which do not fit into the groupings of previous sections. These include works which evaluate models based on demographics and higher-order features.

Demographics-and Questionnaire-Based Models
Several works have investigated the relationship between learner demographics and their success in MOOCs. Similar to research in more traditional educational contexts, the primary focus of this research is in understanding for which groups of students MOOCs may be more or less effective. In general, this work therefore tends toward explanatory or data modeling.
In an analysis of edX's first course, DeBoer et al (2013) find that having taken differential equations (a recommended prerequisite for the course), having a parent who is an engineer, and working with the teacher offline are significant predictors when controlling for other behavioral and academic factors; they do not find a relationship between gender and achievement for the survey completers examined. This finding regarding prior knowledge reinforces Kennedy et al (2015), discussed previously. Similar,y Dupin-Bryant (2004) show that prior computer experience is also a predictor of retention in online distance education courses, likely because such students are better prepared to learn and engage with course content by computer.
Specific findings related to various demographic features are multifarious, but comparing the magnitude of findings across different studies can be challenging when different controls are included in various models. Stein and Allione (2014) find that age is a significant predictor of MOOC completion. Greene et al (2015) also find age, prior education, and prior experience with MOOCs to be significant predictors of both dropout and achievement. Reich (2014) finds that intention to complete is a stronger predictor than any of several demographic traits measured across a sample of 9 MOOCs. In a review of works surveying potential causes of MOOC dropout rates, Khalil and Ebner (2014) also find that lack of time, lack of motivation, lack of interaction, and "hidden costs" (such as paid textbooks needed for reference, or paid certificates of which learners were unaware) contribute to MOOC dropout .
Street (2010) reviews eight different studies of factors for student dropout of distance learning courses. Street finds that several internal factors (selfefficacy, self-determination, autonomy, and time management), external factors (family, organizational, and technical support), and course factors (relevance, design) all significantly impact learners' decisions to persist or drop such courses. Other work surveying participants in e-learning courses finds similar influence of family support, organizational support, relevance, and other individual characteristics on individuals' decisions to drop out in this context Ji-Hye Park and Hee Jun Choi (2009). Gütl et al (2014) directly surveys 134 students who did not complete a MOOC and find that only 22% of these dropouts actually intended to complete the course. 69% of students reported that changes in job responsibilities during the course contributed to their dropout, and 70% reported that it was too difficult to work and study at the same time. No other factor was reported as a contributor to dropout by more than 50% of their survey population, suggesting that the tension between work responsibilities and low-stakes MOOC coursework is strong in their survey population.
Several works have evaluated the predictiveness of demographics in elearning or distance learning courses. While these are not directly analogous to MOOCs, the conclusions of such research can suggest useful starting points for further research into demographic and other factors which may contribute to MOOC dropout. For example, Willging and Johnson (2009) use a postcourse survey to understand explanatory factors underlying student dropout in an online human resources masters program. The authors find that demographics are not associated with dropout in the courses evaluated, and that reasons for dropout vary considerably by individual including personal reasons, job-related reasons, program-related reasons, and technology-related reasons. Brooks et al (2015b) examines whether demographics can improve the predictive performance of activity-based predictive models, showing that demographics "have minimal predictive power when determining the academic achievement of learners enrolled in MOOCs." In particular, Brooks et al (2015b) demonstrates that demographics-based models underperform activity-based models in MOOCs even early in the course when activity data is minimal, and that demographic features provide no discernable improvement over activityonly models (and actually degrade their performance in the second half of the course, as activity data accumulates). This stands in contrast to prior machine learning research in other educational domains, which suggests that demographics may be strong predictors of online course performance in traditional distance learning contexts (e.g. Kotsiantis et al (2003)). The work of Brooks et al highlights how the complexity and heterogeneity of MOOC learners require new and potentially more sophisticated student models, and how demographic findings may be less powerful than other unique, rich sources of data available in the contexts of MOOCs.

Higher-Order Feature Engineering
Other work, utilizing more complex feature types, has also begun to emerge in MOOC research. This includes explorations of higher-order n-gram representations of learner activity data, which has demonstrated promising predictive performance (e.g. Brooks et al (2015b,a); Li et al (2017)). In activity-based n-gram approaches, features are assembled using counts of unique sequences of events or behaviors; these features are then used to construct supervised learning models. This allows for the construction of large feature spaces which capture complex temporal patterns, and the frequency with which they occur. If sequences of behaviors, and not simply counts of individual behaviors, contain important information about learning behavior relevant to the prediction outcome, then these approaches seem appropriate.
As discussed above, Sinha et al (2014b) use n-gram features with a graphical model, and demonstrate that they can achieve reasonable predictive accuracy with only a single week of historical data.
We previously discussed Coleman et al (2015), which applies topic modeling to sequences of learner data to learn "profiles" of MOOC learners based on their activity sequences ("shopping", "disengaging", and "completing") . Each of these works and other sequence-based approaches discussed above (i.e., Balakrishnan and Coetzee (2013); Wang and Chen (2016)) can be thought of as capturing a temporal element of MOOC data. We argue in Section 5.3.1 below that further work in this vein is needed.
As we discuss below, feature engineering (not predictive modeling algorithms) is the primary driver of improvements in predictive modeling in MOOCs to date; future work should continue to pursue higher-order or other unique feature engineering approaches which capture information relevant to student success.

Synthesis, Research Gaps and Opportunities
In this section, we present high-level synthesis and conclusions from the survey above, including trends related to both the methods and the findings of pre-dictive modeling work in MOOCs to date. Additionally, we highlight potential gaps, both in terms of methodology and research questions. We conclude by framing these and other gaps as opportunities for future research.

Synthesis: Trends in Predictive Models of Student Success in MOOCs
In this subsection, we provide data on several dimensions of the predictive modeling research surveyed, profiling the data sources, methodologies, and experimental populations evaluated in these works. We find (a) clear evidence of research into specific MOOC platforms and raw data sources, and (b) a group of dominant methodologies (activity-based features, tree-based and generalized linear modeling algorithms), which suggest a need for future research comparing these methods (particularly when considered in light of the many different success metrics used to evaluate these models across works surveyed).

Platforms, Raw Data Sources, Feature Engineering Methods
Little attention has been paid to the data sources used in predictive modeling research in MOOCs, but (a) understanding which data sources are effective for prediction, and which are unexplored, provides a useful foundation for future work; and (b) because feature extraction requires significant expense, both in terms of development time and computation time, recognizing which data sources are most useful can improve the efficiency of real-time predictive modeling. We provide an overview to this end. Figure 6 demonstrates that clickstreams are the dominant data source for predictive modeling research in MOOCs. In one sense, this is unsurprising: clickstreams provide rich, granular data that the field is only beginning to harness the ability to represent in its full complexity. On the other hand, clickstreams are raw, semi-structured text files that require extensive human and computational effort to parse. Often, their formats are not well-documented, and there can be several levels of aggregation applied to a given entry (i.e., both session and user IDs, such that aggregating at both levels is not possible). In contrast, the other data formats shown in Figure 6 are typically provided as structured relational databases that can be access with simple Structured Query Language (SQL) statements. The fact that clickstreams are so widely used, despite these barriers to accessing and utilizing this data, is a testament to their usefulness in predictive modeling. Gardner and Brooks (2017c) evaluates features generated from different data sources, comparing the predictiveness of clickstream features vs. forum-and assignment-based features; this work verifies that clickstream features are more effective predictors than forum-or assignment-based features when predicting dropout across the entire population of learners in a large sample of MOOCs.
We observe a growing "long tail" of additional data sources, which represents a continued trend toward combining other data sources with MOOC data to gather a more complete picture of learners. We believe this is a worthy and useful goal, but the privacy-protected nature of learner data makes it difficult to combine with other sources. Finally, we note that the discussion in Section 5.2 is relevant to the use of clickstream data. While clickstreams contain complex, potentially useful temporal information about learner behavior over time, most modeling has been limited to simple counting-based representations of these temporal patterns (with few exceptions; i.e. Fei and Yeung (2015); Brooks et al (2015a)). Much of the complexity contained in these interaction logs has likely not been captured in any research to date. We provide data on the MOOC platforms evaluated across work surveyed in Figure 7. These results reflect the dominance of the two largest MOOC providers, Coursera and edX (Shah (2016)). We note that as non-English MOOC platforms, such as the Chinese platform XuetangX, continue to grow, they should be researched more extensively: a substantial segment of the populations who stand to benefit most from global access to MOOCs are non-English speaking, and these learners are likely to differ from the population of English-speaking course takers.
Feature engineering from these data sources is a focal point of MOOC research, and many advances in predictive modeling have hinged on clever or state-of-the-art feature extraction techniques, even when strikingly simple models are used. For example, Veeramachaneni et al (2014) combines a comprehensive set of features with a simple penalized logistic regression model; subsequent work demonstrates that this model is capable of state-of-the-art prediction accuracy despite its algorithmic simplicity Taylor et al (2014). As mentioned in our introduction to Section 3, there is a clear consensus that feature extraction is important to predictive modeling in MOOCs, and that future Research on various platforms largely reflects the distribution of learners across these platforms. Note that certain "platforms" (e.g. HarvardX, XuentangX) may use software of vendor platforms (notably, edX) but do so in a way which is independent of that vendor platform.
work should continue these investigations into feature engineering. Additionally, work which compares the predictive usefulness of various feature sets in a rigorous, experimental way -as in Crossley et al (2016) with activity-and NLP-based features, and Brooks et al (2015a) with demographics and activity features -will be particularly useful as feature engineering continues to diversify. As Sinha et al. note, "[t]he biggest limitation of most of these emerging works is that they focus solely on discussion forum behavior or video lecture activity, but do not fuse and take them into account" (Sinha et al (2014b), p. 1) -the focus on using individual groups of features is holding back predictive modeling research in MOOCs. Figure 5 shows the broad feature-based categories used for taxonomizing the work surveyed here. This data clearly demonstrates the dominance of activity-based feature extraction approaches. Two main factors explain this dominance: first, activity data is simply the most prevalent and fine-grained data available from MOOC platforms, and there are rich, complex patterns embedded in this data that the scientific community has correctly identified as important to explore. Second, activity-based outcomes (i.e., dropout or stopout) have been a focus of MOOC research, as shown in Table 3. Activity features seem a necessary (if not sufficient) set of features for the task. As research begins to explore other outcomes (such as learning), and as feature extraction becomes a less labor-intensive task perhaps due to open-sourced code or open MOOC data analysis frameworks, it is likely that feature engineering will increasingly utilize other feature types either in addition to or instead of activity-based features.

Modeling Algorithms and Model Evaluation Metrics
The statistical models used to map features to predictions are a core component of predictive student modeling in MOOCs, but there is little prior synthesis of the findings of which algorithms are most widely used. Figure 8 provides two perspectives on the modeling algorithms used across our survey.
First, the top panel, Figure 8a shows that tree-based models and generalized linear models are the most common techniques for predictive modeling in MOOCs. The prevalence of tree-based algorithms reflects several useful properties of these techniques: tree-based models can handle different data types (i.e., categorical, binary, and continuous) and are less susceptible to multicollinearity than linear models; they are relatively fast and simple to fit; tree-based models are nonparametric, and make few assumptions about the underlying data while providing highly flexible models; and the results of these models are highly interpretable by inspection of decision rules, variable importance metrics etc., and by visualization. Figure 8a also shows that generalized linear models (GLMs) are also popular for MOOC learner modeling. This reflects several benefits of these models, in particular: GLMs empirically have achieved excellent performance across many large-scale MOOC modeling experiments; they are fast and simple to fit to data, requiring little or no hyperparameter tuning; and they produce interpretable output (which is quite different from tree-based models), including coefficients representing the magnitude and direction of association between each predictor and the response, and the statistical significance of these predictors.
Second, the lower panel, Figure 8b, shows the specific algorithms used across work surveyed, essentially disaggregating Figure 8a. Figure 8b shows how the dominance of tree-based algorithms largely obscures the lack of unformity on which specific algorithms are used; of all tree-based algorithms considered, only random forests were used in more than 10 works surveyed. This makes it difficult to evaluate the effectiveness of any specific tree algorithm across our survey. In contrast, there are relatively few GLM algorithms adopted in the literature; logistic regression (LR) and L2-penalized logistic regression ("ridge" regression, L2LR) account for almost all use of GLM algorithms. As noted above, GLMs, and L2LR in particular, generally achieve excellent performance when used with large and robust feature sets, despite their strong parametric assumptions about the underlying data.
Finally, Figure 8 clearly reveals that there is a "long tail" of modeling techniques represented in the work surveyed here, with nearly half of the work surveyed using an algorithm which is not utilized in any other work (represented by 'Other' in Figure 8a and 8b). In part, this represents an emphasis on novelty in published academic research; this is also indicative of a nascent field which has little consensus on the best approach to solving its prediction problems. We note that none of the algorithms in the work surveyed demonstrate performance which consistently exceeds all other algorithms, suggesting that there is indeed no single "best" algorithm a priori for a given task or dataset Wolpert and Macready (1997). Future work which compares and evaluates the fitness of various predictive modeling algorithms for different tasks in MOOC research would be appropriate at this stage; we advocate such work in Section 5.3 below.
We do note that supervised learning approaches dominate the literature, with few examples of unsupervised approaches; this is likely due to the fact that many of the outcomes (i.e., dropout, certification, pass/fail, grades) are observable for all learners, making unsupervised techniques unnecessary for many tasks.
Our data also reveal a considerable lack of agreement about which model evaluation metrics to use in MOOCs, shown in Figure 9. Compared to the analysis of algorithms and data sources above, this data reveals a slightly stronger consensus around a smaller set of evaluation metrics, most notably accuracy (ACC), Area Under the Receiver Operating Characteristic Curve (AUC), precision (also called positive predictive value) (PREC), and recall (REC) (also called true positive rate, sensitivity, or probability of detection). Strictly speaking, a diversity of metrics is not a problem -different metrics measure different aspects of predictive quality, which vary depending on the task and research goals -but this lack of a consistent baseline leaves readers unable to compare performance across otherwise-similar studies which report different performance metrics. Reporting several metrics would often give a more complete picture of model performance and allow for easier comparison across studies, while still allowing researchers to examine performance according to their metric(s) of interest. Open data or open replication frameworks would allow for more nuanced comparison and would shift the burden from purely on the researcher, to allowing reviewers and critical readers to inspect results using any performance metric of interest.
We note that 9 of the works surveyed -over 10% -report classification accuracy as the only model performance metric, and we consider this practice particularly concerning. Accuracy is useful and interpretable for many readers, but it can be a misleading measurement of prediction quality with highly imbalanced outcome classes; this scenario is very common in MOOCs (i.e., most students drop out, do not certify, etc.). Accuracy is also threshold-dependent, while other metrics, such as Area Under the Receiver Operating Characteristic, measure performance over all possible thresholds. While accuracy is useful as an interpretable metric for readers, it is often difficult to assess the value of work which reports performance only using accuracy. The practice of only reporting accuracy should be discouraged, as computing additional performance metrics from the data used to compute accuracy (namely, predicted labels and class labels) requires minimal additional effort (sensitivity, specificity, F1, Fleiss' Kappa, and several other metrics can be computed from these labels).
Together, the data source, feature extraction method, statistical modeling algorithm, and evaluation metric reflect the accuracy dimension of predictive student models introduced in Figure 2. A key element of any usage of predictive student models requires that these models are effective at predicting the outcome of interest; research into -and methodologial progress in -each of these areas (feature extraction, modeling, and evaluation) stand to substantially improve the accuracy of future predictive MOOC models.

Methodological and Research Gaps
In this section, critically review the experimental methodologies used in predictive models of student success in MOOCs. In particular, we highlight (a) areas where the methods of prior research or the interpretation of its results are biased toward specific populations or overly prone to statistical errors; and (b) opportunities for future research toward modeling and understanding learner behavior in MOOCs. Because these issues are often two sides of the same coin (methodological gaps imply future research opportunities), we discuss them together.

"Small" Data and Experimental Population Filtering
Many of the challenges discussed in this section point to the difficulty of comparing findings across prior predictive modeling research in MOOCs. Because many courses are at least superficially similar to each other and offered in similar contexts, this type of comparison is theoretically possible in MOOCs; making these comparisons would be neither unreasonable nor difficult, and would allow evidence for or against specific predictive modeling techniques to accumulate across experiments. However, under existing research methods, these experimental populations are often not comparable.

Study Subpopulation % of Enrollees Perna et al (2014)
Registered after official course start date and no more than two months after course end date 90% Halawa et al (2014) Joined in the first 10 days of the course and have viewed at least one video Not Reported Fei and Yeung (2015) Students with at least one interaction as measured by 7 features used 46.7% He et al (2015) Students who submitted at least 1 assignment each week.

13%
Greene et al (2015) Completed pre-course survey and completed the first end-of-unit exam.

11.4%
Yang et al (2013) Posted at least once in discussion forum by seventh course week.
6.3% Robinson et al (2016) Started in first two weeks; completed pre-course survey; saw utility value of course; fluent in writing English; intends to complete course; and wrote more than one word on survey.
<5% Table 3 A sample of experimental subpopulations in works surveyed. Percentage of total enrolled students is shown. Such divergent filtering criteria and small, nonoverlapping subpopulations make comparing the results of different predictive work difficult.
Researchers' use of small, highly-subsetted experimental populations in their work is one way in which this generalizability is limited. Often, predictive experiments in MOOCs identify a limited population of learners on which the analysis is conducted; unfortunately, these populations are often so divergent that the results from one experimental subpopulation to another could be entirely different. A sample of some experimental subpopulations is shown in Table 5.2.1. We note that 40 of the works surveyed, or 46%, filtered the sample from the total available population of registrants or participants in some way.
Several comments are warranted here.
First, there is tremendous diversity in the subpopulations evaluated by different works, and it is difficult, if not impossible, to compare findings of otherwise-similar experiments across studies. We simply do not know, for example, how the students who submitted at least one weekly assignment in He et al (2015) might compare to students who watched a video and submitted a problem in Li et al (2016b) or to those who completed the survey with the relevant characteristics considered by Robinson et al (2016).
Second, filtering the subpopulation so significantly -as Table 5.2.1 shows, many experiments which share this data reveal that over 80% of MOOC participants are excluded from their analysis -moves such research further from the goal of understanding large segments of the learner population. While it is useful, for example, to know how natural language features and unigram frequencies in Robinson et al (2016) can be used to predict persistence, this data is of little practical use if it only applies to fluent English-speakers who see the value of the course, intend to complete it, and completed a pre-course survey with more than one word. Indeed, we would expect such learners to be quite different from the average or overall population in such a course, and  such work gives us little information about the broader population. 10 Previous educational data mining research suggests that affect detection models, for example, do not transfer across even regional and demographic boundaries within the United States (Ocumpaugh et al (2014)); it seems even less likely that models trained on subpopulations of globally diverse MOOC learners would generalize effectively across behavioral subgroups. At the very least, presenting the results both in terms of a small subpopulation and in terms of the entire course population would provide a useful point of reference. Third, these highly-subsetted experimental populations are often themselves a "sample of samples", evaluating just one or a few MOOCs which may or may not be representative of the larger population of MOOCs. Conducting research using these types of populations make it difficult to determine how such work might generalize even to the same subpopulation in other courses. Table 4 shows that over 50% of the works surveyed evaluated just one MOOC, with fewer than 20% of these works evaluating 10 or more courses. At best, highly-subsetted populations of an already narrow sample (of the overall MOOC population) can be taken as promising avenues for future research; it would be a mistake to consider the findings of such research fully resolved conclusions. We should be particularly concerned about works which publish "statistically significant" results for single-course populations in a small and highly-contingent subpopulation: these analyses can be subject to high "researcher degrees of freedom" (Gelman and Loken (2013)), and the extent to which these degrees of freedom were exploited during data analysis is rarely reported in published research. This bias may be compounded when predictive models are evaluated on the same course that is used to fit them (Boyer and Veeramachaneni (2015); Whitehill et al (2017)).
Having discussed the limitations and challenges raised by work which utilizes such specific experimental populations, we recognize, of course, (a) the value of such research, even with its limits; and (b) the reasons for doing so in practice. Our intent is not to single out the authors of any one particu-lar study; indeed, the fact that this applies so broadly to many of the most highly-cited works in the field suggests that even many of the most substantial contributors to the field have conducted such research. Many of these works reflect early exploratory research into MOOCs, and were initial efforts at understanding any cross-section of this novel population. We simply note that these limits are often not acknowledged by the broader research community when interpreting these results, and that further research is needed to explore the generalizability of these findings and ensure that the field's knowledge base is constructed on firm ground as the field grows and matures.

Model Evaluation, Comparison, and Replication
A second area where substantial research gaps exist is in the evaluation, comparison, and replication of the predictive models of student success in MOOCs. As work on predictive modeling has expanded across all domains, a substantial research base has emerged on techniques for comparing and evaluating the results of predictive modeling experiments. We find that the work surveyed often lags behind these accepted standards and methods for practice, which can be applied to predictive models in any domain. This also raises concerns about the accuracy dimension of these models, particularly when applied to unseen data.
There is concern in the broader statistical community about issues of multiple comparisons in model evaluation, particularly when applied to large spaces of potential statistical models. The field has begun to move beyond these concerns through to the adoption of simple (if conservative) tehniques for accounting for the many comparisons performed over the course of an experiment (i.e., the methods of Bonferonni or Benjamini and Hochberg (1995), or techniques specific to the evaluation of machine learning models outlined in Demšar (2006)). Bayesian methods have also been increasingly adopted for inference and data analysis, in part due to their robustness in cases of multiple comparisons Benavoli et al (2016);Gelman et al (2009).
However, almost none of the work surveyed utilized appropriate significance testing techniques (according to the standards of Demšar (2006) or Benavoli et al (2016)). Molina et al (2012) was the lone exception, based on our reading of these works, but Monlina evaluates traditional courses managed in Moodle, not MOOCs. We also found no acknowledgement of concerns about multiple comparisons in interpreting the statistical significance of results in any work. This lack of concern exists in spite of the fact that 18 of the works surveyed (20%) reported evaluating more than 20 models as shown in Table 5, which means that at least one Type I Error would be expected at a 5% significance level using a traditional hypothesis test to compare such models. It is possible that works which reported fewer than 20 models evaluated additional predictive models in the course of their experiments, exposing these experiments to an inflated risk as well.
There are clear reasons why these methodological concerns emerged in the first place. First, a complete lack of testing fails to quantifiably evaluate the findings of a predictive model. Some form of evaluation is needed to quantify the degree to which we might attribute observed differences in performance to chance versus to a "better" model or feature extration technique. This is especially important given the small samples of courses in works surveyed shown in Table 4. Second, even when statistical testing is used, often these tests require specific corrections when applied for predictive model evaluation. Many common statistical tests, such as the Student's t-test or ANOVA, are not appropriate or calibrated for testing predictive models Demšar (2006); Dietterich (1998). It is possible that the concerns which motivated these approaches may have been realized in many of the works surveyed 11 . Even the large number of models reported in the work surveyed (note that at least 1 3 of the works did not report the total number of models evaluated) suggest that inferential errors caused by uncorrected multiple comparisons may lurk in the current knowledge base of student success models. This lack of replicability of most work (lack of open code and data, as described in Donoho (2015)), combined with the "file drawer problem" wherein null results rarely published Rosenthal (1979), make it particularly difficult to determine when these Type I errors may have occurred or how prevalent they may truly be in the field of predictive MOOC modeling.
While the appropriate technique(s) for model evaluation vary based on the nature of the comparison (i.e., two models vs. many models; a single dataset vs. many datasets), these procedures do exist and are often simply ignored in predictive modeling research in MOOCs. These procedures are discussed in detail in a future work regarding predictive model evaluation in MOOCs; we refer the reader to that work or to (Benavoli et al (2016);Dietterich (1998);Demšar (2006)) for further details.
Particularly relevant in this discussion are the specific limitations of using the results of cross-validation to compare and draw inferences about the performance of predictive models. Average cross-validation performance was used to evaluate and compare the performance of 31 studies, or nearly 40% of the work surveyed. Again, problems with this procedure have been well-studied. Utilizing average cross-validated performance with unadjusted statistical tests (such as a paired t-test) makes such research susceptible to both high Type I error rates (higher than expected probability of concluding that there is a significant difference in performance when none exists) and low power (low ability to discern true differences in performance when they do exist) (Dietterich (1998); Bouckaert and Frank (2004)). These issues are discussed specifically in the context of model evaluation in MOOCs in Gardner and Brooks (2017c,a) Works which evaluate large model spaces -effectively conducting large numbers of multiple comparisons -inflate this risk. This exposes predictive modeling research in MOOCs to serious and preventable concerns about the reproducibility of its findings, at a critical time when the field's work is growing in both visibility and practical significance.
Taken in sum, the two challenges outlined above -multiple comparisons and a lack of rigor in model evaluation -point to a third challenge in predictive research in MOOCs: replication. We note with some concern that there is a dearth of replication research in MOOCs in particular, and in the field of education in general (Makel and Plucker (2014)). This means that despite the concerns, outlined above, about the inferential and sampling procedures often used in MOOC research, we are unable to estimate the impact of these procedures on the generalizability of many published findings. Of the work surveyed for this evaluation, none was a replication of prior work by new authors, although in limited cases (a) original authors reproduced their analyses on new MOOC datasets (i.e., Rosé et al (2014), or (b) new authors attempted to at least compare their work to algorithms used in others' work as a baseline (i.e., Fei and Yeung (2015)). An initial attempt at replication in MOOC models is shown in Andres et al (2016a,b), but these works replicate predictive models as relatively limited production-rule analyses and do not replicate the predictive models themselves (i.e., by controlling for covariates). Exact replication should be relatively more tractable in MOOC research than in other fields: MOOC data is largely consistent within (and even across) the two major platforms, Coursera and edX. The largest apparent barriers to replication are (a) lack of access to data; (b) lack of clarity in published descriptions of experimental methods; and (c) the lack of incentive to replicate previously-published research.
Particularly with the rapid proliferation of different approaches to predictive modeling in MOOCs, replication would provide a useful basis for comparison. Work that exactly implements the methods from another experiment has been called direct replication by Donoho (2015)); such research is extremely uncommon in educational research Makel and Plucker (2014) but is needed. For example, while existing studies might all use a support vector machine (SVM) and comapre these results, these comparisons ignore important differences in hyperparameter tuning, kernel selection, regularization, and feature selection which can have genuine effects on the performance of these algorithms across experiments. Future work which evaluates predictive models using multiple outcome metrics would also give a more complete picture of their performance (i.e., Fei and Yeung (2015)), even when authors cannot or choose not to openly share their code or data for replication.
In conclusion, the generalizability of many results in the work surveyed is seriously called into question. Future work which (a) adopts more effective inference and evaluation techniques which are robust to multiple comparisons and are appropriate for the evaluation of predictive models and (b) replicates the findings of prior work on new, larger data would make a valuable contribution to the pursuit of robust and generalizable knowledge about predictive modeling in MOOCs.

Realistic Experimental Contexts
A second area in which methodology and experimentation in MOOC modeling stands to grow is the context in which predictive modeling experiments are conducted. In particular, we advocate the use of realistic experimental contexts in future work; the state of the practice largely procuces models which are not actionable.
Building actionable predictive models to support downstream support and intervention is the stated aim of much of the work surveyed -these works often explicitly describe the aspirational use of their predictive models as the linchpins of "early warning" systems for "at-risk" students. Some works surveyed describe planned or hypothetical interventions based on such models; one utilized a student-initiated micro-commitment intervention and explored using student commitment as a predictor of assignment submission Cheng et al (2013), and a single work surveyed actually utilized predictive models for live adaptive interventions Whitehill et al (2015).
However, much of the work surveyed is simply not possible to implement in an active course -we call such experimental contexts not realistic. A realistic context matches the situation in which predictive models would be employed for active use in MOOCs, particularly with respect to the information available at the time of prediction. Many utilize post hoc prediction architectures, where (a) model-fitting requires labels which are not knowable until a course completes, and (b) model evaluation takes place by evaluating test predictions made on the same course used for training -not a disjoint future course. These contexts do not match those in which a real-time predictive model would be used: for example, dropout labels are not known at the time of training and prediction if a course is still in progress; by definition, a users' dropout status is not knowable until the course completes.Of the works surveyed, only Ashenafi et al (2016); Bote-Lorenzo and Gómez-Sánchez (2017); Boyer and Veeramachaneni (2015); Brooks et al (2015a,b); He et al (2015); Kizilcec and Halawa (2015); Wen et al (2014b); Whitehill et al (2015Whitehill et al ( , 2017) -fewer than 10% -examine prediction architectures in which the test predictions could be made for an incomplete course (either by training and predicting on different iterations/courses and using transfer learning, or using some form of proxy labeling).
The degree to which the prediction context, particularly same-course evaluation vs. future-course evaluation, may bias results is unclear. Veeramachaneni et al (2014) find that predicting on future courses generally achieves lower performance than same-course prediction, and that second-to-third transfer is more accurate than first-to-third. Whitehill et al (2017) examine a variety of predictino architectures and find that while same-course (post hoc) prediction architectures optimistically bias estimates of model performance, in situ proxy labeling achieves comparable performance. He et al (2015) find that "prediction models trained on a first offering work well on a second offering", with such models achieving an AUC of 0.8 using only one week of data when predicting on a future iteration. Evans et al (2016) shows that users engage with later runs differently from the way they engage with earlier runs in an analysis of 44 MOOCs, suggesting that such transfer would be less effective than with a model trained on another non-first run (i.e., training on second iteration, predicting on third iteration). Model transfer to future courses is also evaluated in Brooks et al (2015a), which achieves an AUC of 0.9 while predicting on future runs of a MOOC. These works collectively suggest that same-course training and prediction may optimistically bias results, but that accurate prediction on future iterations is possible and that multiple methods for such prediction exist.
Of course, real-time intervention is not the goal of all predictive modeling research in MOOCs. In the case of many explanatory/inferential works, the goal of model-fitting is simply data understanding. In such cases, the issues highlighted above are less relevant. However, for any tasks which do indeed require real-time model-fitting or prediction -which appears to be the "gold standard" for predictive research in MOOCs and the ultimate goal of many of the works surveyed -utilizing techniques which are adaptable to such contexts is a necessity. Without using these architectures, we are left wondering whether the predictive performance achieved by many otherwise-promising works could be achieved under the constraints of real-time prediction or model-fitting.
Our goal in this section is not to suggest that prior work is useless, or even incorrect -we believe that the search for effective predictive modeling techniques is an iterative process that requires initial experimentation and exploration, even in laboratory contexts which do not fully mimic real-world constraints -but it certainly suggests a promising avenue for future research, which might be able to test previous feature engineering and modeling approaches, re-architected in ways that allow for model training and prediction in "live" environments.
An additional methodology that appears particularly useful for efficient real-world model training are incremental or pre-training approaches, which can efficiently update predictive models to incorporate new data without requiring additional passess over previously-seen data. Such techniques have been demonstrated in the incremental training utilized in Kotsiantis et al (2010); Sanchez-Santillan et al (2016); and in the pre-training techniques used to incrementally grow the neural network models in Whitehill et al (2017). We hope that future research adopts the use of incremental techniques, which would allow for smoother adoption of predictive models in practice.
Just as Section 5.2.1 highlighted how a large portion of the work surveyed fails to consider or model massive segments of the learner population, this section indicates how much of this research may fail to provide accurate or actionable insights for the segments it does evaluate.

Opportunities for Future Research
Several of the trends and methodlogical gaps outlined above directly suggest areas for future research. As we note above, this includes work which examines large, unfiltered, and multi-MOOC experimental populations; work applying rigorous model evaluation and comparison tests to identify effective feature engineering techniques and algorithms for prediction (including, especially, when such approaches may be statistically indistinguishable in terms of their predictive performance); and work utilizing training and prediction contexts which match those in which a predictive model might be deployed in a live course environment.
Our survey identifies two additional research gaps: (1) adoption of temporal modeling techniques, and (2) bridging the "two cultures" of statistical modeling in MOOC research.

Temporal Modeling
There is a clear temporal element to prediction in MOOCs: many courses are offered using a cohort-based model (for example, with new cohorts beginning at monthly intervals); course activity and learning takes place over time, with most courses lasting several weeks; data is collected incrementally, with little usage data being available during the early phases of a course and more data collected as it progresses; learner behavior evolves over the duration of a MOOC. This suggests that models which can account for and explicitly model the complex, time-dependent patterns in MOOC learner data are likely to form a more complete picture of this behavior than those which ignore the element of time. However, research to date has been limited in its use of temporal modeling techniques.
Most prior research which does account for temporality falls into two broad groups. (1) One group utilizes "weekly" feature sets to broadly capture separate collections of features over time periods, typically for each week of a course. Many of the works surveyed here utilize this approach, e.g. Kloft et al (2014); Vitiello et al (2017a). Xing et al (2016) refer to this as "appended" feature extraction. While this type of modeling does capture different features over time, it does not explicitly model these features as being captured sequentially, and treats those predictors as otherwise independent when they are actually related across time steps (Wang and Chen (2016)). (2) A second broad class of work utilizes survival models. This includes Rosé et al (2014); Wen et al (2014b); Yang et al (2013Yang et al ( , 2015Yang et al ( , 2014. Many of the methods used in these experiments, as the Cox Proportional Hazards Model for survival analysis and logistic regression, are forced to make the statistical assumption that student dropout probability at different time steps is independent Wang and Chen (2016) -an assumption which is almost certainly violated, and which limits these models' ability to model correlation between student dropout probabilities at different steps over time.
Attempts to capture more complex temporal patterns in MOOC data have been limited. Hidden Markov Modeling has been used for some dropout models, most notably in Balakrishnan and Coetzee (2013), but this is a generalized form of sequence modeling, not strictly a time-series methodology. A nonlinear state space model is used to capture longer-term information in student interaction sequences for dropout prediction in Wang and Chen (2016). Some work has explored the use of higher-order time series data, utilizing n-gram models of feature sets or behavioral patterns Brooks et al (2015a); Li et al (2017). Fei and Yeung (2015) explore the use of a form of complex neural network model, a Long Short-Term Memory (LSTM) Network. This LSTM model takes as inputs sequences of weekly feature vectors, and is used to predict dropout in this context. Future work which explores these approaches more deeply (such as by exploring other survival modeling approaches such as random survival forests Ishwaran et al (2008)), or which applies other time series approaches, would be valuable and is likely to uncover both informative patterns in data, and gains in predictive modeling performance, improving both the accuracy and theory-building components of future models.

Bridging the "Two Cultures"
Another significant opportunity for future MOOC research is work which unites highly complex, predictive models with techniques for understanding and inspecting the relationships these models uncover, increasing the theorybuildingness and actionability of these models without sacrificing accuracy.
In his seminal 2001 essay Statistical Modeling: The Two Cultures, Leo Breiman argued that the field of statistics was (at the time) divided between a data modeling culture, concerned primarily with understanding the underlying data generation processes and which emphasized the use of inspectable, generative models such as linear regression; and an algorithmic culture, concerned with maximizing predictive accuracy and employing sophisticated (but largely uninterpretable) "black box" machine learining models to this end. At the time, Breiman felt that the algorithmic culture was a troublingly small minority of statisticians -a concern which may ring less true today when considering the rapid growth and adoption of machine learning which has at least partially penetrated the field of academic statistics. However, Breiman's distinction between these two cultures is still, to a large extent, visible in the respective techniques employed by each. This division between the data modeling culture and the algorithmic culture is clear to any reader of predictive modeling research. Both cultures contribute useful knowledge in the context of MOOCs: data models have the potential to inform course design and learning theory by revealing the underlying associations and mechanisms driving student outcomes; algorithmic models have the potential to support real-time early warning and intervention systems with highly accurate predictions even in the absence of interpretable knowledge about the underlying factors behind these predictions.
However, recent research in other fields, including the broader machine learning research community, has begun to erode the distinction between these two cultures, bringing us closer to having the best of both. Several streams of work have begun to make highly complex models more interpretable, gaining theory-building benefits without sacrificing the accuracy of those models. These include approximation approaches, which fit complex models and then approximate the final model using more interpretable linear (Ribeiro et al (2016)) or decision tree models (Craven and Shavlik (1996)), and perturbation approaches, which are used to inspect and explain individual predictions Baehrens et al (2010). This work, along with others, suggests that predictive models are increasingly able to capture the benefits sought by the algorithmic culture -notably, accurate predictions of student success in MOOCs -while also achieving the interpretability or theory-building results sought by the data modeling culture. Both Breiman and Domingos note that these more complex models typically fit the data better (which is why they are preferred by the algorithmic culture) Breiman (2001); Domingos (1999) -and therefore an interpretable version of these models is likely to be more informative and useful than the simple models traditionally used by data modelers, even for their own goals (understanding parameters and relationships in the data). Domingos argues that the notion that simpler models are preferable because simplicity is a goal in itself amounts to a mere preference for simple models (which implies that the data modeling culture and the algorithmic culture simply have different preferences, but that neither approach is more "correct" a priori ). Domingos (1998Domingos ( , 1999 demonstrates that there is no trade-off between accurate and theory-building models: the notion that more interpretable models achieve better performance is demonstrably false under most conditions. Future research in predictive modeling in MOOCs should continue to explore techniques for making complex, highly accurate models more interpretable, following the lead of initial work by Nagrecha et al (2017). This work is particularly salient in the case of educational student models, where the goal of such research is not only to understand the mechanisms underlying these models but also to intervene to support students, and to actively support their achievement of certain outcomes (learning, sustained engagement, etc.). With a clear understanding of the patterns and relationships predictive models are identifying in MOOC data, many stakeholders in MOOCs would be able to act on this insights to support students. This includes course instructors, platform developers, course designers and content producers, support staff, community mentors, and even learners themselves. Furthermore, detailed inspection of models can help identify and reduce algorithmic bias in predictive student models (Luo et al (2015)).

MOOCs and Long-Term Learner Success
Connecting learners' course performance to anything outside the course is a challenge for future MOOC research to address. Little research has explicitly evaluated the connection between MOOC performance and future career or academic success (Wang (2017) is a notable exception.) Studies which evaluate real-world outcomes or link MOOC students to out-of-MOOC outcomes would be especially informative, because it is likely that such research more closely measures the outcomes we seek for many MOOC learners: it would be desirable for MOOC learners to experience career advancement and academic success outside of the platform, not for them to simply watch all course videos or persist until the end of a course (even though these are also useful, and relevant, outcomes in many cases). As we have previously mentioned, privacy protections to MOOC data and a reliance on optional learner questionnaires serve as barriers to this type of research. Using a diversity of available outcome measures (i.e. both engagement and learning) to evaluate existing predictive models would at least provide some indication of the all-encompassing learner success that these long-term outcomes represent, and is tractable with existing research methods and data.

Conclusion
In less than a decade, MOOCs have emerged as a global source of educational opportunity and have reached millions of learners. Predictive models have been central to understanding user engagement and outcomes in MOOCs, and a diverse space of features and modeling techniques have been explored to this end. This includes diverse data sources, experimental subpopulations, feature extraction methods, modeling algorithms, prediction architectures, model evaluation techniques, and prediction outcomes. However, to date, little synthesis, survey, or critical evaluation of this work has been published. Such synthesis is necessary to survey the existing research and scientific consensus (or lack thereof) emerging in the field, and to direct future work. Furthermore, these student models can be used to actively support future MOOC learners, but only if they are sufficiently accurate, interpretable, and generalizable. This novel educational context requires a corresponding shift in research methodologies, which this survey demonstrates have achieved only incomplete adoption on the field of student success modeling to date. The MOOC research community stands to benefit substantially from the adoption of many of the techniques outlined in this paper.
In particular, our recommendations based on this work are that the field needs to move towards more robust model evaluation; broader experimental populations; and realistic experimental contexts. This will encourage growth toward building accurate, actionable, and theory-building student success models. Actionable models in particular are lacking in current MOOC research. We envision student success models supporting personalized, targeted interventions in MOOCs which are able to deliver effective support for students to reach a variety of goals. This vision can only be achieved, however, by closing the gaps outlined above: ineffective model evaluation will lead to poor generalizability and inaccurate identification of at-risk students; restrictive experimental sub-populations will yield models which are not applicable to large segments of learners; the use of unrealistic experimental contexts will produce models that are simply not operationalizable, requiring data which is not available at the time of prediction. However, if these barriers are overcome, MOOCs can truly deliver on their promise of providing effective educational opportunities for all learners. This is a critical time for the community to consider, and to repair, these methodological gaps. MOOCs, and the field of MOOC research, is transitioning from a nascent domain into a fully-fledged field of research, with canonical findings and scientific consensus beginning to emerge on key questions. However, failing to recognize where these gaps may affect scientific knowledge may result in this consensus forming around findings which have limited generalizability, methodological flaws, or practical barriers to implementation. Each of these gaps can be filled by adopting small changes to methodology in future research, and they can be further ameliorated by the construction of tools which enable researchers to follow these procedures without constructing the infrastructure themselves. In particular, we believe that future research which (a) replicates prior research using more rigorous statistical evaluation techniques, and (b) provides research tools and frameworks which support replication and benchmarking of published research, would be valuable contributions.
At the pace of current developments, we are optimistic about future developments in the field, and eager to see the impact these developments will bring to a generation of future MOOC learners.

Cite
Prediction Outcome  Table 6: Literature review matrix of predictive modeling MOOC research. Ellipsis (. . . ) represents missing information that was not reported or not apparent upon review of published work; asterisk (*) represents fields that are not applicable to a study. MANY indicates more ≥ 10 unique values for a given cell. Abbrevions used are listed in Table 7 below.  Table 7 Abbreviations used in literature review matrix (Table 6).