{'fraction_non_alphanumeric': 0.022704040458384894, 'fraction_numerical': 0.011136816054231452, 'mean_word_length': 5.283975659229209, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Assessment Of Online Degree Programs: Lessons From Recent Alumni

The main focus of this study was the assessment performed by recent alumni as an important component of online degree program outcomes assessment. A model of components of the online learning environment was developed and tested to predictive various levels of educational outcomes of online degree programs separately for bachelor and master degree programs' alumni. The educational outcomes include direct educational outcomes and attributed educational outcomes. The model was then validated in predicting summative outcomes assessment. The model played an important role in understanding degree program's online educational outcomes and its predictive validity across all outcomes and degree levels is very high. The alum assessment of the quality of the learning model was found to be the most dominant predictor of educational outcomes for all assessment criteria and for all levels of degree programs. Finally, the explanations and implications of these findings were discussed.


The Problematic Nature of Course Evaluation as Part of Assessment
While several constituencies were involved in the aforementioned efforts, the focal point remained on faculty internal assessment, experts outside the university, and policy makers. To the extent that graduating students or recent alumni were involved, their involvement was predominantly focused on course evaluations or general student satisfaction surveys. Student evaluations at the end of each course were the most sought upon student input in the assessment process. Furthermore, results of these evaluations were incorporated into various programmatic or even institutional decision making (e.g., faculty retention). Nevertheless, the continuing reliance on course evaluations for the assessment of learning and outcomes as well as for academic decision making remains problematic due to lack of their predictive validity. Sitzmann et al. (2010) conducted a meta analysis of 137 independent studies measuring the relationship between student self-assessment of knowledge in the course and actual cognitive learning in the course. They found the weighted correlation to be only 0.27. In other words, only student self assessment of knowledge explained less than 9 percent of the variation in actual knowledge while 91 percent of student actual learning in a course could not be explained by the popular course evaluation.

Recent Alumni Assessment has a High Level Consequential Validity
Very few universities, in comparison to course evaluations, used recent alumni as a source of assessing learning outcomes although recent alumni and graduates are undoubtedly in a better position to assess the learning outcomes immediately after completing their degree programs. The role of such assessment is salient for universities in general, as this information can influence the alumni relations with their institution in the future with potential ramifications for the institution's attractiveness and reputation. For analyzing learning-model based online degree programs, recent alumni assessment as well as self reporting engagement was found to have a consequential validity (McCormick & McClenney, 2012). This consequential validity served well university faculty, leadership, and policy makers. The consequential validity of a recent graduates exit instrument can provide these groups with the assessment of the various components of the online learning model and various dimensions of policy-relevant program outcomes. Furthermore, qualitative assessment by recent alumni can play an important role in identifying program strengths and weaknesses and thus can be incorporated into the strategic planning process. In sum, alumni assessment by an exit survey of online degree programs fulfills all the needs of consumers of these evaluations including those identified by Thompson & Irele (2007): justification of investment; measuring progress toward program objectives; measuring "quality" and/or effectiveness; providing basis for improvement; and informing institutional strategic planning and decision making.

General
The context of this recent alumni assessment of their online degree program was an online learning model similar to the one discussed by Neumann and Neumann (2010). The pedagogy included a completely interactive threaded discussion that allowed students to interact and engage with faculty members as well as each other. For each course, the learning model included problem-based learning through case studies and project-based learning through a signature assignment. Self-reflection was added as part of the course as well through a required essay at the end of the course. Assessment including faculty and outside experts were done separately for each program by using multiple indicators and then aligning institutional learning outcomes, program learning outcomes and course learning outcomes.
In addition to the direct and indirect measures mentioned above, we performed a recent alumni survey. It included, among other indicators, indicators relating to the learning environment, indicators relating to program outcomes, and a qualitative assessment of the degree program and the university as a whole.

Subjects
The population for this study was all alumni who successfully completed their degree program during two semesters. As part of the final degree audit, all alumni participated in required assessment and the completed questionnaires were submitted anonymously (Several studies used a similar approach, e.g., Spooner et al., 2008;Findlay-Thompson & Mombourquette, 2013). Graduating students were given ample time to thoughtfully complete the questionnaire. The overall population of this study consists of 843 master level recent alumni and 564 bachelor level recent alumni. These recent graduates served as the benchmark for continuing improvements as well as reporting summative assessments and created time-series dashboards to pertinent stakeholders.

Measures
The assessment of Online Learning Environment included three facets: the assessment of learner-centered support services, the assessment of the quality of faculty, and the assessment of the learning model.
Recent alumni assessment of the learner-centered support services provided by the university and the program consisted of six items ranging from 1 to 5 (from "very low" to "very high"). The questions were formulated to assess how responsive to student needs and concerns were each of the following: advisers, administrative personnel, IT services, student support services, the office of the registrar, and financial aid services.
Recent alumni assessment of overall faculty quality included four items on the same five point scale where recent alumni rated the overall quality of the faculty; the faculty dedication to providing a quality of learning experience; the accessibility of the faculty; and faculty responsiveness to issues that the recent alum wished to discuss.
The assessment of the learning model used the same five-point scale where recent alumni rated the following outcomes: 1. the quality of the learning model in comparison with other learning opportunities that alumni experienced; 2. the degree to which the learning model was responsive to the alumni educational needs; and 3. the degree to which the learning model succeeded in fostering and supporting effective learning.
Educational Outcomes of the Degree Program originally consisted of two levels of outcomes: direct learning outcomes and attributed learning outcomes.
Recent assessment of direct learning outcomes used the same five point scale and included four items: 1. alum rating of the overall standards of education that he/she received; 2. alum comparison of the quality of learning that she/he received to that of peers or colleagues; 3. alum rating of acquiring both communication and critical thinking skills; and 4. alum assessment of the overall quality of learning in the degree program.
Attributed learning outcomes included five items (five-point scale) assessing to what extent did the degree program learning experience contribute to personal growth; intellectual growth; career preparation and enhancement; and social and cultural awareness.

Analysis
Each concept was measured by a scale consisting of the sum of the items belonging to the domain divided by the number of items. Consequently, all scales were between the range of 1-5 ranging from very low ("1") to very high ("5"). The first step was to assess the norms and reliability of each scale separately for bachelor and master degree programs. The second stage of the analysis included several multiple regression analyses where the three assessments of online environment scales were the predictors (independent variables) of each of the two facets of educational outcomes (dependent variables). These regression models again were performed separately for the bachelor level and for the master level. The relative importance of each predictor was assessed by its standardized regression coefficient (beta). The results of the quantitative model were then validated by adding a summative assessment.
The overall conceptual framework and regression models are presented in Figure 1.   .58** * P < 0.05 ** P < 0.01 Table 2 presents for recent bachelor degree graduates, the predictive model of educational outcomes for direct education outcomes and for attributed education outcomes where the three indicators of online learning environment serve as the predictors.
The first analysis presents the direct educational outcomes as the dependent variables. Overall, the three online environment independent variables exhibited a very strong predictive power of the direct educational outcomes (with an R 2 coefficient of 0.75; p<0.01). Clearly, one predictor was the most influential in explaining direct educational outcomes and it was the quality of the learning mode (beta coefficient of 0.57; p<0.01). This predictor was so strong that by itself it controlled 71 percent of the variation of direct educational outcomes. Quality of faculty was the second important explaining variable while learner-centered support services added very little to the predictive power of the model.

Copyright by author(s); CC-BY 72
The Clute Institute In explaining attributed educational outcomes for recent bachelor alumni, the overall predictability of the assessed online learning environment variable was still quite strong (R 2 is 0.58) but considerable lower that the predictability of the assessment of direct education outcomes (R 2 of 0.75). The assessed quality of the learning model was still the dominant predictor attributed educational outcomes (beta coefficient of 0.42; p<0.01). Again, this predictor was so important for the assessment of recent alumni attributed educational outcomes, which it by itself explained 53 percent of the variation (R 2 ) of attributed educational outcomes.
Although the overall predictability of each model was high but distinctively different (R 2 coefficients of 0.75 and 0.58 for direct and attributed educational outcomes respectively) the dominant predictor in both cases was the same (the assessed quality of learning model). In both cases, faculty quality and learner-centered support services were positively related to educational outcomes but played relatively minor predictive roles. .57** * P < 0.05 ** P < 0.01 Table 3 presents for recent master degree alumni, the predictive model of educational outcomes both for direct education outcomes and for attributed education outcomes. The predictors were the same three indicators of online learning environment. The results for recent master degree alumni were very similar to the results for recent bachelor alumni.
Overall, for recent master degree alumni, the online learning environment played a strong role in predicting direct educational outcomes (R 2 = 0.77; p<0.01) and attributed educational outcomes (R 2 = 0.57; p<0.01). The most dominant predictor in both cases remained the quality of the learning model (with beta coefficients of 0.63 and 0.61 respectively; p<0.01). Faculty quality played a statistically significant role in predicting direct educational outcomes with a noticeable beta coefficient (0.26) although much smaller than the main predictor. Learner-centered support services, by themselves, were not found to have a meaningful effect.

MODEL VALIDATION THRIOUGH THE SUMMATIVE OUTCOMES ASSESSMENT
In order to validate the differential roles of online learning environment in predicting recent alumni direct and attributed educational outcomes, we have added a summative outcomes assessment to our survey. The summative outcomes assessment scale consists of two items measuring the extent to which the degree program contributes directly to successful career and the extent to which the degree is worth the financial expense incurred by the recent alum. Both items were rated as the most important goals of education by alumni. Each item was measured on a five point scale from very low ("1") to very high ("5"). The overall summative assessment scale is the mean of the two items.  Table 4 presents the characteristics of the two items summative scale. Recent bachelor degree alumni rated this item at 4.45 while recent master graduates rated it at 4.36. The Cronbach's reliability coefficients for this scale were 0.75 and 0.78 respectively. .60** * P < 0.05 ** P < 0.01 In both bachelor degree and master degree alumni (Table 5), the online learning environment has a strong predictive validity in determining the summative outcomes assessment with R 2 coefficients of 0.61 and 0.60 respectively. At the bachelor degree level, the quality of learning model was the strongest predictor while the quality of faculty was a close second. Learner-centered support services did not play a meaningful role. At the master degree level, the quality of learning model played the dominant role in predicting summative outcomes assessment. The other predictors played a minor role in explaining summative outcomes assessment.

CONCLUSION
Several important conclusions can be derived from this study. First, recent alumni perceptions of their learning environment were extremely important to understand their assessment of direct, attributed, and summative educational outcomes. The overall assessment online learning environment was measured by highly reliable scales and the same conclusion was valid for scales measuring direct, attributed, and summative learning outcomes. Overall, the mean scores of all scales were quite high and using them as benchmarks to assess continuous improvements would present positive challenges for faculty and leadership. With that in mind, online degree programs can establish those measures as benchmarks against which new improvements and other changes in online degree programs can be assessed.
Second, the major determinant of direct learning outcomes was the assessment of the quality of the learning model. This conclusion was confirmed both for recent graduates of the bachelor and master degree programs. The R 2 coefficients for this model were the highest (0.75 and 0.77 respectively) indicating that the quality of learning model is the dominant factor in addressing direct educational outcomes. The same conclusion about the relative salience of the quality of learning model holds also for predicting the assessment attributed learning outcomes although the coefficient of determination (R 2 ), while still very strong (0.58 and 0.57 for recent bachelor and master graduates), was somewhat lower than in the prediction of direct learning outcomes. The Clute Institute Third, the summative assessment of educational outcomes which examined how much the degree was worthwhile pursuing both intrinsically and extrinsically further validated the superiority of the quality of learning model as the dominant determinant. The second predictor, faculty quality also played a role (though to a less extent than the quality of the learning model) only for recent bachelor degree graduates.
The lack of strong predictive roles of faculty quality (in some cases) and the learner-centered support services might be because a robust learning model (which was the learning model used in this study) incorporated some or all the elements of these concepts into both the design and implementation.
Our findings clearly indicated that the same reliability and predictive patterns existed for both bachelor degree programs and master degree programs and that one dimension of the online learning environment (the quality of learning model) consistently and dominantly predicted the outcomes of direct learning, attributed learning, and the summative assessment of the degree program.
We lumped together all alumni for master degree on one group and for bachelor degree on another group. We tried to see if running our model for the aggregate bachelor and master groups are consistent with single program results. Thus, we applied the same model for a group of MBA alumni to ascertain whether or not the same pattern would emerge. The results indicated that same pattern was confirmed. In assessing summative educational outcomes the online learning environment predictors accounted for 60 percent of the variation (R 2 =0.60; p<0.01) while beta coefficients for quality of the learning model, faculty quality, and learner-centered support services were 0.63, 0.13, and 0.06 respectively. Again, the quality of learning model was the dominant predictor.
We applied the same model to a small group of recent online doctorate recipients and also received a similar pattern. We used the simple correlation instead of the multivariate regression model due to the small sample size. Although we expected the quality of faculty to be the most important predictor of summative outcomes assessment of recent doctoral alumni, the quality of learning model turned out to have a higher level of predictive validity. That is, there is a correlation of 0.63 between the learning model quality and summative outcomes assessment versus a correlation of 0.55 between faculty quality and summative outcomes assessment.
Finally, given the accuracy of the predictions and the reliability of the various scales, our findings clearly confirmed that there is a distinct and quite important role for recent alumni in a further development of the assessment of online degree programs. This assessment enriches the educational outcomes data and accountability of online education.