{'fraction_non_alphanumeric': 0.025405466712033573, 'fraction_numerical': 0.011058183055461041, 'mean_word_length': 5.464442815249267, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Responsive Outcome Evaluation as an Internal Quality Assurance Mechanism Alternative at IHLs in Malaysia

Proliferation of higher learning institutions and massification of enrolment in higher education demand quality assurance of academic programs. Institutions of higher learning (IHLs) should be more responsive and routinely take proactive role in quality assurance, in self-evaluating the merit and worth of learning institutions and programs offered to the stakeholders as measures for continuous quality improvement. This can be done by enhancing, complementing and developing IHLs’ internal quality assurance mechanisms. Hence, the first part of this paper will observe the current practice of quality assurance in the IHLs in Malaysia. The second part will further propose and elaborate the development of a responsive program evaluation as a complement to the existing quality assurance mechanisms. In the proposed qualitative and responsive program evaluation, the quality of an academic program is explored from the perspectives of stakeholders. The domains of change are the main reference in the proposed responsive program evaluation. These domains of change focus on knowledge, attitude, skills and aspirations (KASA) and broad indicators of outcomes derived from the program outcomes, course learning outcomes and graduate qualities promised to be catered to the stakeholders. The proposed responsive outcome evaluation as an alternative internal quality assurance mechanism is believed to complement the existing quality assurance mechanisms by including the aspects which relate directly to the stakeholders such as employers and programs graduates. With relevant minor modification, the proposed responsive outcome evaluation could be employed in institutes of higher learning. ©2012 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of the Faculty of Education, University Technology MARA, Malaysia.


Introduction
Quality of higher education has been getting a major limelight in recent years in every nation. According to Awino and Agolla (2007), impacts from the demands for accountability, socio-economic and political development, international market competition and advances in information technology have set new educational paradigms. Frazer (1994) asserts the importance of accountability factor towards the society, clients and subjects when discussing about quality in higher education (p. 103). Besides the accountability factor, the hike in competitiveness between providers of higher learning and the decline in public funding have also been raised in relation to the importance of efficiency and performance of institute of higher learning (Jung & Harman, 2009).
According to the statistics disclosed by Ministry of Higher Education (MOHE), there are 20 public universities; 33 private universities; 4 foreign university branch campuses; 22 polytechnics; 37 community colleges and about 501 private colleges in Malaysia. The fast rising numbers of citizen pursuing higher education have contributed to this development. This massification of enrolment has been one of the major 21 st century higher education realities discussed globally (Altbach and Salmi, 2011;Jung and Harman, 2009;Morshidi, 2006): "The implications of massification have been immense, however, with major financial implications, infrastructure challenges, questions about quality, and potentially diminished returns in labor markets with more university graduates than the economy can sustain" (p.13).
This proliferation of higher education institutions in Malaysia demands an effort in quality assurance and can be looked at as alarmingly negative if the programs offered are not up to the quality that the educational provider promised. The government and educators should be aware and more responsive in their efforts to ensure quality parallel to the current massification of education (Nethi 2005, p.15) .
There are several cases of higher learning institutions halted by the government in Malaysia, as a result of incompetency in operation as reported by Carvalho (2011, p.N24). Graduates produced by the institutions that offer and award low quality programs and diplomas, would be a burden instead of helping building up and translating into high income nation with sustainable and inclusive economy. Hence, it is crucial to embrace the language of quality and to institutionalize and routinize components of quality at each higher education institution (MQACEO, 2011).
In April 2002, Quality Assurance Division or QAD was set up by the Ministry of Higher Education in Malaysia to ensure the quality of public universities. Similarly, the National Accreditation Board or Lembaga Akreditasi Negara (LAN) functioned as quality guardian of private higher education institutions. This duality in functions of QAD and LAN was reorganized when both agencies were merged in November 2007 and known as Malaysian Qualification Agency or MQA, guarding the Malaysian Qualifications Framework (MQF) in assuring quality at par excellence and to inspire the confidence of stakeholders (MQACEO, 2011).

Quality Assurance (QA) in Education
The importance of quality and effective quality assurance mechanisms in education have become crucial and the top priority in national agenda (Materu, 2007;Jung and Harman, 2009). Quality assurance system should comprise of two categories of mechanisms which are the internal and external quality assurance mechanisms (Grant, 2000). Internal mechanisms should be a part of a continuous process in educational administration and should be available to the public and subjected to external quality evaluation (quality audit). Proactive role of quality assurance should be taken, in self-evaluating the merit and worth of learning institutions and programs offered to the stakeholders by having internal quality assurance mechanism. External quality assurance is provided by agencies and bodies outside the institution aiming at the issue of accountability to the society. Internal or institutional quality assurance on the other hand refers to all internal assessment done to assure quality that is usually focused on academic issues. Guba and Lincoln (1989) defined evaluation as a form of inquiry focusing on some program, process, organization, person, etc. which results in a merit and/or worth judgment. Patton (1997, p.23) defined program evaluation as "...the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming"(As cited by Shaw et al, 2006).

The Quality Assurance (QA) in relations to Program Evaluation (PE)
There are similarities in the way QA and PE are looked at. According to Leahy et al (2009, p.72), QA and PE are differentiated accordingly "to their origins -different in perspective, not intent". It was further elaborated that QA was from the industry that focuses on innovation to combat competition while PE was from the promotion of compliance through government legislation. However, "there is no solid boundary between the two terms".
According to Mizikaci (2006, p.41), program evaluation can be defined as "… a systematic operation of varying complexity involving data collection, observations and analyses, and culminating in a value judgment with regard to the quality of the program being evaluated, considered in its entirety, or through one or more of its components". It is very important in order to determine the extent of the effectiveness in improving quality in educational practices and outcomes. It goes beyond a measurement tool in quality assessment systems as it suggests a system offering "…systematic, scheduled and focused examination of the control of quality". Mizikaci (2006) relates the PE and QA with a statement, "Program evaluation is related to the technique of quality assurance" (p. 42).

The Current Practice of Quality Assurance System
Quality assurance system used by institutions of higher learning (IHLs) in Malaysia employs both internal and external quality assurance mechanisms. For external QA mechanism, IHL in Malaysia are subjected to the accreditation process of Malaysian Qualification Agency or MQA. Other external mechanisms include quality audit, ISO Standards and Peer Reviews. Internal quality assurance mechanisms include self-evaluations of resources and activities in teaching and learning process; self-accreditation, educational assessment and students/staff feedback.

The Gap in the Current QA System
According to Frazer (1994), a comprehensive quality assurance system should examine quality from the inputs, processes and outputs factors. Quality is a multi-dimensional concept composed of three interrelated dimensions, the quality of the human and material resources available (inputs), the quality of the management and teaching/learning processes taking place (process), and the quality of the results (outputs or outcomes) (Grauwe & Naidoo, 2002). Unfortunately, the current mechanisms implemented in Malaysian IHLs mainly focus on the input and process dimensions of programs Logic Model and lacking the output and outcome element that made up a comprehensive quality assurance system.
Additionally, the current practice of QA system at IHL has been observed to take the top-down managementoriented approach in dealing with quality by promoting a system of compliance to standards. This managementoriented approach according to Guba & Lincoln (1989) has several weaknesses. Firstly, management bias referred to the judgment based on goals and intentions of policy makers. Secondly, the findings are hardly used in decision making and the lastly, there is no use of stakeholders' experiences and expertise and dialogue with and between stakeholders while their interests are at stake (Abma, 2005, p.1).
Thus, it is the aim of this paper to offer an alternative to the current quality assurance practice in IHL in a form of responsive outcome evaluation. This proposed self-evaluation of a diploma program will be the blueprint in the development of a tool as internal QA mechanism complementing the existing internal and external QA mechanisms of peer review, quality audit and MQA accreditation. The following discussions provide the framework for the proposed mechanism. In addition, practical suggestions regarding which information needed from which stakeholders are provided. To ease the interpretation of the meanings derived from the information gathered from the stakeholders, a list of broad indicators for each domain complements the discussions on the proposed responsive outcome evaluation.

The Responsive Program Evaluation
Responsive program evaluation was first introduced by Stake (1989) focusing on "…redirecting data gathering and interpretive efforts around emerging issues of importance" (Abma, 2005, p.280). Responses of participants and stakeholders in the evaluation are taken into account in defining the "real meaning" of their life experience going through the program. This will help the stakeholders to become better acquainted with the quality of the program (Stake, 2004, p.8). Educational issues are of importance so as to let the stakeholders gain understanding of the evaluation at hand. Interest of stakeholders is of concern on what is good and bad about the program or the identification of program's quality. Stake (2004) reminds that the idea is to convey the sense of value through personal experience and this is done by describing the case (p.26). Stories of change can be useful in giving some sense of merit and shortcoming from the vivid experience told by participants.
According to Duque & Weeks (2010, p. 85), the focus now is shifting to students as the primary consumer of higher education service and "…uses perceived quality and satisfaction ratings as the main measures of service performance. They further argued that this approach might be controversial if students are perceived as passive recipients and they are viewed as commodities. This approach and focus may be a good recent development in quality assurance that views students as active participants in the learning process.
The proposed qualitative evaluation is responsive in nature by responding to the concerns and issues raised by its stakeholders about the quality of programs offered. It is also responsive in nature because in the evaluation processes and results, different value perspective of the audience or stakeholders will be referred. It takes a participatory and naturalistic stance and will be focusing on quality as transformation and quality as fitness for purposes. Perspectives of stakeholders include a wide arrange parties: curriculum officers, lecturers, graduating students, graduates and employersare gathered in the proposed responsive outcome evaluation. The "meaning of life experiences" of curriculum officers and lecturers in implementing the program; the graduating students and graduates going through the program, are explored and analysed in terms of knowledge, attitudes, skills and aspiration or in short KASA. The framework for the proposed responsive outcome evaluation is as shown in Figure 1.

Evaluating using domains of change and guided by broad indicators
In the proposed responsive outcome evaluation, the evaluation of outcomes will be framed by "degree of change" as proposed by Bennett's Level 3 of Targeting Outcomes of Programs Hierarchy (Bennett, 2004, Mathison, 2005. This degree of change is referred to as domains of change as proposed by Davies & Dart (2005) in his Most Significant Change Technique -MSC. These domains of change include Bennett's KASA: knowledge, attitudes, skills and aspirations. The framework for using domains of change and broad indicators in the proposed responsive evaluation is as shown in Figure 2. In this proposed responsive outcome evaluation, the meaningful changes based on life experience of stakeholders should be collected through in-depth interviews and focus groups. Then the "meaning" of all the KASA "domains of change" becomes the focus of analysis. It is guided by "broad indicators" derived from the program outcomes, course learning outcomes and also the graduate qualities which are deliberately microscopic. The basis of analysis will be in broader categories of indicators of quality and following an emergent design flexibility whereby the broad categories of indicators may have additions as "… understanding deepens and as the evaluation unfolds" (Patton, 2003). In other words, in this proposed evaluation the focus is rather on "…questions concerning what outcomes mean to participants rather than how much of an outcome was attained" (p.3)

"Often indicators refer to parameters based on statistics (statistical indicators). This indicator is
usually the most relevant when evaluating student flows, staff/student ratios, research records, cost per student, laboratories and other equipment, libraries, information technology, equity, etc. It is less relevant when the quality of outcomes, teaching, or links to research are described. In practice, indicators are used in a broader sense, in a qualitative way, to describe the object" (Hämäläinen, Mustonen & Holm, 2004:17) The framework for qualitative data collection and analysis for the proposed responsive evaluation is as shown in Figure 3 below: The central investigation in this proposed responsive outcome evaluation is on how is the quality of the program in terms of its outcomes (quality as transformation and as fit for purpose). The following list of questions could be considered when attempting to confirm the program quality through the proposed responsive outcome evaluation.
Quality as transformation 1. What are the lecturers' perspectives on the quality of the program in transforming students with knowledge, attitudes, skills and aspirations (KASA)? 2. What are the graduating students' perspectives on the quality of the program in transforming them with knowledge, attitudes, skills and aspirations (KASA)? 3. What are the graduates' perspectives on the quality of the program in transforming them with knowledge, attitudes, skills and aspirations (KASA)?
Quality as fit for purpose 4. How is the quality of the program in terms of its graduates' knowledge, attitudes, skills and aspirations (KASA) from the perspectives of employers?
Likewise, the following are some suggestions for the broad indicators which could be referred to when interpreting the information gathered.

Conclusion
With the proliferation of IHLs and massification of enrolment in academic programs offered, demands for quality have been a major issue in our education system. Each and every higher learning institution has to take proactive and responsive steps in developing own internal quality assurance mechanisms to ensure quality of their academic programs i.e. diploma or degree. The proposed responsive outcome evaluation may act as a selfdeveloped internal quality assurance mechanism complementing existing internal and external quality assurance mechanisms. It complements by including important aspects which were neglected in the current quality assurance practices; the stakeholders' feedback. With appropriate modification such as on the indicators derived from the respective program, this internal quality assurance mechanism may be adapted by respective IHL in Malaysia to assure quality and tackling academic issues by considering and taking into account the major concerns of stakeholders of higher education system itself.