{'fraction_non_alphanumeric': 0.029755102040816328, 'fraction_numerical': 0.01689795918367347, 'mean_word_length': 5.358941084868933, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Student Evaluations of Teaching in Universities of Pakistan: Analysis from the Perspective of Closing the Feedback Loop

ARTICLEDETAILS ABSTRACT History Revised format: November 2018 Available Online: December 2018 Internationally, centralized systems for collecting students’ feedback have become an increasingly common practice in higher education institutions [HEIs] for monitoring quality of teaching as well as for professional development of faculty members. The collection, analysis and reporting of evaluation results are carried out systematically in many HEIs across the globe. However, how to effectively close the feedback loop with students as well as teachers on the results of student evaluations of teaching [SETs] remain an issue to be addressed. Consistent with global trend, HEIs in Pakistan are also supposed to conduct SETs. In this context, the main intention of this study was to determine whether the cycle of teaching evaluation process is completed, and feedback loop proceeds effectively towards closing around the SETs or not in Pakistani HEIs. To achieve the objective, the triangulation design was used in which an online search was carried out in 130 Pakistani HEIs’ official websites to collect qualitative data. Concurrently, a questionnaire comprising 13 close-ended items, with “yes-and-no” scale, was administered in online format to collect quantitative data from a sample of 507 faculty members and 110 administrators from 130 Pakistani HEIs. Based on content analysis of documents and descriptive analysis of participants’ responses, this paper concludes that universities in Pakistan are bound by HEC to collect feedback from students using central system of Quality Enhancement Cells, but limited attention has been placed to close the feedback loop with students and teachers to inform improvements. Finally, this paper recommends the need for universities in Pakistan to genuinely listen to students’ voices and to act on their feedback as part of quality assurance.


Introduction and Literature Review
Formal systems of evaluating faculty teaching, typically comprising student feedback employing varied instruments and mechanisms, are common in higher education institutions [HEIs] across the globe. SETs are generally required by HEIs as a means to explore students' perspectives on the faculty instruction (Hativa, 2013); to evaluate various aspects of faculty teaching quality and to "compare it across different courses, teachers, departments and institutions" (Goos & Salomons, 2016). Stein et al. (2013) have revealed that faculty members are usually supposed to support student evaluations of their teaching as a part of an institutional evaluation system. These studies, however, also indicate that a substantial difference exists in how institutions practically implement a formal evaluation system (Stein et al., 2013); how students, teachers and administrators engage with the practice of evaluation (Moskal, Stein & Golding, 2016) and use the evaluation results for closing the feedback loop.
The literature on SETs is extensive and wide-ranging which ranges from the variety of the experiences of different cohorts of students (Tucker, 2013) to effect of students' feedback on teaching and learning (Hattie & Timperley, 2007); "the need for effective student engagement" (Krause & Coates, 2008) and on the importance of seeking experiences of first year students (Kift, Nelson, & Clarke, 2010). Furthermore, literature also reveals the "link between student satisfaction and student feedback" (Grebennikov & Skaines, 2009); procedures used and their effect on students' satisfaction and response rates (Moskal et al., 2016) and the strategies used for improving students' satisfaction (Nelson, Smith & Clarke, 2012).
Even though the substantial studies have been conducted formerly, there is paucity of knowledge about the practices that HEIs adopt in relation to the systematic use of students' feedback for improving teaching, enhancing students' experiences and renewing course designs (Tucker, 2013). A number of scholars (i.e., Leckey & Neill 2001;Nair, Mertova, & Pawley, 2010;Shah & Nair, 2009;Watson, 2003) found that students are disinclined to take part in upcoming SETs surveys, if they do not see any improvements ensuing from their feedback. Earlier scholars argued that failure of universities in systematically closing the SETs feedback loop leads to various hazards comprising decreasing response rates from students (Leckey & Neill, 2001); their disinterest in the feedback process (Nair et al., 2010;Shah & Nair, 2009) and lack of trust between students, academics and universities on improvements owing to their voice (Tucker et al., 2008;Watson, 2003). This failure to close the feedback loop on SETs creates such environment that students do not provide their feedback seriously and can also result in frustrating students, rather than providing constructive feedback (Tucker, Jones & Straker, 2008). Moreover, "failing to close the loop on feedback questions the quality assurance framework in institutions and the extent to which they are used to enhance educational quality" (Shah, Cheng & Fitzgerald, 2017).
The concept of 'closing the loop' is based on the notion that there is a cycle of feedback and action. Figure 1 presents the generic feedback cycle with allied key moments illustrated eloquently by Harvey (2003).

Figure 1: The cycle of feedback
Note: Adapted from Harvey, L. 2003, 'Editorial', Quality in Higher Education, 9(1), 3-20 Student feedback is viewed as a significant measure for assurance of teaching as well as institutional quality (Goos & Salomons, 2016)  review processes. However, simply collecting and analyzing such feedback is unlikely to lead to improvements (Shah & Nair, 2009) unless its results are communicated in a way that is informative to stakeholders, mainly students and teachers i.e., closing the loop. Symons (2006) emphasized that universities need to complete the feedback cycle not only with students but strategies should also be developed for ensuring the successful closing of the loop with teachers. Writing in the same vein, Scott (2006) advocates that simply collecting students comments and providing each faculty member a list of these comments would not be sufficient to provide them with adequate information on experiences of students. Vital to the dissemination of student feedback comments to stakeholders, is their "analysis into meaningful data" (Symons, 2006) and timely provision of feedback results.
'Closing the feedback loop' is the term, generally, refers to the "process of informing respondents about what happens to the results of any survey" (Watson, 2003). In this article, closing the feedback loop refers to systematic processes in collecting, analyzing, reporting and sharing the student feedback results with stakeholders particularly students and faculty members; and "timely actions taken...as a direct result of student feedback" (Shah et al., 2017) as well as implementing and "monitoring the effectiveness of actioned improvements" (Shah et al., 2017). How to close this loop effectively has been an issue that universities have struggled with since the 1990s (Watson, 2003).
The rationale behind this paper is that universities/HEIs worldwide are using surveys at institutional and national level to assess students' experiences regarding faculty's teaching effectiveness. For example, the National Student Survey (NSS) is employed in the UK, the Course Experience Questionnaire (CEQ) in Australia and the National Survey of Student Engagement (NSSE) is employed in the USA and Canada (Price & Baker, 2012). In the context of international scenario, Higher Education Commission [HEC] of Pakistan also introduced a system for monitoring and carrying out periodic teaching evaluations of university faculty through students to assess faculty's teaching effectiveness as well as students' engagement in learning (Batool, Qureshi & Raouf, 2010). Institutions, using the results of SETs surveys, identify their weak and strong areas and advertise their strengths for marketing themselves to prospective students. Furthermore, these results are used for recognizing, rewarding and reviewing teachers and in some instances "distributing internal funding" (Shah & Nair, 2013). Likewise, there is also evidence of using student feedback results for making decisions about institutional rankings and appraising institutional performance (Johnson, 2000).
However, how HEIs effectively use students' feedback to improve teachers' professional practices and students' learning outcomes remains an area to be explored, specifically in the context of expansion and diversification of higher education as well as higher students' expectations. Furthermore, there is lack of evidence concerning effective engagement of stakeholders in communicating feedback results and in taking actions for improvement i.e., closing the feedback loop effectively; despite the fact, this is an important strategy to enhance students' response rates and to identify areas in need of improvement (Grebennikov & Shah, 2013).

The Present Study
The literature demonstrates that the cycle of closing the loop on SETs involves four key aspects i.e., data collection, analysis, reporting and finally providing feedback to stakeholders. Based on international practices, this paper analyzes SETs practice in universities of Pakistan from the perspective of closing the feedback loop and argues for the need to develop innovative ways to use feedback results for faculty professional development and engage academics as well as students in quality assurance activities. However, to date, there is limited research on the usage patterns of results of student feedback in Pakistan for improving faculty instructional practices and the students' learning experience.
Keeping in view this background, present study mainly analyzed the practice of closing the feedback loop on SETs in Pakistani universities at policy and practice level to genuinely know that how results from student feedback have been utilized for improving students' learning as well as faculty instructional practices. The rationale behind this research work was also the emphasis of Pakistani higher education sector on quality assurance mechanism, management of academic quality, the enhancement in faculty professional development as well as improving learning experiences of students.

Research Questions
The major purpose of this study was to determine whether teaching evaluation process is completed from holistic perspective and feedback loop proceeds effectively towards closing around SETs or not in Pakistani HEIs. Following research questions were formulated for analyzing practice of Pakistani universities regarding closing the feedback loop around SETs: a. Do policies of HEIs provide any guidelines for reporting of results to various stakeholders and closing the feedback loop on SETs? b. What is the nature of data collection process in HEIs of Pakistan in relation to closing the feedback loop on SETs? c. Are the data analyzed in meaningful format to report the stakeholders? d. Are the results of analysis reported to all the stakeholders? e. Is the feedback resulting from SETs survey communicated to teachers and students to effectively close the loop on SETs in HEIs of Pakistan?

Design and Methods
To achieve the objective, concurrent triangulation approach (Creswell, 2009) was used in this study. The key objective of this design is to use qualitative and quantitative approaches simultaneously to explore in depth the same aspects of the research problem (Creswell, 2009). The major benefits of this traditional mixed methods design include: the shorter data collection time and well-validated findings (Creswell, 2009 The participants from all the 130 HEIs were invited to complete the survey in online format using a 'census approach'. From the 6933 email invitations that were sent to obtain informed consent of teachers and administrators including staff members of QECs, 1783 (25.7%) participants (1417 teachers and 366 administrators) from 91 universities across Pakistan agreed to participate. Finally, link to online survey questionnaire, administered via Google forms, was e-mailed to these 1783 potential participants. After 2 to 5 reminders with the interval of 10 days, total 617 completed questionnaires were received from 507 teachers and 110 administrators including QEC staff members. Respondents were representative from the general, engineering, agriculture, medical, business and arts HEIs with a range of positions. The response rate was about 36% and 30% respectively which is reasonable for a voluntary survey. Table 1 presents the demographic information of all the participants who participated in this study. A self-developed questionnaire comprising 13 close-ended items, with "yes-and-no" scale, was used to elicit perceptions of the participants. Section A of the questionnaire asked for demographic information while section B explored current practices (Q1-13). The questionnaire was piloted with 17 participants from the five faculties of one public university i.e., social sciences, natural sciences, commerce and business management, engineering, agriculture and veterinary along with registrar and director quality enhancement cell (QEC). A number of changes were incorporated on the bases of respondents' feedback to make the language and format of items more understandable and for ensuring the alignment of items with research questions. Expert opinion was also sought for improving the questionnaire items' content validity.
For the analysis of qualitative data (i.e., online accessible documents), a summative approach to qualitative content analysis (Hsieh & Shannon, 2005) was used. This approach involves both the 'manifest content analysis' and 'latent content analysis' aspects. 'Manifest content analysis' quantifies the particular concepts in textual material and deals with the descriptive and objective overview of the "surface meaning of the data" (Dornyei, 2007, p. 246).
Latent content analysis refers to the process of interpretation of content (Hsieh & Shannon, 2005). As the details of targeted concepts and key words were unavailable in accessible documents, the analysis could not proceed to 'latent' level and stopped at 'manifest' level (Kondracki & Wellman, 2002). Participants' responses on close-ended questionnaire were analyzed by applying descriptive statistics (i.e., Frequency Counts and Percentage) and the results were presented in Table 3, Table 4 and Table 5.

Findings regarding Manifest Content Analysis
In total, 47 online accessible documents and consolidated teaching evaluation reports were examined. The analysis specifically focused on searching key words related to all the aspects of SETs process i.e., data collection, analysis, reporting, use of results and provision of feedback with a specific focus on exploring the existence of policy guidelines for closing the loop on feedback. It is worth noting that even a trivial indication of some evaluation activity was considered as an evidence/example of the existence of SETs process and frequency of occurrences was presented in Table 2. The analysis of documents displayed in Table 2 revealed that all the HEIs in Pakistan were bound to conduct SETs as mandatory policy by HEC and instrument for evaluation of faculty teaching by students was available on the websites of all the 130 HEIs. The content analysis of documents further showed that detailed information and policy guidelines regarding all the aspects of SETs process as well as about the process of closing the feedback loop with students and faculty members was unavailable in documents obtained through search of institutional websites.

Findings regarding Participants' Perceptions
The participants were asked to respond in 'yes' or 'no' to key aspects of the SETs process (i.e., data collection, analysis, reporting and provision of feedback) to determine the extent to which these aspects are effectively and efficiently attended during SETs. The other intent of this section was to determine whether SETs process is completed and feedback loop proceeds effectively towards closing around student evaluations or not. Table 3 summarizes the frequency counts and percentages of "yes" responses concerning the process of data collection; Table 4 related to the process of data analysis and reporting of results while Table 5 about provision of feedback. In response to the question (Have you ever experienced/used student evaluation of teaching?) asked to assess the actual practice/regularity in the use of SETs, the analysis of data in Table 3 revealed that majority of the respondents (i.e., 67.4%) believe that SETs are conducted on regular basis in their institutions. It shows, however, almost one third (32.6%) of the participants perceive that this practice is still missing in their institutions and not being actually done regularly though exist in documented policy.
The less than half of the participants (42.5%) reported that their universities use HEC prescribed/developed student evaluation of teaching instrument, 24.8%, the lowest number of participants, also claimed that they use selfdeveloped/adapted as per format of HEC student evaluation form while a considerable number of participants (32.7%) reported that they have no knowledge about the nature of instrument used. This result suggests that all the stakeholders either not involved/ consulted or do not take interest in development/selection of instrument. Analysis further revealed with 88.1% responses that almost all the HEIs in Pakistan use student evaluation instrument at the end of each semester.
The percentage of 'yes' responses for the delivery of instrument depicts that there is general consistency across the country that faculty members are kept away/at arms-length from the process of data collection during SETs. Most often, QEC staff administers the student evaluation instrument for collection of data either manually (41.8%) or online (56%). Majority of the participants (9.6+44.7%) reported response rate below 50%.  Table 4 reveals that collected data is analyzed by staff of QEC and teachers as well as departments are mostly kept away from the teaching evaluation process at this step also in majority of the HEIs. The question regarding reporting of data attracted 41.6% 'yes' responses which indicates that aggregated and analyzed data is not reported to stakeholders in majority of the HEIs in which student evaluation is actually conducted. Further analysis revealed that data is mostly reported in the form of mean scores (86.1% yes), graphic form (6.4% yes) or in standard deviation (4.6% yes).

Provision of Feedback on Evaluation Results
To students 23 5.5 To teacher 137 33.0 Not Provided 256 61.5 Table 5 indicates that often students' comments are available to chair (62.7% yes) and staff of QEC (69.9% yes) and teachers (37.7% yes + not provided 11.5% yes) are rarely provided access to students' comments. Almost same situation can be observed regarding access to end results of student evaluations. These results point out a big weakness in the existing SETs process i.e., no access to students' comments and end results to all the faculty members which is the most frequent and effective source of formative feedback to teachers.
Analysis in Table 5 further revealed that evaluation results are never fed back to students (5.5% yes) and rarely to their teachers (33.0% yes). This result suggests that the results are simply filed away and not disseminated to those who could use them at both faculty and institutional level. It means that the time and energy spent in the collection and analysis of this valuable information is wasted and that reports on the student feedback results do not reach to students and faculty members who are the most relevant stakeholders in university community.

Discussion
This mixed method study analyzed the Pakistani universities' practices of closing the feedback loop on SETs. A number of scholars (i.e., Hattie & Timperley, 2007;Shah & Nair, 2013) argue that universities ought to develop and disseminate detailed guidelines for all the stakeholders regarding their roles and responsibilities in SETs process, use of evaluation reports; interpretation of results as well as provision of feedback for effective implementation of institutional teaching evaluation system. Writing in the same vein, Shah and Nair (2013) emphasize that similar guidelines should also be developed for members of promotion panels regarding the use of SETs results. The results of content analysis of documents in this study, however, revealed that clear-cut policies and detailed guidelines are unavailable in accessible documents regarding all the aspects of SETs process i.e., closing the feedback loop. Limited evidence is available in documents on systematic usage of feedback from SETs surveys by academics in Pakistani universities to review curriculum content, assessment designs and teaching methods in a timely manner. The analysis of participants' responses also confirms the results of content analysis.
Moreover, provision of feedback resulting from SETs is increasingly being considered essential across the globe as a means to guide teaching practice, (Hattie & Timperley, 2007), to enhance faculty professional development (Blair & Noel, 2014), to improve students' learning and to strengthen faculty development policies and practices (Catano & Harvey, 2011). However, the findings herein suggest that Pakistani universities do not make systematic usage of the student feedback results to inform improvements. The results of this study further postulate that SETs process in Pakistani HEIs, is generally limited to the collection and analysis of data, and results are filed away, not fed back and communicated to stakeholders particularly students and teachers.
Overall, the results of this study are aligned well with the studies conducted by (Scott, 2006;Shah & Nair, 2009;Symons, 2006) who concluded that generally universities simply collect and analyze SETs data and do not complete feedback cycle with stakeholders. But these results contradict with the suggestions of Harvey (2003) who advocates that "views of students should be integrated into a regular and continuous cycle of analysis, reporting, action and feedback" to make an effective contribution to quality improvement in HEIs. Harvey (2003) further stressed that ensuring an appropriate action; providing feedback in transparent, objective as well as consistent manner and making reports publicly available to all the stakeholders is more important to gain support and trust from all the stakeholders than only concentrating on having mechanisms for data collection.

Conclusion
The results of content analysis of documents and survey responses evidently suggest that closing the feedback loop is an area which requires improvement in Pakistani HEIs. Although, all the universities in Pakistan are bound by HEC for collecting, analyzing and reporting student survey results, but there is limited evidence of effective use of student feedback to enhance faculty professional development and improve students' learning experiences. It was concluded that though SETs exist in Pakistani HEIs as a documented policy but the extent of their actual use was notably low and there were problems/challenges of actual implementation on regular basis. There also appears to be little meaningful feedback of student evaluation results to students and faculty members, mainly because universities have not developed clear-cut policies and detailed guidelines for execution of SETs process. Universities in Pakistan conduct SETs only to fulfill HEC administrative requirements not with any formative or summative intention. It was also concluded that HEIs in Pakistan are experiencing low response rates on SETs, which is worldwide concern of HEIs ( Smithson et al., 2015). The matter of low response is reflective of the skepticism of the students regarding the surveys and feedback. This aspect of low response, therefore, should be examined. This situation calls for the development and implementation of innovative ways by universities to listen to the students' voices because students, being fee payers, are constantly demanding that their opinions should be heard and acted upon.