{'fraction_non_alphanumeric': 0.02628008283939156, 'fraction_numerical': 0.009878835487633603, 'mean_word_length': 5.085759814573374, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Online student response systems and student engagement in large EFL classrooms

English as a Foreign Language (EFL); online student response system (OSRS); student engagement; student response system (SRS); Taiwan. This study investigated how the use of online student response systems (OSRS) in conjunction with an active question and answer technique affected student engagement and achievement in on-ground classes. Quantitative and qualitative data from 118 undergraduate nursing students, related to the change from classical to online question-answer (QA) activities, were collected using pre-tests, posttests, questionnaires and interviews. Two systems (Zuvio and Socrative) were introduced, allowing all students in large classroom environments to share their answers with personal smartphones before teacher feedback was given. The findings indicate that the question-answer activity using OSRS improves and retains student engagement in large classes. Two factors that contribute to the high engagement include anonymity and personalized feedback.


Introduction
Getting all students to engage with what they are studying is a top priority of many teachers. Students who engage in learning activities tend to practice and study more, enjoy it more, and be able to understand more of their course content than students who passively receive what we teach. Research has demonstrated that active learning strategies help increase student engagement and improve their academic performance (Mohrweis & Shinham, 2015;Nelson & Crow, 2014). Instead of just getting students to listen and take notes, active learning is "any instructional method that engages students in the learning process" (Prince, 2004, p. 223). Of all active learning activities, question and answer (QA) gives teachers a chance to immediately assess whether individual students understand a particular concept that has been covered in class. In a classical QA activity, the teacher makes pauses during his or her lecture, asks a few questions on what was just covered, and randomly calls students to answer the questions. QA is similar to Elicitation. Both techniques are used by teachers to obtain information about what their students do or do not know. However, the former uses questions while the latter involves using pictures, actions, and descriptions.
In the classroom, English teachers often ask questions to check students' understanding of course content. These teachers are presented with a huge challenge, which is that having each student in large classes answer questions takes up too much class time. Because of this, teachers often only call a small number of the students. However, there are two problems involved with only asking a few students. First, it does not allow for the teacher to hear the responses of all individual students in class and to check how many students actually understand what is being taught. Second, if only a few students are required to participate in the lecture, it can be more difficult to keep other students engaged when they are not given the opportunity to share their ideas with the class (Chen et al., 2016).
Our solution to these problems is to find a technological component that would allow all the students to share their ideas with the class. Socrative, Poll Everywhere, and Plickers are examples of online student response systems (OSRS) -a web-based technology that runs on any device with a web browser and internet access, quickly gathers student feedback, and tells us whether each student understands what is being taught. Research has proved that OSRSs can facilitate active learning pedagogical approaches and increase student engagement. Mork (2014) surveyed 214 students using Socrative in their EFL classes at two Japanese universities. She found more than 95% of her students enjoyed using the system to respond teacher-led comprehension checks. Beside enjoyment, her study claims that using OSRS motivates students to learn and allows students to get rapid feedback on their knowledge and performance from all their peers and the teacher. In the second study, Chen et al. (2016) conducted a ten-week study with 231 Japanese university students who were learning English grammar in their school. Chen divided those students into experimental (n = 124) and control groups (n = 107). The students were tested using weekly tests to measure the accuracy by which students applied the English grammar rules they were taught, and surveyed for their perceptions of the benefits that Poll Everywhere (PollEv) could add to an active learning Think-Pair-Share (TPS) activity. In the whole group comparison, the experiment group using PollEv performed better on grammar test scores as compared to the control group that did not use it. The percentage of students who enjoyed TPS in the experiment group (77.42%) was also greater than that of the control group (65.93%).
Having appeared on campus only half a decade ago, OSRS has become a ubiquitous presence in language classrooms because teachers who use it seem to have engaged students in the learning process and improved their learning outcomes. With the many options, we considered the question variety, training and support when adopting Socrative and Zuvio for our research. First, they allowed teachers to make use of various question types. This includes short answer questions that are applicable across a wide range of courses in languages and other disciplines. Second, the lead researcher for our project has nearly two years' practical experience of implementing the two apps into the classroom. He could train our faculty and staff to train others and teach us to troubleshoot the common problems on our own. Finally, both apps were freely available at the time of writing. They provided a full user guide, numerous video tutorials, an excellent FAQ, and a community support forum.
While there is a growing body of scholarly work on the relationship between online student response systems (OSRS) and student engagement/learning outcomes, we noticed the impact of OSRS on the effectiveness of active learning strategies, engagement levels, and achievements of students in large classes had not yet been fully explored. In order to fill that gap in the literature, we conducted an investigation combing quantitative (tests, surveys) and qualitative (interviews) methods that helps us to better understand the relationships of using two different OSRS with the experiences and performances of university students using such tools in EFL settings. To enable teachers to gather quick feedback from all students in the classroom, we selected two technologies: Socrative (www.socrative. com) and Zuvio (www.zuvio.com.tw). The study was guided by the following research questions: RQ1. Can the use of online student response systems (OSRS), combined with the Q&A technique, affect student reading and vocabulary test scores in large classes?
RQ2. Does the use of OSRS enhance student engagement? RQ3. What is the impact of using OSRS on students' perceptions and attitudes towards learning English language in large classroom environments?

Language learning strategies
Learning strategies are actions taken by students to improve their own learning (Oxford, 1990). Learning strategies, according to their functions, can be divided into six groups. Cognitive strategies such as analyzing and classifying is the first type of learning strategies that help students to understand and recall new information. Metacognitive strategies are used to plan and to evaluate students' own progress towards communicative competence. There are also affective strategies, which develop self-confidence and dedication for more active involvement in the learning process. Social strategies stimulate interaction of students with others, while memory strategies support students and help them store, retain and later retrieve information. The last type is compensation strategies. These strategies such as guessing or using gestures provide emphasis and meaning to fill the knowledge gaps of students (Oxford, 1990). In foreign language education, learning strategies are important as they help students to be more involved and self-directed. If students can get more control of their learning process, they will have a higher chance of success in developing communicative competence. Hence, teachers play a critical role in helping students to apply these strategies. Online student response systems (OSRS) can help teach metacognition that develops students' ability to reflect what they learn through questions, think of ways to improve, and try again and go back to reflection.

From constructivism to active learning
In contrast with a model of instruction whereby knowledge is transmitted from teachers to students, active learning means students are active in the learning process as they take more responsibility for their own learning. Active learning is based on a theory called constructivism. This theory views learning as an active process in which learners gain a deeper understanding of a subject through their own action and reflection (Cattaneo, 2017;Freire, 1993;Jonassen, 1991). In schools that use constructivist teaching methods, students learn new knowledge and skills for themselves by doing and reflecting on their academic progress rather than by passively absorbing information from teachers. Students are asked to bring prior knowledge as the basis for the construction of new knowledge into a learning situation. Also, students are encouraged to work with and give feedback to their peers, think critically, and reflect on what they have done and how their understanding has changed.
Within the active learning framework proposed by Edwards (2015), long-lasting learning outcomes "come through direct experience and interaction with the intellectual, social, and physical environments" (p. 26). In an intellectually active learning classroom, students engage with the content using higher, more active levels of cognitive thinking in Bloom's Taxonomy such as applying, analyzing, evaluating, or creating. Examples of instructional strategies include concept maps and synthesizing research for presentations. Equally important as intellectual involvement is being social active. When students feel connected to their teachers and peers, they put more focus on "learning the material and building academic skills" (Furrer et al., 2014, p. 5). Small group and whole class discussions are two methods for having students involved socially as they learn content in the classroom. Classrooms can also be arranged to accommodate the needs of the physical activities that require students to move during lessons. Several strategies such as board games and creating videos assist students to release energy and at the same time, to stay focused.
Research attributes the following benefits to active learning (Bonwell & Eison, 1991;Lee & Hines, 2012): an increase in student engagement and understanding. Active learning gives students greater involvement and control over their learning than traditional (passive) lectures, which in turn creates their interest with the content of a subject and keeps them motivated. In addition, active learning requires students to construct new understanding by interacting with teachers, peers, and artifacts. For example, teachers make pauses during their lectures or presentations, ask questions (or have students ask each other questions) to check understanding, and provide more frequent and immediate feedback to students.
Potential challenges of active learning are not difficult to imagine (Bonwell & Eison, 1991;Ungar et al., 2018). First, teachers may notice they need to cover more content in less time when they implement active learning in their classroom. Preparation is also a common struggle for many teachers who already have an excessive workload. They do not have enough time for preparing active learning activities. In large classes, teachers also tend not to use active learning because they think they would not have enough time to monitor all students' learning progress. Bonwell and Eison (1991) argued that although active learning decreases lecture time, there are other ways to ensure that students learn assigned course content such as using reading and writing assignments. Second, preparing for active learning activities does not take more time than preparing for new lectures. In a class larger than 40 students, teachers can use a variety of methods to involve all students in plenary activities. For example, the class can be divided into small groups for discussions. Finally, good teaching may not result in good learning. There is often a gap between what teachers have taught and what students have actually learned, and that disconnect happens more than many teachers realize. To mediate the disconnect, a pedagogical approach proposed by Duckworth (2006) suggests to teachers that they should learn about their students and how to help students learn effectively either individually or as a whole group. An application of this approach is teachers listen for common themes and questions among their students. Based on a firm understanding of the students' needs, teachers provide advice or a new framework that guides the students to improve their understanding and outcomes.
A more recent literature found that another frequently mentioned barrier to teachers' use of active learning techniques was lack of technological knowledge and experience (Ungar et al., 2018). As technology advances, it changes how students learn in school. Many teachers are expected to design more active, learner-centered tasks using technology. However, they do not know how to choose the right tool or how to operate it for the tasks they have prepared in their lessons. Some teachers also worry technological failure would interrupt the lesson flow if it takes considerable time to repair. Additionally, the technologyrelated insecurity limits teachers' use of active learning. When teachers become aware of the fact that their students have more technological knowledge and experience than themselves, often they start to feel insecure, and then avoid using technology to make the learning environment more active. A carefully-designed implementation process is perhaps most needed to overcome such barriers (Kelly, 2015). In the process of designing the implementation, school administrators must collaborate closely with teachers. The ways of engaging teachers include full discussion with teachers on the needs and goals of using technology such as supporting the active learning. Also, school administrators should identify teachers with experience successfully using technology in their curriculum and invite them to work as coaches because they can "describe their own successes and obstacles" and positively influence other teachers (Kelly, 2015, p. 42).

Classroom spaces
Over the past two decades, classroom space has received a great deal of attention from researchers because changing a traditional classroom into a new setting that accommodates active learning pedagogies can enhance the learning outcomes (Hyun et al., 2017;Phillipson et al., 2018). In the Student-Centered Active Learning Environment for Upsidedown Pedagogies (SCALE-UP) project, North Carolina State University (NCSU) created classrooms where student teams sit at a round table and have whiteboards nearby. This round table approach works to foster collaboration and to encourage sharing. Comparing data (classroom videos/audio recordings, interviews/focus groups, pretests/posttests, portfolios of students work) of nearly 16,000 traditional and SCALE-UP students taking physics, NCSU's researchers found that students in the redesigned classroom not only have better abilities to solve problems and to understand physics concepts, but also better attitudes than traditional students (Beichner et al., 1999).
There are a few scholars who are interested in preparing teachers to teach in these spaces. Forman (2014), for example, describes the professional development workshops provided by the University of Iowa in which all active learning classroom instructors learned active learning pedagogies. Examples include inquiry-guided learning, peer instruction, and team-based learning. Another focus of interest is instructors' experiences during their time in active learning classrooms. Phillipson et al. (2018) interviewed seven novice teachers about their experience in an active learning classroom (ALC) project at Queen's University in Canada. The study indicated that all the teachers perceived that teaching in the ALC was "a unique experience that shifted their behaviours and perceptions-both about student learning and about their own roles in the classroom" (p. 13). Transforming into teacher-learners, these teachers felt excited about being in such a classroom just like their students, and expressed intention to employ active learning approaches in the future. Being able to help students to become independent thinkers by giving them space for enquiry and by putting them in charge of learning made all the teachers enthusiastic about ALC. Unfortunately, it was not financially practical to change all the classrooms on campus to ALC. This situation has not prevented researchers seeking to promote active learning but inspired them to investigate whether utilizing active learning activities in classroom that were set up to accommodate traditional lecture style teaching could bring positive changes in students' engagement in the classroom.

Students' perceptions on large classroom learning
Many schools only offer large English language classes (may consist of 50 or more students) if they do not have enough budget, space, or faculty. This may increase levels of anxiety among some university students because if classes are too big they cannot adequately learn the material and get help when needed, while others are more comfortable being anonymous in a crowded classroom. Koenig et al. (2015) conducted a survey of 75 college students for their assumptions about large classes. It was found that the most common reasons why students preferred large classes were related to class content (courses not needing individual instruction, 31.6%), decreased responsibility (skipping class more often, 28.8%), and student number (e.g. having more students in class, 25.4%).
However, some problems are created by overcrowded classrooms. Less individualized focus is perhaps the biggest challenge that is nearly impossible to find a solution to overcome. When there are too many students, teachers cannot spend the same amount of time with each student and give additional attention to students who struggle to make progress. Additionally, Elson et al. (2018) examined the responses of 266 students in an accounting course to questions inquiring about in-class experiences, and identified instructor-student interaction as a key factor that influences student perceptions of course effectiveness in large classes. Students who had more interaction with the instructor expressed satisfaction with the course, compared to students who had less interaction.

Correlation between the use of OSRS and student engagement
Research has shown that student response systems improve student engagement and performance by creating an active learning on-ground classroom (Abir, 2017;Dong et al., 2017;Dunn et al., 2013;Miles & Soares da Costa, 2016). This contention is supported by Stevens et al. (2017), who found 82% of their 161 third year medical students who enrolled in a clinical microbiology course at a college in Ireland agreed that the teaching sessions where the clickers were used were more engaging than the sessions without the clickers. In another study, Terrion and Aceti (2012) explored the reactions of 200 students (177 freshmen, 16 sophomores, 4 juniors, 3 seniors) in a large introductory chemistry class at the University of Ottawa in Canada to eInstruction's Classroom Performance System (http://www.einstruction.com/) with a five-point Likert-type scale. The findings indicate that there was a positive correlation between clicker implementation and student engagement (r = .0678, p < 0.01), and students believe that using clickers as part of the class lecture help them to more effectively learn the course material (r = 0.577, p < 0.01).
Online student response systems (OSRS) can work with a multitude of devices and operating systems without buying expensive hardware. OSRSs offer various types of questions that teachers can give to students. Also, they deliver the questions directly to students' personal Wi-Fienabled devices such as their smartphones or tablets (Chen et al., 2016;Shea, 2016). There are many useful and free OSRSs available for teachers to creatively engage students using their personal smartphones. Balta and Awedh (2017) successfully promoted collaboration among 112 students at a university in Turkey through Socrative. Teachers first logged in and shared four to six physics questions that they prepared in advance on Socrative. Students then logged in, worked in pairs or small groups for 15 minutes, and submitted responses on their own devices.
Not all the studies involving the use of SRS/OSRSs in the classroom have shown an improvement in student engagement. For example, Zapf and Garcia (2011) compared the perceptions of engagement and class grade point average (GPA) from 405 students at a regional Midwestern university in the United States. Their findings indicated that a clicker class enrollment failed to change student perceptions of level of engagement in classroom activities. However, Zapf and Garcia did not provide the amount of time that they gave to the students for submitting their responses. Such methodological nuances among all the investigations may explain why we found the capricious results.

Method Subjects
The study took place at an urban private university in Taiwan. The researchers of the study worked as English teachers at this private school. 118 students who participated in the study were first-year undergraduate students enrolled in a twoyear nursing program. The students were, as participants, predominantly female. Their ages ranged between 20 and 22 years. They took their integrated English skills course in the fall semester of the school year 2019. Most students have been trained to use metacognitive strategies such as using syllabus as a roadmap for learning English. The study lasted for six weeks. Each week, students were required to spend two hours in the classroom, with 12 total hours of in-class instruction. The course focused on developing students' reading comprehension and vocabulary skills. The course instructors taught the course using Zuvio and Socrative in conjunction with an active question and answer technique. All the classrooms at the university are already equipped with projectors, Wi-Fi technology, and support for using computers. The applications are integrated with our PowerPoint and Keynote software. Teachers open the PowerPoint/Keynote software in conjunction with free versions of Socrative and Zuvio, and engage students with the questions made for the lesson they are teaching.
There were six reading passages assigned in advance of every class meeting. Each passage contains 600-800 words and focuses on one single issue. Students who registered for the courses were approached by the course instructors to discuss the research agenda. The instructor informed the students that the courses would be facilitated using Socrative or Zuvio. The instructors provided the explanations of both systems. Then, the students were asked for their consent with a choice to opt out of the study. Their course grades were not affected by their decision in any way. Finally, the students were guaranteed that their names and academic records would not going to be included in the study.

Research design and instrumentation
A convergent parallel mixed methods design was used to discover if the use of Socrative or Zuvio increases the value of the questioning activity, and if the two services affect student achievement and their perceptions towards learning English. In this design, the researchers collected qualitative and quantitative date concurrently, analyzed them separately, and then merged both data sets. The reason for collecting both quantitative and qualitative data is to have multiple perspectives on the impact of online student response technology both with students' own devices and without. For answering the 1st research question (Can the use of online student response systems, combined with active calling technique, affect student reading and vocabulary test scores in large classes?), test scores were used to test the hypothesis predicting that the uses of online student response service would positively influence the reading comprehension skill for the first year students in the two-year program. Also, the survey results were used to understand how these students respond to the classical QA activity and its two technology-enhanced variations, which would help the research team to answer the second and third research questions (Does the use of Socrative or Zuvio enhance student engagement? What is the impact of using Socrative or Zuvio on students' perceptions and attitudes towards learning English language in large classroom environments?).
To seek information that might not had been found in the survey, the researchers conducted individual interviews with a total of 12 students who were randomly selected from two OSRS groups using a random number generator (six students from each group) for more input on the QA activity. On the scheduled dates, the researchers met with the selected students at the office and followed a designed interview protocol, consisting of seven open-ended questions (see Appendix B). The interviews lasted from 30 to 60 minutes. They were audio recorded, carefully transcribed and analyzed. The researchers employed thematic content analysis to find common patterns across the interview data. The four steps that the researchers follow are: 1) reading through the transcript interview responses several times; 2) coding the whole text; 3) searching for themes or patterns within the data; 4) creating a narrative that includes quotes from the teachers.
Being randomly assigned, one class (G2, n = 32) used Socrative, the other classes used Zuvio (G3, n = 45), and still another class did not use either of the two services (G1, n = 41). It was expected that a cause-effect relationship exists between the use of OSRSs and the accuracy by which students used their reading and vocabulary skills they were taught in the course as measured by their performance on pretest and posttest written by the researchers. The two tests were used as a summative assessment tool to assess English ability of newly admitted students, to ascertain the students' acquisition of the reading and vocabulary skills that they learn in the class, and to detect any change that OSRS brings. Both tests are identical with 25 multiple choice questions, worth four point per question. One administrative assistant scored all tests using an answer card reader and an answer key. The researchers of the study, who taught English reading more than 10 years, were responsible to design the pretest, posttest, and answer key. After revisiting the overall objectives for the courses and determining which goals we intended to evaluate with the tests, we created an objective test with 50 multiple-choice questions to measure students' performance in reading comprehension and vocabulary knowledge. The test items were then assessed on the basis of three key criteria: representative, ambiguity and clarity (Angleitner et al., 1986;Delgado-Rico et al., 2012). The same tests were administered to the same small group of three students who enrolled in the night nursing program twice at different time points. The test-retest correlation between the two sets of their scores was at 0.85, indicating a good reliability.
Before starting to integrate Socrative or Zuvio into lectures, the teacher held a 20-minute introductory session with a PowerPoint or Keynote presentation in a computer lab, and made sure all students had a smartphone and access to internet. The students were then asked to (a) visit Socrative (www.socrative.com) or Zuvio (www.zuvio.com. tw) homepage and sign up with their English first name, last name, email and password, (b) look for a confirmation email and click the link in that email to verify their email address, (c) download the Socrative or Zuvio application to their iOS or Android smartphone, (d) log in with their registered email and password, (e) enter a code provided by the teacher so they can join the Socrative or Zuvio presentation, and (f) respond to questions. The teacher supported the students while they worked through the steps.

Data collection procedure
The class met once a week for 120 minutes for the course lecture and activities. The on-ground classroom teachers implemented two five-minute teacher-led sessions that incorporated a Q&A active learning activity each session (Q&A format was the only one used so the relationship might be able to be established to demonstrate it has an effect without another factor that can explain the relationship as well)-one in the middle and the other in the end of the lecture to help students practice reading skills, as well as communicate the facts and ideas of the reading passage assigned for that week. During the session, the teacher summarized what has been taught within the first minute and asks one multiple choice question. Students will have 60-90 seconds to think and formulate a response. The teachers will use the last 2 minutes to give feedback for correct and incorrect responses. All of the groups in the study participate in these activities. The teachers use the same teaching materials and measurement instruments. The only difference between the control group (G1), Socrative group (G2), and Zuvio group (G3) is the method that the teacher used to gather student answers. In the control class, the teacher called on students to share their answers with the class aurally. In the OSRS classes, the teacher used Socrative/Zuvio to collect all of the students' responses anonymously.
The students in the Socrative and Zuvio groups answered the same questions as those given to the control group as well. They had 60-90 seconds to solve problems before casting their votes. The result of the voting for two OSRSs groups appeared on the teacher's computer and was projected on the screen. The teacher opened a whole class discussion where the students defended their choices. Finally, the teacher highlighted the correct option and explained why it was the correct one and why others were not. Figure 1 shows an illustration of the three variants of the QA session, adapted from the SRS flowchart by Arnesen et al. (2013).
Because our students were randomly placed into the course based on their total entrance exam scores instead of their English subscores, it was first necessary to measure English proficiency of the three groups to better understand the effects of the interventions. For this reason, OSRS was not implemented in the class until the fourth week of the course. The pretest was administered by two trained student research assistants to all 118 students in the third week of the semester, and the posttest in the 10th week. A modified version of the Student Response to Instructional Practices Survey (see Appendix A, written permission is obtained to use the survey), using a five-point Lik-ert scale was administered twice, first in the third week and again in the 10th week to determine if there is any change in students engagement after using the OSRS (DeMonburn et al., 2017).
One-way Analysis of Variance (ANOVA) was used to compare the means of the three groups and see which approach creates a highest level of student engagement and academic performance in large classes. The independent variable (IV) in the study is the type of Question & Answer activity, whereas the two dependent variables (DV) are student responses to instruction and test scores.
The null hypothesis for the one-way ANOVA is that there is no significant difference among the groups. After cleaning the data, the researchers tested the assumptions of ANOVA by calculating the F-ratio and the associated probability value (p-value). If the p-value associated with the F is smaller than .05 then the null hypothesis is rejected. If the null hypothesis is rejected, we conclude that the means of all the groups are not equal, and we run post-hoc tests (t tests) to examine where the group differences lay.

RQ1. Can the use of Socrative or Zuvio, combined with an active calling technique, affect stu-dent reading and vocabulary test scores in large classes?
The pretest and the posttest scores were first compared using an ANOVA test (see Table 1). The results suggest that no obvious difference (p = 0.38 for pretest; p = 0.11 for posttest) was observed in each test among the three groups (G1, G2, G3). We proceeded with the Tukey post hoc test to see between-group difference, but found no significant difference in pairwise comparisons. How-ever, in the Socrative treatment condition (G2), a significant change was found from pretest to posttest (p = 0.0137). Table 1. Pretest and posttest scores.

RQ2. Does the use of Socrative or Zuvio enhance student engagement?
In response to the first statement (see Table 2), more than 65% of students in each OSRS group (G2, G3) reported they almost never or seldom disengaged themselves from the question and an-swer activity, compared to the classical group (G1) with only 27%. 72% of G2 students participated actively, while less students were found in the other two groups (G1-55%; G3-46%). When asked whether or not they pretended to participate (levels of student engagement), 61% of the G2 student respondents indicated that they almost never pretended while less than half of the students in G1 (47%) and G3 (46%) did. Approximately half of the students in all groups agreed that the required effort/ responsibility helped them to learn the material. Table 2. Student response to instruction. Further, the statements or variables were classified into two major factors-participation/engaging and perceived value of activity-by using exploratory factor analysis (see Figure  2).

RQ3. What is the impact of using Socrative or Zuvio on students' perceptions and attitudes to-wards learning English language in large classroom environments?
Individual interviews were conducted with the 12 randomlyselected students (see Table 3) to elu-cidate the impact of the apps on their experiences in large classes. There were seven questions that students answered at the interviews (see Appendix B). Table 3. Profile of interviewed students.
By examining the interview transcripts of twelve participants, the researchers identified five themes (see Table 4) to progress further: accessibility, assessment, management, competition and challenge. The following direct quotations were originally written in Chinese and translated into English by the authors.
The two apps allowed every student an easy way to contribute their opinions through the use of their own mobile phone or tablet. One student said "I grab my smartphone and complete the quiz;" another student said "I can see the questions clearly on my phone and answer them quickly." Two students told the interviewers they liked the apps because the designs made it easy for them to an-swer questions. The apps had one question per page instead of multiple questions on a page. Stu-dents saw the question within a single viewable area of the screen and selected `submit answers` to go to the next page. One student suggested incorporating competition into the activity to make it more engaging. She said "To encourage participation, the Q&A activity can be redesigned to make students compete against each other by earning points based on answering questions correct-ly in the least amount of time." The OSRS supported teachers to control and monitor their classrooms. Teachers live polled stu-dents to see how well they understand the material, and controlled when feedback was sent to their screens, immediately or later. Also, the apps created the opportunity for teachers to track student login history and quickly generated a report that shows each student's answers to the quiz and their quiz score. Four student quotes about instructional affordances are: "It keeps records of our login/logout times," "The app made it easy for teachers to get all students involved in the activity," "Teachers instantly gather students' feedback and give comments," and "Teachers see the sta-tus of each student's work, their progress and answers".
Unfortunately, there were several constraints that may limit teaching and learning. First, students from the Socrative group said the app crashed multiple times, so they had to retake the quiz. One of the student quotes is "The app won't respond." From the Zuvio group, students reported they could not submit their answer multiple times to the same question. Once students clicked the `submit` they were unable to go back and change any answers. A few other students expressed concerns about cheating. They noticed some students comparing answers or looking to peers for cues, and thought such behavior was wrong because it provided an unfair advantage.

Discussion
In this study consisting of three groups of undergraduate students, we found that the six-week ex-periment of Socrative substantially increased reading comprehension and vocabulary. Most of the participants benefited from using Socrative compared with using Zuvio, and the average increase in test scores was around 8.5%. When using Socrative, the participants also became involved more with the question-answer activity. Simply being involved more, Table 4. Themes and quotations. however, was not the sole mecha-nism for the increased test performance. When using Socrative, the participants were more in-volved in the activity, and more of those involvements were active.
A higher percentage of students using OSRS participated actively and stayed focused for most (>90%) of the questionanswer activity than the classical group. This matches Lim's (2017) find-ing that Socrative improved concentration in the classroom. The positive change could be partially due to the students' anticipation of obtaining a fast, personalized and private feedback for their work (Freeman et al., 2006). The improvement might be caused by novelty. Students did better because the classroom became different and interesting. This novelty effect will probably fade away as soon as students gradually become used to the systems. Another form of bias that we could not avoid was the Hawthorne effect-students work harder and perform better when they are being observed. To reduce such effect, we decided to conduct our study as part of a lesson cycle so students would be more likely to act naturally.
When assessing Socrative and Zuvio, researchers found their free versions easy to set up, use and administer. Without calling on students for answers, teachers were able to evaluate particular needs of students. While students had grown up in a world surrounded by technology, it would be dangerous to assume each of them could use Socrative or Zuvio immediately after a short training session. Most students still needed teachers as a guide to help them use these digital apps in order to stay engaged in the learning activities. Researchers also found they preferred the app stability with Zuvio overall, but multiple submission of Socrative was necessary if students wanted to make updates to their submitted answers.

Limitations
The limitations of this study should be noted. First, after we completed our interpretation of the findings, we discovered that the way in which we collected the data limited our ability to conduct a full analysis of the results. This study suggests a need in future research to revise the survey for gathering more information on different levels of familiarity with tech or different levels of dis-positions towards the use of tech in the classroom. Second, the time available to investigate our research problems and to measure change within the students' reading skills is constrained by the end date of our one-year grant. The experiment was conducted out over only six weeks, so it was too short to conclusively determine that the true effect of using OSRS on academic performance. The treatment might be effective only when it was new to students, and the success would not be repeated over the long term. To increase our ability to measure the effect, longer intervention du-rations (Chwo et al., 2018) of eight weeks or more, are recommended for future studies focused on student response systems and active learning strategies.

Conclusion
It is clear that the two apps Socrative and Zuvio can provide the desired anonymity, instant per-sonalized feedback and multi-sensory instruction, which were positively linked to student willing-ness to actively participate in large class activities. While openly asking questions and allowing all students to process and respond to teacher prompts is now possible on Zoom with a large class of undergraduates students in a typical university classroom, the findings in the study are im-portant for teachers who are struggling to engage all of their students, especially when the class offers in-person and online learning at the same time. The study also needs to be replicated with a longer intervention plan to judge with confidence whether or not the use of questionanswer activ-ities in conjunction with the online student response systems improves test scores in vocabulary and reading comprehension tests.