{'fraction_non_alphanumeric': 0.029960981047937568, 'fraction_numerical': 0.01511984392419175, 'mean_word_length': 4.884788847888479, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Investigation of first-year university students’ performance

The students enrolled in introductory physics courses at Loránd Eötvös University (ELTE) are tested with the Mechanics Baseline Test (MBT) since 2016. It is used as a pre-test at the beginning of the first semester and as a post-test at the beginning of the second one. In this study, an account is given about the newest results of the test and about our endeavor to make introductory courses more effective. Therefore, the effect of the size of the study group, the reduction of curriculum and motivation on students’ performance were investigated. The preliminary results show that smaller groups are more efficient, and the reduction of the curriculum does not play a major role in the possession of the knowledge concerned. Conversely, the motivation of the students increases the understanding of concepts and therefore, the performance at MBT.


Introduction
According to nation-wide surveys, the level of physics knowledge of secondary school students has continuously decreased in the last 20 years. There are several reasons for that. First, it is well-known that the students' attitude towards science, including physics, is now quite negative. This is especially true for Hungary, where the unpopularity of physics has been increasing. This strongly influences the physics knowledge of the students enrolled in universities.
Since 2016, the students enrolled in introductory physics courses at Eötvös University (ELTE) are tested with the Mechanics Baseline Test (MBT) constructed by Hestenes and Wells [1]. The MBT has been used all over the world to assess the level of understanding of basic Newtonian concepts. It is universal in a sense that students at any level can be tested by it (except for students at advanced physics curses). It can be used either as a placement test or for post-instruction evaluation. We use it to estimate the physics knowledge of first-year students with majors of Earth Sciences, Physics and Physics Teaching. Students are tested in the first week of the semester when they start attending introductory physics courses (pre-test), and also the effectiveness of the introductory physics courses is measured by testing them in the first week of the following semester (post-test). Only basic parameters, like the mean value or standard deviation, were calculated, and the group as a whole was evaluated of the first surveys [2]. The MBT was also used to determine the topics, which are extremely difficult for students at ELTE in kinematics and dynamics to give special attention to these topics at practice lessons.
According to our survey, only 20 % of the Physics BSc students are able to reach the "master level", which is defined by 0.8 on average on the MBT test [1]. Contrarily, more than 60 % of the students with Earth Sciences, Physics or Physics Teaching major cannot achieve more than 0.4 on average on the MBT. It indicates that these students have significant deficiencies in their physics knowledge as they begin their introductory physics courses. We concluded that, in accordance with McDermott's previous experience [3], our first-year students are not able to recognize what they do not understand and how to ask themselves the types of questions necessary for the functional understanding of the material. This suggests that to achieve meaningful learning, students need more assistance than they can obtain through listening to lectures, reading textbooks, and solving standard quantitative problems. As a first attempt "tutorial physics criterion courses" were introduced in the physics program at ELTE. These courses were mandatory, and the concepts were taught through problem-solving. However, in our previous investigation, we found that the advances of students due to these courses were considerable only at groups of small size (the number of students was less than 20) [2]. In groups with more than 25 students, the changes in the results of the pre-and post-MBT were negligible (less than 5 %). It seems that the deficiency in knowledge cannot be decreased efficiently merely by adding more courses, but the size of the group may have an effect on the efficiency. A possible reason for that could be that the learning methods and the ways students obtain information have also changed nowadays. The lifestyle and socialization of the new generation of students (Net Generation) are strongly different from those of the earlier ones. They "socialize on colorful, rapidly changing, exciting films and hypes full of movements, they have got used to watching one-tenth-second-long scenes rather than longer ones" [4]. Therefore, the methods, which were effective 20 years ago, are no longer effective, and the consequences are well noticeable even at universities.
In this study, we continued to map the factors that may influence the physics knowledge of ELTE students, such as the number of students in the physics courses, the reduction of the curriculum and enhancement of engagement. For this more detailed investigation, the previously applied evaluation method was not sufficient; therefore, codes were introduced to follow the performance of each individual.

Methods
The MBT was used in the same way as in our previous investigation, i.e. the students at introductory physics courses were tested in the first week of the semester (pre-test) and also in the first week of the following semester (post-test). In the first two years, we found that although the students had 45 minutes to complete the test, many of them finished it earlier. However, the results of the last few questions are below average, so we concluded that the students' concentration may decrease during test writing. (26 items could be too much for them). Therefore, we decided to shorten the MBT according to the item response analysis of Cardamone and co-workers [5]. According to their research, for the discrimination between the students of different abilities, a subset of MBT with 15 questions is enough. From 2018, we used this subset of 15 items completed with Problem No. 5, which concerns the acceleration in non-linear motion [1]. (The understanding of this topic is relevant from the point of view of ELTE, and the reliability of the test was not influenced by the addition of item 5.) In Hungary most of the students are taught conventionally during primary and secondary school; therefore, other methods, like cooperative methods or the Mazur method [6] are hard to apply since students react to these non-conventional methods quite slowly [7]. Hence, we investigated the effect of different methods that can be implemented in the framework of conventional teaching [8]. First, we reduced the curriculum (2017/18), then, in order to increase the engagement of the students, instead of midterm exams, we tested the students by four mandatory homework assignments (2018/19). The assignment consisted of 5 problems; each of them contained 4 to 6 items, similar to the tutorials of McDermott Group [9].
The curriculum remained the same as in 2017/18. During these three years, the effect of the size of the group on the average performance, and from 2017, the changes in individual performance were evaluated. In addition, the results were evaluated by the SPSS software.

Comparing means
The results of the investigation performed with the MBT both at the beginning of the first semester (pre-test) and the beginning of the second semester, after finishing the introductory mechanics course (post-test) in the last three years are shown in Table 1. Although improvement is visible between the pre-and post-test in the groups of Physics BSc, most of the students in these groups do not exceed the conceptual threshold even after one semester of introductory mechanics course (4x45 minutes lectures and 4x45 or 2x45 minutes practical course a week, 14 weeks). (The conceptual and mastery threshold for basic Newtonian concept is defined as 0.6 and 0.8 on average, respectively [1].) The situation is somewhat different for the students in the Teacher Training Program. The students in these groups can reach the conceptual level after a semester, but still far from the mastery threshold. Contrarily, in the case of the Earth Science students, despite learning physics for a full semester (3x45 minutes lectures and 2x45 minutes practical course a week, 14 weeks), the difference between the pre-and post-tests is negligible ( Table 1). The absence of improvement in physics is probably due to the lack of interest: students with major of Earth Science focus rather on their main topics, geology and geography, and care less about learning physics. It is worth noting, that the physics courses have been reduced drastically in the curriculum of Earth Science Program since 2018, therefore, we think that from the further investigation of these students a substantial contribution to understanding student knowledge problems cannot be gained. Hence, hereinafter we do not include the data from Earth Science students.

Follow-up evaluation
The evaluation of the data of the first year of testing revealed that the results of the tests taken anonymously cannot be easily interpreted. For instance, the students who left the university after a few months because of failing an exam were pre-tested but were not post-tested, making the sampling biased. Hence, from the second year of the experiment, we asked the students to identify themselves. This made us able to only evaluate the results of those students, who were both pre-and post-tested. Table 2 shows the results of these students. It can be seen that students' results in each group improved significantly. We used Cohen's d method to measure the improvement of the students [10]. The results are shown in Table 3. Following the Sawilowsky's rule of thumb, 2017/18 and 2018/19 Teacher students' improvements are considered to be large (1.2>d>0.8) and very large (d>1.2), while the improvements of Physics BSc students in 2017/18 and 2018/19 are considered small (d<0.5) and medium (0.8>d>0.5), respectively [11]. These results show two trends. First, the improvement of the students in the teacher training program in physics is better than the improvement of the students in Physics BSc. Second, the students in 2018/19 improved more than the year before.
The greater improvement of the students with Physics Teaching major can be the consequence of two factors. First, at ELTE students participating in the teacher training program receive more focus from teachers and mentors due to the small number of applicants. Also, the motivation of these students is higher. The difference in the improvement obtained between the two years can be attributed to the change in the teaching methodology described in the Methods section. It means that the increased amounts of individual work can help in deepening the meaningful understanding of basic concepts of physics. It is worth noting that we think that the introduction of homework assignments was not only a good way to increase the time spent with physics problems but also initiated conversation among students (especially among students in Physics Teacher Training) about physics, similarly to the Mazur method.

Analysis of covariance
Due to student identification which was applied in the last two years, we can determine the factors affecting the improvement of students' performance in the light of the MBT. The following factors were investigated with the ANCOVA method: the sex, the high school physics final exam level (mid or high level), the significance of which practical course group was the student in, the practical course mark and the lecture course mark. We used all the four sets of data available for the analysis (Physics BSc 2017/18, Physics BSc 2018/19, Physics Teacher 2017/18 and Physics Teacher 2018/19) despite the fact this way the sample cannot be considered as a truly homogeneous sample. However, this allowed the usage of statistical approach with a total sample size of 86 students. The result of the posttest was used as the dependent variable, while the results of the pre-test were defined as a covariate. In this investigation three factors were found insignificant, the sex (p=0.074), the level of the final exam (p=0.600) and the significance of which practical course group was the student in (p=0.271). Conversely, the practical course mark (p=0.050) and the lecture course mark (p=0.011) showed a positive correlation with the test results, implying that the marks are consistent with the physics knowledge of the students.
Despite the fact that practical course group factor showed little impact on performance, further investigation revealed that the group size had an effect on the performance in Physics BSc groups. While in 2017/18, the size of the groups was the same along with the score, in 2018/19, however, the increase in the group-size caused a visible decrease in the results (Fig. 1). Moreover, the smaller Physics BSc group (n=14) performed comparably to the Teacher Training group in the same year, while the larger one (n=35) performed significantly worse.

Conclusion
In recent years, there has been a consensus among university professors that most of the first-year students are lacking the knowledge of physics and mathematics necessary to master, without extra help, the basic concepts of the introductory physics. We have tried to objectively examine the background of this fact and to find out how students' content knowledge can be improved during the first semester. To measure university students' improvement in mechanics during the first semester, we have adopted Hestenes' Mechanics Baseline Test. Our results have shown that requiring increased amounts of individual work from the students resulted in greater improvement. Physics teacher students' performance improved more than that of Physics BSc students, partly due to their strong commitment. The obtained mechanics marks in the first semester showed a positive correlation with the test results of the students in each training group, implying that the marking system is consistent with the physics knowledge of the students. Our results also showed that smaller group sizes can be more effective. Based on these findings, we have prepared some new textbooks and changed the tutorial methods.