{'fraction_non_alphanumeric': 0.03578226113437381, 'fraction_numerical': 0.014921964217738866, 'mean_word_length': 5.0229252636405315, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students

Purpose This study aims to understand the characteristics of smart device-based testing (SBT) by comparing the typical characteristics of students’ satisfaction with SBT, its usefulness, advantages, and disadvantages when compared with existing testing methods. Methods A total of 250 students from the first to third year were selected as the final targets of the study and the questionnaire was developed by faculty members who participated in the survey from the start of the SBT. The total number of questions is 12, and the questionnaire used a 4-point scale. The data obtained were analyzed using the IBM SPSS ver. 23.0 (IBM Corp., Armonk, USA). Results Answers to the “satisfaction with SBT” were generally negative, while answers to the “usefulness of SBT” were generally positive. There was no difference in satisfaction across gender and smart device ownership, whereas there were significant differences across years. With reference to the usefulness of SBT, students responded positively, while about the overall configuration and completeness of SBT, students responded negatively. Students also seemed to show a greater preference toward the pencil-paper test. Conclusion On the other hand, students generally thought that SBT helped to assess medical knowledge better and was a more objective method of knowledge assessment than a pencil-paper test. We believe that students preferred the traditional paper-pencil test due to their unfamiliarity with SBT. We believe that an appropriate and careful remedy for drawbacks of the SBT will have a significant impact in the accumulation of actual clinical knowledge and in the improvement of practical skills for medical students.


Introduction
The development of the computer along with the advances in information and communication technologies has brought about changes in many areas related to education. It is also changing the field of medical education and computer-based tests (CBT), and many institutions are already replacing the paper-pencil test as a method of assessment. With the recent development of smart devices and their high penetration, smart device-based testing (SBT) is gaining prominence as a useful and accurate evaluation method in various areas of education [1,2]. Compared to the paper-pencil test, SBT has many more advantages such as cost reduction and improvement of learning efficiency [3,4], as well as space utilization and mobility, which are cited as shortcomings in CBT. Hence, expectations from SBTs are growing. This transition is expected to become a necessity in assessment tools because the audio-visual materials that are in use in traditional medical education today are more effective through platforms such as e-learning; these platforms utilize the strengths of technology and networks in conjunction with classroom training [5][6][7][8].
In the field of medical education, SBTs are expected to have the following advantages: First, printing and administrative costs spent in preparation of the test can be saved. Second, it can save manpower, time, and expenses required to assess, compute results, and facilitate analysis and feedback [9,10]. Third, it can apply various types of questions through multimedia data. In practice, medical judgment on a patient's treatment or progress is very important when using dynamic imaging information such as computed tomography and ultrasonography. If medical education and evaluation are carried out using such multimedia information, it is thought to have a good effect on actual clinical knowledge accumulation and improvement of practical functions.
In order for SBT to be recognized as a useful and efficient assessment tool in the field of medical education and to establish its status, various kinds of studies are required, and the actual characteristics of SBT must be identified. Studies on SBT as an assessment tool are gradually increasing and domestic and international studies have been published on smart device ownership, proficiency, preference, and so forth, or its relation to existing pencil-paper tests [10][11][12][13]. However, previous studies in Korea limited either the length of use or the number of people involved. And almost studies were conducted on a single year.
Therefore, we planned this study, based on 3 years of SBT operational experience, for medical students from the first to third year at medical schools or medical graduate schools to understand the characteristics of SBT as a means of assessment, by comparing the students' satisfaction with SBT, its usefulness, advantages, and disadvantages with existing testing methods.

Statistical analysis
The data investigated were analyzed using the IBM SPSS ver. 23.0 (IBM Corp., Armonk, USA

General characteristics of the subject
The response rate of survey was 98%. The average age of the subjects was 24.2±2.7 years, comprising of 155 men (61.5%) and 95 women (37.7%). A total of 164 students (65.1%) owned smart devices. With respect to the SBT experience, "more than 6 times" made up 75%, while "2 to 5 times" comprised 23.4% of the participants.
One student had never experienced SBT before and one who had experienced SBT only once (Table 1).

Satisfaction with SBT
With respect to the convenience of a smart device, 12.3% answered, "strongly agree" and 41.3% responded with "agree." When asked whether they were satisfied with the overall composition and completeness of the smart device test, 38.9% answered "disagree" and 17.9% reported, "strongly disagree." When asked whether they preferred SBT over traditional paper testing methods, 38.9% answered "disagree" and 25.4% responded, "strongly disagree." When asked about applying the SBT to other subjects, 42.5% answered "disagree" and 15.9% reported, "strongly disagree." When asked about taking the same test using computers instead of smart devices, 40.5% answered "disagree" and 22.6% responded, "strongly disagree." Others suggested that SBT could be used to refer to images and photographic materials on smart devices ( Table 2). The score for SBT was 4.8±2.4 on a scale of 10.

Advantages, disadvantages and other features of SBT
The following were the advantages of SBT, which is indicated by the following data: 168 (69.7%) answered that "it is easy to recognize letters or picture" and 131 (54.4%) answered "possibility of multi-media implementation, such as video and sound." A total of 86 (35.7%) answered "test process is more convenient than a paper test," 41 (17.0%) answered "no possibility of cheating," 35 (14.5%) answered "reduced test time," and 19 (7.9%) answered "increased test concentration." Another answer included "There was no mistake in marking the OMR." The following disadvantages were indicated by the data: A total of 161 (64.7%) answered "unable to write on exam paper," 144 (57.8%) answered "system error during test," 144 (57.8%) answered "difficult to enter letters and numbers," 105 (42.2%) answered "dissatisfaction with the order of the test questions," 65 (26.1%) answered "slow pad drive," 25 (10.0%) answered "lack of pad mounting functions such as bookmarks," 14 (5.6%) answered "smaller screen than a paper test," 11 (4.4%) answered  "difficult to recognize letters or pictures," and 3 (1.2%) answered "possibly of cheating such as hacking." Other answers included "it's hard to get back to the test questions" and "can't see the whole problem at one glance." Other opinions on SBT were "it is recommended to provide a Bluetooth keyboard rather than a smart pen for a descriptive problem" and "pads don't perform well," and pointed out poor operations in the preparation of exams.

Analysis according to gender, year, and smart device ownership
Satisfaction and usefulness of the SBT by gender did not convey significant difference statistically. However, convenience, satisfaction, and preference of SBT by different years displayed a significant difference as first and second year students mainly gave negative responses, while third year students mainly gave positive responses.
Third year students answered with more positive responses, especially with regard to the usefulness of SBT (Table 3). In the analysis by smart device ownership, with respect to the overall satisfaction with SBT, smart device owners mainly responded negatively, while non-owners mainly responded positively, which displayed a significant difference according to smart device ownership. There were no significant statistical differences in other areas according to smart device ownership (Table 4).

Discussion
Although SBT is claimed to be a useful objective assessment tool, this alone does not warrant a complete replacement of the pencil-paper test. SBT will only be able to settle into a complete medical education assessment tool when we understand exactly how SBT is accepted by actual examinees, identify various features in the stages of exam preparation and completion, and remedy any related drawbacks. To that extent, this study was conducted in order to understand the current state of SBT in medical education and help overcome any shortcomings.
As mentioned above, for questions related to the convenience of smart devices, students generally gave a positive response, while for questions related to the overall configuration and completeness of SBT, students mainly gave a negative response. Students also seemed to prefer the traditional pencil-paper test. On the other hand, students generally thought SBT helped assess medical knowledge better and that it is a more objective method of knowledge evaluation than a pencil-paper test. This led us to believe that students just preferred the traditional paper-pencil test due to their unfamiliarity with SBT in the context of medical school examinations.
With respect to potential SBT application in national medical license examination, the concept of adding smart devices in pencil-paper tests for providing media and photographic materials has been suggested several times.
This use of smart devices is believed to be the greatest benefit of SBT in that it incorporates dynamic and visual forms of additional information with the existing written information. This was also consistent with the fact that students generally reported a convenience in the recognition of letters and pictures and the incorporation of multimedia images as one of the benefits of using SBT.
The perception of SBT according to gender proved to be of no significant difference. Lim et al. [13] proved that SBT was preferred over the pencil-paper test and CBT, and there were no significant differences in the analysis of satisfaction and preference according to gender, age, and so forth. In addition, the study by Lazarus et al. [14] proved that there was no difference in the perception of the tablet technology in the field of medical education with respect to racial or socioeconomic differences. Therefore, we concluded that SBT could be a fair assessment method.
Meanwhile, the study by Lim et al. [13] proved that SBT was the most preferred, but in our study, the students preferred the pencil-paper test the most, followed by SBT and CBT. Our results, as mentioned earlier, may be due to preference for traditional methods or poor operations in the preparation of exams. In the previous study, it was found that there was no significant relationship between SBT exam scores and ownership of smart devices, and that SBT exam scores were neither affected by gender nor smart device proficiency. In addition, the association between accumulative rank in the pencil-paper test and SBT scores was found to be significant [11], and a study by Kim and Hwang [12], proved that the difficulty of pencil-paper test and the multimedia test using tablets was not significantly different [12]. Therefore, it is believed that SBT will not have any special effect on the results even if it replaces the existing pencil-paper test.
The limitation of this study is that a small number of assessments and sample size does not allow it to be a generalized study. More accurate results can be obtained if data from various schools, different years, and diverse subjects are collected in the near future. Moreover, it seemed that the students under study tend to focus more on the functional problems of the device or on the regulation of the study itself, rather than on the perception of the SBT method.
Smart devices for learning and/or testing are widely being utilized in the field of medical education in many foreign countries [15]. Consistent with the study by Baumgart et al. [16] that demonstrates improved exam performance using tablet learning, the development of such smart devices will improve the efficiency of medical education. Accordingly, this study analyzed the general perception of SBT with respect to its pros and cons. Hence, we believe that appropriate and careful remedy for drawbacks of the SBT, without a doubt, will have a significant impact in the accumulation of actual clinical knowledge and improvement of practical skills for medical students.