{'fraction_non_alphanumeric': 0.028790374561340686, 'fraction_numerical': 0.011100766310964693, 'mean_word_length': 5.429097605893186, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Rating criteria to evaluate student performance in digital wax-up training using multi-purpose software

PURPOSE The aim of this study was to introduce rating criteria to evaluate student performance in a newly developed, digital wax-up preclinical program for computer-aided design (CAD) of full-coverage crowns and preliminarily investigate the reliability and internal consistency of the rating system. MATERIALS AND METHODS This study, conducted in 2017, enrolled 47 fifth-year dental students of Okayama University Dental School. Digital wax-up training included a fundamental practice using computer graphics (CG), multipurpose CAD software programs, and an advanced practice to execute a digital wax-up of the right mandibular second molar (#47). Each student’s digital wax-up work (stereolithography data) was evaluated by two instructors using seven qualitative criteria. The total qualitative score (0-90) of the criteria was calculated. The total volumetric discrepancy between each student’s digital wax-up work and a reference prepared by an instructor was automatically measured by the CAD software. The inter-rater reliability of each criterion was analyzed using a weighted kappa index. The relationship between the total volume discrepancy and the total qualitative score was analyzed using Spearman’s correlation. RESULTS The weighted kappa values for the seven qualitative criteria ranged from 0.62 - 0.93. The total qualitative score and the total volumetric discrepancy were negatively correlated (ρ = -0.27, P = .09, respectively); however, this was not statistically significant. CONCLUSION The established qualitative criteria to evaluate students’ work showed sufficiently high inter-rater reliability; however, the digitally measured volumetric discrepancy could not sufficiently predict the total qualitative score.


INTRODUCTION
Remarkable developments in digital technology have increased the popularity of dental prostheses fabricated using computer-aided design-computer-aided manufacturing (CAD-CAM) technology in clinical settings. [1][2][3][4][5] Accordingly, there is an increased need to provide technical skill training using CAD-CAM technology in undergraduate preclinical practice in dental schools. 1,4,6,7 However, overcoming the high costs of preparing and maintaining CAD-CAM systems and CAD software programs is a major challenge since hands-on training exclusively uses commercially available CAD-CAM systems for the fabrication of crown and bridge prostheses. 1 Furthermore, a validated and reliable evaluation system is needed to assess students' achievements in undergraduate hands-on training. In conventional wax-up training for the fabrication of crown-bridge restorations or tooth sculpting training with plaster, student products are only visually evaluated by instructors. 8,9 Evaluation bias and variability in grade evaluation still exist despite sufficient calibration of the instructors. [10][11][12][13][14][15][16] Therefore, recent studies have indicated the necessity of an objective evaluation system based on digital technology. 10,15,[17][18][19][20] Low-cost multipurpose CAD software was utilized for several years in our institution to train students in designing dental prostheses. As this software is inexpensive and easily available online, students can install the software onto their computers. Moreover, the software is not restricted to designing dental prostheses; therefore, it can be used for three-dimensional (3D) design and assessment of certain digital products. These advantages led us to develop a new 3D digital designing education system for crown restorations (a digital wax-up training system) capable of evaluating students' works.
The 3D volumetric difference between the students' work and a reference prepared by an instructor as an evaluation tool was utilized. 21 This evaluation method aids in depicting the convex and concave portions on the 3D surface data derived from the reference in the 3D cyberspace. Kwon et al. 15 reported that the 3D volumetric difference comparing the digitally scanned surface data of the wax-up product to the reference data showed high reliability. However, Garrett et al. 19 reported that estimations based on conventional visual evaluation, which is crucial in a clinical setting, were not significantly correlated to estimations based on the digital quantitative evaluation. This incongruence was partly attributed to lower reliability in the conventional visual evaluation criteria and partly to the lack of essential evaluation items relevant to clinical significance in the digital quantitative evaluation. Therefore, a qualitative evaluation system was developed to evaluate students' digital wax-up performance with essential tools relevant to clinical significance with improved reliability, using the multidirectional visibility of the digital quantitative evaluation system. To the best of our knowledge, no previous study has successfully evaluated the reliability of qualitative morphological data of digital wax-up work using CAD software.
This study aimed to introduce rating criteria to evaluate student performance in a newly developed preclinical hands-on student training protocol of designing crown restorations using multi-purpose CAD and computer graphics (CG) software programs and to perform a preliminary estimation of the reliability and internal consistency of the protocol's criteria for evaluating students' designing performance. The inter-rater reliability (reproducibility) of the rating criteria proposed in this study was evaluated according to a reliability-estimating standard by Landis and Koch. 22 The null hypothesis tested in this study was that there would be no statistical correlation between the total volumetric discrepancy between each student's work and an instructor's reference and the total qualitative score of the rating criteria proposed. The fundamental practice using a CG software program sessions (volumes 1 and 2) included information regarding import and export of the 3D data (.obj as a Wavefront OBJ format) and moving and orienting the object (rotate, zoom in, and zoom out). Three practical cases were prepared for students to enable recovering the form of the artificially deformed #47 without occlusal surface undulations and with an ill-fitted marginal line.

MATERIALS AND METHODS
The fundamental practice using multipurpose CAD software program session included information regarding how to rotate, move, split, and mirror the entire 3D mandibular dentition data. In this practice, the methods for importing and exporting the 3D data were also included (.obj and .stl in stereolithography The occlusal surface data (.obj) were converted to STL data and transferred back into the CAD software at the pre-set location. The margin line in the 3D data of #47 was drawn manually by each student using the 3D data from the abutment tooth (.stl). The obtained margin line data were then merged with the main 3D data (.stl).
The surface data of the abutment were digitally relieved to allow appropriate cement space in the main 3D data (.stl) to build the inner surface of the restoration. The occlusal surface 3D data (.stl) was integrated into the margin line with an axial wall surface to build the outer surface. The inner and outer surfaces were automatically converged at the margin line of the abutment tooth. Lastly, the line angle, subgingival contour, gingival contour crest, and proximal contact shape were optimized according to the morphology of the opposite corresponding tooth (#37) (Fig. 1C).
The integrated inner and outer surface data of each student were exported to STL data to fabricate a composite resin crown based on the student's design. Each finished STL datum that students designed was used for their work evaluation. Two instructors separately evaluated the digital work on a high-resolution liquid crystal display monitor (1366 x 768) using the qualitative assessment criteria with the same magnification set for each evaluation item ( Table 1).
The evaluation time for each criterion for a single student's work was less than one minute. The inter-rater consistency of the two assessment scores for each criterion was calculated to investigate the inter-rater reliability of the qualitative assessment criteria. The average of the two instructors' assessment scores was used as the final score of each assessment criterion. The total score of all assessment criteria for a student's digital work was calculated as the sum of the total qualitative score of their digital wax-up work.
The total 3D volume of each student's digital waxup work was digitally measured and compared to that of the instructor's reference restoration using commercially available CAD software (Power Shape; Autodesk, San Rafael, CA, USA) to quantitatively evaluate the students' works. STL data that students and the instructor designed were used for the evaluation.
Specifically, the total volumetric discrepancy between each student's work and the instructor's reference was quantified as the total volumetric difference (mm 3 ), including the concave and convex volumes relative to the surface of the reference. 21 The relationship between the total volumetric discrepancy and the total qualitative score was analyzed.
All students were requested to fill out a self-admin- For the statistical analysis, the inter-rater reliability of the qualitative assessment criteria was analyzed using a weighted kappa index. The achievement rate of each qualitative criterion was calculated using the following formula: score achieved by student/the highest score in each criterion x 100 (%). The relationship between the total volumetric discrepancy (mm 3 ) and the total qualitative score of the students' digital wax-up products was analyzed using Spearman's correlation.    Table 2. The mean achievement rates of the crown margin integrity, accessory ridge, and groove criteria were > 80%; however, the mean rates of the occlusal contact and axial crown contour criteria were < 50%. The mean total achievement rate of the students was 59.4% ± 17.5% (highest, 88.9%; lowest, 33.3%).
The weighted kappa values of the seven qualita-tive criteria ranged from 0.62 to 0.93. According to Landis and Koch, the reliability level of the qualitative assessment criteria can be estimated as more than "substantial". 22 The total volumetric discrepancy seemed to be negatively correlated with the total qualitative score of the students' digital wax-up work. However, no statistically significant correlation was observed between the total volumetric discrepancy and the total qualitative score of their digital wax-up work (ρ = -0.27, P = .09).
All students answered the self-administered questionnaire, with the statements and results of the questionnaire shown in Table 3.   The Journal of Advanced Prosthodontics all the evaluation criteria was evaluated as "substantial agreement" or "almost perfect or perfect agreement." 22 In a previous study in which instructors visually evaluated the reliability of students' conventional crown wax-up products, the inter-rater reliability (measured using the intraclass correlation coefficient) was < 0.4. 15 Therefore, the criteria for the qualitative evaluation established in the digital wax-up practice seemed sufficiently operational and showed sufficient inter-rater reliability.
A substantially negative but not statistically significant correlation between the total volumetric discrepancy and the total qualitative score was observed, indicating that the standard evaluation under digital circumstances seemed to be introduced by qualitative assessment based on clinical requisites.
The same statistical trend of a weak correlation between the rubric evaluation and the digital quantitative evaluation of some abutment tooth preparation practices was reported in two previous studies, with varying statistical significances due to diverse sample sizes. 20,23 Thus, the total volumetric discrepancy measure can describe the overall estimation of the restorations, but using this measure solely may not be sufficient for evaluating the criteria shown in Table 1.
However, we believe artificial intelligence technology will assist in creating an automatic qualitative evaluation methodology to easily estimate the students' work and the process of the practice.
Regarding the self-administered student questionnaire, approximately 60% of the students strongly agreed or agreed to the question "Did you master basic operation procedures of the CG and CAD software programs?" Basic operational practice should be assigned as homework, considering the short time allowance for mastering the numerous commands of the CG and CAD software programs in a classroom setting. More than 90% of students strongly agreed or agreed to the question, "Did you understand the process of digital wax-up with the CG and CAD software programs?" Furthermore, > 80% of students strongly agreed or agreed to the question, "Was digital waxup education meaningful to you?" Based on these answers, we believe this training worked well in satisfying the specific behavioral objectives of the preclinical practice.

CONCLUSION
A newly developed hands-on practice for digital design, digital manufacture of crown restoration using multipurpose CAD and free CG software programs (digital wax-up training) was introduced for undergraduate dental preclinical education. In addition, a set of qualitative evaluation criteria that displayed sufficient inter-rater reliability in estimating the students' overall performance in designing digital waxup work was introduced. A substantially negative but not statistically significant correlation of the total volumetric discrepancy between each student's digital wax-up work and an instructor's digital wax-up reference with the total qualitative score calculated by the set of qualitative evaluation criteria was observed.