{'fraction_non_alphanumeric': 0.029722720925593457, 'fraction_numerical': 0.009774586076201875, 'mean_word_length': 5.121286121286121, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
StuArt: Individualized Classroom Observation of Students with Automatic Behavior Recognition and Tracking

Each student matters, but it is hardly for instructors to observe all the students during the courses and provide helps to the needed ones immediately. In this paper, we present StuArt, a novel automatic system designed for the individualized classroom observation, which empowers instructors to concern the learning status of each student. StuArt can recognize five representative student behaviors (hand-raising, standing, sleeping, yawning, and smiling) that are highly related to the engagement and track their variation trends during the course. To protect the privacy of students, all the variation trends are indexed by the seat numbers without any personal identification information. Furthermore, StuArt adopts various user-friendly visualization designs to help instructors quickly understand the individual and whole learning status. Experimental results on real classroom videos have demonstrated the superiority and robustness of the embedded algorithms. We expect our system promoting the development of large-scale individualized guidance of students.


INTRODUCTION
Classroom observation serves as a means of assessing the effectiveness of teaching, which has been continuously studied [1,2,3]. The utility of traditional classroom observation is limited due to the manual coding that is expensive and unscalable. Thus, automatic classroom observation systems built on the advanced technologies are quite essential.
In this paper, we present an automatic individualized classroom observation system using one RGB camera, named StuArt. First, to reflect the learning status of students, five typical behaviors that occur during the classroom teaching and their variation trends are presented, including handraising, standing, smiling, yawning and sleeping. To improve the behavior recognition performance, we build a large-scale student behavior dataset. Second, to observe students continuously, we propose a multiple student tracking method in the classroom. We utilize locations of students (row and column of seats) as their unique identifications to protect personal privacy. Moreover, we design several intuitive visualizations to display the evolution process of the individualized and holistic learning status of students.
As far as we know, StuArt is the first comprehensive system of automatic individualized classroom observation. All techniques embedded in it are designed for classroom scenarios, and systematically evaluated effects in both controlled and real courses. The robust performances prove the usability and generalization value of our system.

SYSTEM DESIGN OF STUART
Our proposed individualized classroom observation system, StuArt, is shown in Figure 1, which comprises three main steps. First, the data of students' learning process is recorded by one high-definition RGB camera. Then, the video is parsed to generate educationally-relevant features, and the variation of learning status for each student. Finally, we visualize the individual and holistic situations of students from a bird-view according to their seat positions.
I. Equipment Overview: Equipment of StuArt has three aspects: video data capturing by the RGB camera, models training and system deployment in running device. We use a single camera to capture videos for two motivations. Firstly, CNN-based algorithms can detect objects quite accurately in real classrooms [11,5,12,8]. Secondly, camera deployment is portable, especially in the we-media era. A camera with 1K (1920×1080) resolution costing about $300 meets our requirements. For large-scale data processing and training, we equipped storage arrays and multiple GPUs. In inference, StuArt only needs one 3080Ti GPU. Considering computational cost, we also implemented an integrated system on the more lightweight edge device Jetson TX2 8GB module costing about $500.
II. Real Instructional Videos: We obtained the access rights of generous real instructional K-12 course videos, which are collected and managed by the institute of education in Shanghai city. Generally, we divide these courses into three categories including primary, junior and senior, shown by sub-images in Figure 2 B, C and D. In Figure 2 A, we also built our own simulated teaching scene. As can be seen, students in classrooms are visually crowded, which brings low resolution and severe occlusion challenges in rear rows. Thus, the performance of deep vision algorithms trained on public datasets is often degraded. Domain behavior datasets are essential for adapting to the classroom scene.
Specifically, we selected about 300 videos from 20+ schools as the raw data. Each course video has an average duration of 40 minutes with 25fps. Firstly, we use OpenCV [14] and FFMPEG [15] to do preprocessing. Then, we set an interval at 3 seconds to evenly sample frames. We select  five behaviors to annotate: hand-raising, standing, sleeping, yawning and smiling. To prevent the wrong recognition of standing with teachers walking into the student group, we add one more category namely teacher. During annotation, we choose the tool LabelMe [16] and store the location (x, y, w, h) and category of behaviors in an XML file as Pascal VOC [17]. In order to ensure the labeling accuracy and efficiency, we finished annotation by our team's researchers familiar with the goal following the active-learning strategy.
Some annotation examples are in Figure 2 bottom.
III. Behavior Recognition: We treat the behavior recognition task as the object detection problem using YOLOv5 [18], which keeps a better trade-off between accuracy and computation than its counterparts [19,20]. Figure 3 shows detection examples. Due to the size gap among yawning and smiling with other three behaviors, we failed to incorporate them into the multi-behavior detector. For yawning, we allocated a single detector and proposed a mouth fitting strategy to distinguish it with talking. For smiling, we adopted face detection MTCNN [21] and emotion classification CovPoolFER [22] to detect it. By simply reducing YOLOv5 from large to small, we obtain light-weight student behavior detectors for running in edge devices. IV. Student Tracking: This part, we present how to reasonably count the times of behaviors, match behaviors with individual students, and then track the behavior sequence of the whole class according to the seat of each student.
i. Interframe Deduplication: The input of behavior detector is sampled frame without temporal information. However, a real behavior may last across consecutive frames. Thus, we need to suppress duplications. We adopt bounding box matching to filter out repeated behaviors with maximum IoU among interframes. In practice, we set our overlap threshold to 0.2 for robust IoU matching. If a behavior box is not matched in subsequent T frames, we assume it happened once. We here set T = 2 in case of occasional missed behavior detection.
ii. Behavior Matching: How do we know who are performing these behaviors? We need to complete the matching of behaviors. Except for hand-raising, the other four behaviors have naturally identified the corresponding students with their bounding boxes (refer Figure 2). The hand-raising box only contains partial area of the raising arm, which is likely to be confused with the wrong individual student's body when matching, especially the simultaneous occurrence of dense hand-raisings among crowds [23,24]. To tackle it, we adopted OpenPifPaf [25] to generate student body keypoints as auxiliary information for hand-raising matching. We plotted skeletons for illustrating in Figure 4 top. However, due to inherent challenges of hand-raising gestures, OpenPifPaf may still fail to detect arm joints. Thus, we designed a heuristic strategy combining greedy pose searching and weight allocation priority to maximize the matching accuracy.
iii. Seat Location: After finishing behavior matching, we still keep unknown about who or where they are. We elegantly chose to use the row and column number of students seats to uniquely represent them. Concretely, we divide the seat location into five steps shown in Figure 4 bottom: (A) Based on body keypoints, we generate representative points by averaging coordinates of upper body joints. (B) With these points, we filter out the teacher point if detected. (C) Then, if using a wide-angle camera, we need to correct the barrel distortion of these points. (D) After that, to make 2D points under camera view with affine deformation in distinct order, we rearrange them from top-view by applying projective transformation [14]. (E) Finally, the row and column can be obtained by applying K-means clustering after projecting points on the y-axis and x-axis, respectively. In Figure 4, all students' located seats are plotted. We can automatically recognize that two students sitting in R2C7 and R4C9 are raising his or her hand. A smiling student is sitting in R4C9.
iv. Behavior Tracking: By now, we can automatically recognize student behaviors, suppress repeated behaviors, match behaviors with students, and locate student seats with given total rows and columns. Based on these results, we could obtain individual behavior tracklets of every student. Their seating positions indexed by RxCy are regarded as tracklet IDs. In this way, if a classroom equipped with i rows and j columns seats, we would totally generate i×j ×5 tracklets for tracking five specific behaviors.
V. Individualized Observation Visualization: To assist teachers quickly observe each student's status and know the classroom situation, we designed visualizations from both micro (individual student) and macro (overall students) levels.
i. Non-interactive Designs: As shown in Figure 5 topleft (Seating Table), using our simulated video as an example, we draw and update the number of five behaviors in all table ceils representing student seats as time goes on. The seats occupied by students have gray background. Behavior categories are distinguished by different font colors. If a behavior is detected, the corresponding table ceil will be highlighted with background color related to its category. Furthermore, as in Figure 5 bottom-left (Activate Heatmap), by assuming that the counts of positive (hand-raising, standing and smiling) and negative (yawning and sleeping) behaviors roughly reflects engagement, we can get the dynamic tendency of students' engagement by converting behavior counts into the range [0, 1] for rendering a heatmap with diverse cold and warm colors ranging between blue and red. These presented details are synchronized with class time, and the detection and tracking results will be refreshed automatically every three seconds. No intervention required makes this way suitable for being used when in class.
ii. Interactive Designs: After class, the non-interactive designs may be unsatisfactory for reviewing a student in detail. Inspired by EmotionCues [7], we designed configurable interactive Link List and Point Flow based on the recorded behavior sequence of each student, as shown in Figure 5 right. In each link list, we arrange student behavior counts into a list of horizontally stacked bars on the left space. Teachers can re-sort them according to the target behavior type. These bars are clickable and linked with detailed personal behavior sequences. Instead of observing an isolated student, we can also present the integrated information of the classroom by depicting the whole behavior trend at each sampling moment forming the point flow. This design is a supplement to the previous link list interaction.

Data for Training and Testing
We have obtained a total of nearly 36k images which include more than 110k instances. Specifically, for the training of multi-objects model including hand-raising, standing, sleeping and teacher, we have gotten about 70k, 21k, 3k and 15k samples respectively. The number of images and samples used for training single yawning detection model are 3,162 and 3,216. For smiling, we got a total of 129k positive and 181k negative facial images. Additionally, to test the accuracy of the two vital parts (behavior matching and seat location) in student tracking, we selected six and four courses respectively. Among them, the six courses used to evaluate matching include primary, junior and senior schools, as well as left and right observation perspectives. The four videos used to test the seat location effect include distorted and undistorted videos, as well as left and right views.

Effectiveness Evaluation
Behavior Recognition: We adopt the metric mAP (mean Average Precision) which is wildly used in public object detection datasets Pascal VOC [17] and COCO [26] to evaluate our behavior recognition methods. On the behavior validation set, our multi-objects model which contains the detection of hand-raising, standing, sleeping and teacher achieved 57.6 mAP(%). The model used to detect yawning alone reaches 90.0 mAP(%). For smiling recognition, we got an average of 84.4% accuracy for binary classification.
Behavior Matching: We only report the evaluation of the most difficult hand-raising behavior matching on six selected Seat Location: Table 2 shows the quantitative measurement details of the seat location on four selected courses. We randomly chose five seats from RxCy in each class, and manually observed the location result of every student. The higher the values of x and y, the farther away from the camera the students are, and the lower their resolution is. Then, we manually counted out F l legal frames in which student's location is distinguishable, and F c frames where students are correctly located. The location accuracy Acc s of a single student in one course can be roughly calculated by F c /F l . Finally, the overall student location accuracy Acc a is about 83.3%. Although the location accuracy of some students (e.g., R4C3 in video 2, R5C5 in video 3) is limited, we argue that most of them are located well enough to support rough student tracking and individualized observation.

CONCLUSION
In the traditional classroom for K-12 education, we proposed StuArt to realize individualized classroom observation by automatically recognizing and tracking student behaviors using one front-view camera. Firstly, we selected five typical classroom behaviors as entry perspectives. Then, we built a dedicated behavior dataset to accurately detect target behaviors. For behavior matching and seat location, we chose the classroom with a traditional seating arrangement, and used the row and column of the seat as a desensitized identification to realize the automatic tracking of student behavior. Based on these observation results, we have designed suitable visualizations to assist teachers quickly obtain student behavior patterns and the overall class state. In experiments, we conducted quantitative evaluations of the proposed core methods. The results confirmed the effectiveness of StuArt.