{'fraction_non_alphanumeric': 0.040665591306562526, 'fraction_numerical': 0.0192715850241956, 'mean_word_length': 4.668912415784408, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Pacemaker: Intermediate Teacher Knowledge Distillation For On-The-Fly Convolutional Neural Network

There is a need for an on-the-fly computational process with very low performance system such as system-on-chip (SoC) and embedded device etc. This paper presents pacemaker knowledge distillation as intermediate ensemble teacher to use convolutional neural network in these systems. For on-the-fly system, we consider student model using 1xN shape on-the-fly filter and teacher model using normal NxN shape filter. We note three points about training student model, caused by applying on-the-fly filter. First, same depth but unavoidable thin model compression. Second, the large capacity gap and parameter size gap due to only the horizontal field must be selected not the vertical receptive. Third, the performance instability and degradation of direct distilling. To solve these problems, we propose intermediate teacher, named pacemaker, for an on-the-fly student. So, student can be trained from pacemaker and original teacher step by step. Experiments prove our proposed method make significant performance (accuracy) improvements: on CIFAR100, 5.39% increased in WRN-40-4 than conventional knowledge distillation which shows even low performance than baseline. And we solve train instability, occurred when conventional knowledge distillation was applied without proposed method, by reducing deviation range by applying proposed method pacemaker knowledge distillation.


INTRODUCTION
Over the past few years, the areas of applying CNN has expanded in the field of computer vision. For better performance in computer vision tasks such as classification [2,3], segmentation [4,5] and detection [6,7], the size of model has been developed larger and deeper. However, this trend causes that it is difficult to use CNN model in low performance systems because the huge amount of memory and computational complexity is needed.
There are various studies to solve this problem such as lightweight [8,9], pruning [10,11] and quantization [12,13] etc. Knowledge distillation (KD) [14,15] is another popular approach in model compressing. The core idea of KD is transfer large teacher model's knowledge to small student model. So, student can be trained to mimic teacher's output such as logits [14,17] using Kullback-Leibler divergence or feature [15,16] using Euclidean distance. So far, KD has been mainly studied to improve the performance of shallow depth student model with conventional CNN using N × N shape filter. Conventional CNN uses N×N filter so, it needs to read image data N line pixels from memory. However, (b) on-the-fly CNN need only 3 pixels to compute the convolution output and this on-the-fly process do not required using memory.
However, we need to use 1×N filter CNN for on-the-fly system, as shown in the Fig. 1., N×N filter conventional CNN can be used for both teacher and student. And student has shallow depth than teacher for conventional KD (CKD). However, on-the-fly CNN student has same depth as teacher and uses 1×N filter for proposed PMKD. So then, student's capacity is reduced about (N−1)/N parameters. For example, if N=3, teacher uses 3×3 square filter and on-the-fly student uses 3 × 3 filter with a reduction of approximately 66 % parameters than teacher. Besides, a student can only consider the horizontal not the vertical receptive field.
And we find that applying CKD directly to on-the-fly student causes bad performance due to the large capacity gap and on-the-fly student's limited ability. It shows a lower or slightly better performance than a baseline. To help the student overcome these limits and reach teacher stably, we propose PMKD, intermediate teacher acting as pacemaker in marathon. With PMKD, on-the-fly student model can be trained step by step to emulate reliably the original teacher.
Our contributions: (1) we studied a novel approach that the on-the-fly CNN, using 1×3 filter, can be used for low performance system requiring on-the-fly processing; (2) unlike the CKD transfer knowledge directly, we propose a novel approach adding an intermediate teacher called PMKD; (3) we solve the training instability occurred when the CKD method was used for on-the-fly student and make significant performance improvement on CIFAR and SVHN datasets. PMKD is made up of the total three times distillation and all feature's W, H, C shape is totally same. The 1st distillation is done by pre-trained original teacher (OT, yellow) and the pacemaker (PM, green) and PM learns from the OT. PM is consisting of ensemble of two models using 1×N row filter and N×1 column filter, respectively. And the 2nd distillation is that PM serves as an intermediate teacher model for training student model (blue). Then, for 3rd distillation, the student model loads PM's pretrained 1×N filter model weights and learns from OT.

BACKGROUND
As briefly mentioned above, KD can be divided into two types, using logits and feature for knowledge. Hinton et al [14] propose to transfer soften class probabilities which is softmax divided with temperature parameter using KL-divergence. To explain the equation, let , refer teacher, student logits and = ( / ) , = ( / ) refer teacher, student soften output: On the other hand, Romero et al [15] propose to transfer feature information using Euclidean distance. The feature dimension size is different from teacher and student because they use a shallower but wider teacher than student. So, they add a regressor to match the dimension size. Let the regressor and , refer teacher and student feature: With these two distillation losses, student is trained by adding distillation losses to cross-entropy loss, let is class label:

PROPOSED METHOD
One important point is that the information distortion is easily found in CKD, because it usually uses shallow depth student model than teacher. However, our on-the-fly student has the same depth as teacher. So, on-the-fly student is relieved from this defect as it does not need to use transformation method like [15] regressor or other method occurring information distortion. Additionally, on-the-fly student can learn from all hidden layer output of teacher as feature knowledge.

Training Loss
Proposed PMKD use both feature and logits from teacher with cross-entropy loss, that is PMKD is mixed version of [14] and modified [15] for on-the-fly CNN with adding pacemaker. As a result, the cost function of PMKD can be written as: Above hyperparameter and are used for balancing losses. We set = 4, = 0.9 which are commonly selected value in [14,16]. And is set various values over a wide range from 0.01 to 5 depending on the dataset and network in our experiments.

Learning Procedures
This section is to describe the detail of three procedures. It is somewhat different each phase of three phase of distillation.

Datasets
We conduct the experiment with classification benchmark dataset, CIFAR and SVHN. CIFAR10 and CIFAR100 [18] contain 32×32 size RGB nature image. And both have 50,000, 10,000 images for train, test set and each has 10, 100 classes. Standard data augmentation [19] is applied for CIFAR. Google's The Street View House Numbers (SVHN) [20] contains 32×32 size center closed-up RGB house digit image. It has 73,257 and 26,032 images for train and test set with 10 classes. And no data augmentation [21] is applied.

Networks
We conduct experiments with various network architectures: VGG [3], ResNet [22], WideResNet [23]. Note that we replace all intermediate pooling to stride process for on-thefly on VGG. Because it is more suitable and efficient for onthe-fly CNN computational process. And we do not need to modify the other networks because these networks do not require intermediate pooling process. In addition, we avoid replacing pooling at first and last layer like [13] strategy for all networks. For VGG network, we select the first and intermediate layer features just before the channel increased and logits as knowledge. And the first and all block features and logits for the series of residual blocks models.

Settings
For all experiment, we run 200 epochs and set the batch size 128 using SGD with momentum 0.9 and weight decay 1e-5. The initial learning rate is 0.1 and dropping 0.2 per [60, 120, 160] epochs. And we set the filter size of N=3 for all model teacher, pacemaker and student. So, teacher used only 3×3 filter, pacemaker use 1×3 and 3×1 filter for ensemble and student use only 1×3 filter for intermediate layers. To sum up, three kinds of filter are used. To match the same spatial size, teacher use zero padding 1 while student use (0, 1) and both have stride 1. And we repeated the same experiment 5 times and report the average value of the results.

Problem Statement
As below Table 1, 2, 3, applying CKD to on-the-fly CNN occurs bad performance or slight improvement on CIFAR and SVHN datasets. It's because of large capacity gap between teacher and student [24]. And an on-the-fly CNN student model, can only consider horizontal field. Due to these weak points, it is very hard to student to learn the knowledge directly from teacher. So, the bad performance and instable train problems can be occurred, and it is hard to solve these problems on student itself.

Performance improvement by proposed PMKD
We can find the problem on below Table 1, 2, 3. For example, as shown in Table 1, VGG16 model on CIFAR10 shows underperformance when directly applied CKD such as KD [14] and FitNet [15] than baseline −2.73% and −1.69%, respectively. And note that if we use both KD and FitNet together, it shows worst performance −5.44%. This result means that due to low capacity, on-the-fly student cannot acquire much information at once. However, when we add proposed PM, it shows improved performance +0.88% and +0.73%, respectively. Note that PMKD, using PM with KD and FitNet, shows best performance improved +1.46%. And we can find similar results on Table 2, 3 too. Table 1: Experiments results of top 1 accuracy on CIFAR10. For easy understanding, we denote ↓ when student underperform than baseline, not applied CKD, ↑ as improved performance, and bold as best performance. These denotations are used equally on

Why should the pacemaker be an ensemble?
Intuitively, using only 1×N model as PM will not help for onthe-fly student, because it is the same as student. As shown in Table 4, when PM is consisting of ensemble models, the onthe-fly student shows better performance than only use 1×N model. So, we can make sure that PM should be ensemble of 1×N & N×1 dual model than only uses N×1 filter model.

CONCLUSION
In this paper, we study to use the on-the-fly CNN as student model which use 1×N filter for low performance systems [1] and focus on improving the student model performance using knowledge distillation. We find that the student can show underperformance or slightly better than baseline, when the conventional knowledge distillation is applied directly to onthe-fly CNN. This problem is due to large capacity gap [24] between N×N teacher and on-the-fly student which has the limit that it can only consider the horizontal receptive field. So, to solve the problem and make the on-the-fly student improved, we present pacemaker knowledge distillation, intermediate temporary teacher. And we find out that on-thefly student shows better performance when pacemaker is ensemble of 1×N & N×1 dual model rather than only N×1 model. We believe that by applying proposed method pacemaker to knowledge distillation, low capacity on-the-fly student model shows significantly improvement that directly use knowledge distillation to on-the-fly student model.

ACKNOWLEDGEMENT
This project is partially supported by Samsung Electronics CO., LTD and partially supported by the Technology