File size: 1,681 Bytes
ff68c55 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
I used the bert-base-uncased to evaluate the model's power towards the Model distillation. The model( bert-base-uncased ) performed exceptionally in terms of performance(Accuracy, speed and size) while compressing the model.
## Observations from the experiment:
# Base Model -F32: 110M params having Accuracy - {'Accuracy': 0.893}
# Distilled Model(Reduced to half): 52.8M params having Accuracy - {'Accuracy': 0.924}
# Quantized Model(Reduced to 2.8 times): 38.8M params having Accuracy - {'Accuracy': 0.922}
Details are below:
I trained two classifier models, one for the teacher(size and accuracy same as base) and the other for the student(model size reduced to half and accuracy increased than base model by 3%).
# Teacher (validation) -Accuracy: 0.8933, Precision: 0.9078, Recall: 0.8756, F1 Score: 0.8914
# Student (validation) -Accuracy: 0.9244, Precision: 0.9704, Recall: 0.8756, F1 Score: 0.9206
Further, I quantized the model to 4 bits and got the magical performance:
# Pre-quantization Perf -Accuracy: 0.9244, Precision: 0.9704, Recall: 0.8756, F1 Score: 0.9206
# Post-quantization Perf -Accuracy: 0.9222, Precision: 0.9703, Recall: 0.8711, F1 Score: 0.9180
Conclusion: Model parameters were reduced to 2.8 times while achieving an accuracy increase of 3% over the original base model.
GPU: Tesla T4
Dataset: shubh2ds/data-phishing-site-clf
Consumption: 
Final Model: https://huggingface.co/shubh2ds/bert-base-uncased-phishing-classifier_student_4bit
https://wandb.ai/shubh2ds/huggingface/runs/yt0pcw58/workspace?nw=nwusershubh2ds
|