Evaluation on Quantized model yields identical results across bit-precisions
I have been trying to benchmark some models across different bit precisions. I useoptimum quantoandBitsAndBytesto achieve this.When I usetrainer.evaluate()however, the output is nearly identical between the quantized models and Float models (even for 2-bit integer precision), which seems unlikely to me. Fine-tuning the model yields different results. I have a hunch that thetrainer.evaluate()function does not use the quantized layers or something similar to this, as this is also the case for different models and different PTQ methods.To quantize, I useoptimum.quanto.quantize(model)andoptimum.quanto.freeze(model).Does anyone have an idea how this happens?As a sidenote, to usetrainer.evaluate()on quantized models, I created a subclass that inherits from theTrainerclass to circumvent errors related to training.
2024-06-20T18:45:07Z
Figure with results for reference:mrpc_eval_accuracy1400×800 53.4 KB