Why are model_q4.onnx and model_q4f16.onnx not 4 times smaller than model.onnx?
I see onhttps://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/tree/main/onnx:File NameSizemodel.onnx654 MBmodel_fp16.onnx327 MBmodel_q4.onnx200 MBmodel_q4f16.onnx134 MBI understand that:model.onnxis the fp32 model,model_fp16.onnxis the model whose weights are quantized tofp16I don’t understand the size ofmodel_q4.onnxandmodel_q4f16.onnxWhy ismodel_q4.onnx200 MB instead of 654 MB / 4 = 163.5 MB? I thoughtmodel_q4.onnxmeant that the weights are quantized to 4 bits.Why ismodel_q4f16.onnx134 MB instead of 654 MB / 4 = 163.5 MB? I thoughtmodel_q4f16.onnxmeant that the weights are quantized to 4 bits and activations are fp16, sinceConfigure Quantization — mlc-llm 0.1.0 documentationstates:qAfB(_id), whereArepresents the number of bits for storing weights andBrepresents the number of bits for storing activations.andWhy do activations need more bits (16bit) than weights (8bit) in tensor flow’s neural network quantization framework?indicates that activations don’t count toward the model size (understandably).