Inference 8 bit or 4 bit bit models on cpu?
hello, is it possible to run inference of quantized 8 bit or 4 bit models on cpu?
2023-08-02T21:06:41Z
I don’t believe so since the bitsandbytes library is just a wrapper around cuda functions which are for gpu.
2023-08-03T08:14:03Z
for those still searching, I found some sources aboutoptimum intel for quantization on intel cpus🤗 Optimum Intelcore ml has quantization tools for apple cpusCompressing Neural Network Weights