Could you please provide a 16-bit version or teach me the steps involved in converting it myself
#1
by
zletpm
- opened
Thank you for the 8bit txtonly version of mistral, the model perform well in mac studio for txt generation task, excced the official or customized 8bit mutimodel version in genaration speed, i wonder if there are 16bit could be much better as the balance of accuracy and speed.
Could you please provide a 16-bit version or teach me the steps involved in converting it myself.
thank you!
Execute the following code step by step in a virtual environment:
pip install mlx-lm==0.22.3
python -m mlx_lm.convert --hf-path anthracite-core/Mistral-Small-3.1-24B-Instruct-2503-HF
Finally, you will get the model with 16-bit MLX version.
zletpm
changed discussion status to
closed
zletpm
changed discussion status to
open