What are the recommended command line flags to use with oobabooga/text-generation-webui?
#2
by
wezfaas
- opened
eg:
--load-in-4bit
--use_double_quant
wezfaas
changed discussion title from
Recommended command line flags to use with oobabooga/text-generation-webui?
to What are the recommended command line flags to use with oobabooga/text-generation-webui?
eg:
--load-in-4bit
--use_double_quant
Don't use any command-line args. Just set them inside the webui itself. Pick exllamav2 or exllamav2_hf as the model loader. You'll need more than 12 GB VRAM for this model.