RedHatAI
/

Meta-Llama-3.1-70B-Instruct-quantized.w4a16

Text Generation

text-generation-inference

4-bit precision

Model card Files Files and versions

alexmarques commited on Oct 10, 2024

Commit

172ead4

·

verified ·

1 Parent(s): 2e832eb

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -435,7 +435,7 @@ lm_eval \
 ```
 lm_eval \
   --model vllm \
-  --model_args pretrained="neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16",dtype=auto,max_model_len=4096,tensor_parallel_size=1",enable_chunked_prefill=True \
   --apply_chat_template \
   --fewshot_as_multiturn \
   --tasks leaderboard \

 ```
 lm_eval \
   --model vllm \
+  --model_args pretrained="neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16",dtype=auto,max_model_len=4096,tensor_parallel_size=1,enable_chunked_prefill=True \
   --apply_chat_template \
   --fewshot_as_multiturn \
   --tasks leaderboard \