Llama 2 70B on a cpu
I was just using this model here on HuggingFace. I noticed that it referenced a cpu, which I didn’t expect at all. Here’s the link:huggingface.coExplore Llamav2 With TGI - a Hugging Face Space by ysharmaDiscover amazing ML apps made by the communityBeside the title it says:“Running on cpu. Upgrade.”Is this just theendpointrunning on a CPU?
2023-07-28T16:19:49Z
No its running with inference endpoints which is probably running with several powerful gpus(a100). Running a 70b model on cpu would be extremely slow and take over 100 gb ram.I don’t know why its running on cpu upgrade however.
2023-08-23T01:37:37Z
I hava test use llama.cpp infer Llama2 7B、13B 70B on different CPUThe fast 70B INT8  speed as3.77 token /s( AMD 9654P 96C/768G memory)run command:build/bin/main -m /home/apps/models/wizardlm-70b-q8_0.bin -gqa8-eps 1e-5-t96-n1024–repeat_penalty1.0–color -c512–temp0.6-p“Please introduce me something about vipshop holdings ltd.”token speed:llama_print_timings:        load time =1576.96msllama_print_timings:      sample time =10.12ms /396runs   (0.03ms per token,39114.97tokens per second)llama_print_timings: prompt eval time =1500.66ms /15tokens (100.04ms per token,10.00tokens per second)llama_print_timings:        eval time =104635.98ms /395runs   (264.90ms per token,**3.77tokens per second**)It’s make sense:)