Help Needed: Installing Llama 2 70B, Llama 3 70B & LLaMA 2 30B (FP16) on Windows Locally
Hi everyone,I’m trying to install Llama 2 70B, Llama 3 70B, and LLaMA 2 30B (FP16) on my Windows gaming rig locally that has dual RTX 4090 GPUs. I aim to access and run these models from the terminal offline. I’ve hit a few roadblocks and could really use some help.Here are the specifics of my setup:Windows 10Dual MSI RTX 4090 Suprim Liquid X 24GB GPUsIntel Core i9 14900K 14th Gen Desktop Processor64GB DDR5 RAM2x Samsung 990 Pro 2TB Gen4 NVMe SSDHas anyone successfully installed and run these models in a similar setup? If so, could you provide detailed steps or point me to relevant resources? Any tips on optimizing the installation for dual GPUs would be greatly appreciated as well.Thanks in advance for your assistance!
2024-06-03T13:09:00Z
Hi,First I’d create a Python virtual environment in which you install PyTorch, and verify the following works:import torch

print(torch.cuda.is_available())
print(torch.cuda.device_count())This should printTrueand indicate that you have 2 GPU devices available. This requires CUDA to be installed on your computer (CUDA is the software which runs the hardware).Next, you can install the Transformers library as explainedhereand perform inference with LLaMa models as shown in thedocs.