Multi GPU Build Possible?
I’m a little worried that I’m spending a considerable amount with no ability to actually build something that can handle a 32B LLM. I get mixed messages. Either they reassure me it’ll work with a bunch of janky PCIe mining risers or they call me an idiot… I’d just like to know how to get started. Help? I’m a noob. I know, but we all have to start some where. Be gentle…Intel® Core™ i9 14900KF processor2xKingston FURY Beast 64GB DDR5 RAM 5600MT/s CL36 (svart)x4MSI GeForce RTX 4060 Ti Ventus 2X Black OC 16GB grafikkortMSI PRO Z790-A MAX WIFI ATX LGA1700 MotherboardCorsair HX1500i ATX 3.0 1500WNoctua NH-L9x65 chromax. blackKingston Fury Renegade med Kylfläns 2TB
2025-01-19T23:21:36Z
MSI GeForce RTX 4060 Ti Ventus 2X Black OC 16GB grafikkortIt’s much more powerful than my GPU, but even in 4-bit quantization, the 32B model is about20GBin size, and it also consumes a little extra VRAM at runtime, so I think it will exceed the 16GB VRAM and spill over into RAM. Well, it should work because there is enough RAM, but it’s not clear whether it will be usable comfortably.If you have a model with around 16B, you should be able to use it comfortably with 10GB in 4-bit quantization. If you use 16-bit precision without quantization, even an 8B model will run out of VRAM…In this case, you could either sacrifice precision and use a 32B model with 3-bit or 2-bit quantization, use a more powerful GPU, compromise with a smaller LLM, or put up with a little slower speed.Oh, if you have a multi-GPU (16GBx2), you should be fine as long as you use 4-bit quantization. According to reports on forums, etc., it seems that the load may not be evenly distributed in some cases, but there are often no particular problems with executing the model itself.
2025-01-19T23:33:46Z
I understand you want to inference a 32B?You can do offloading. Here are the numbers.FP32:32 billion parameters * 4 bytes/parameter = 128 billion bytes128 billion bytes / (1024 * 1024 * 1024) = 120 GBFP16:32 billion parameters * 2 bytes/parameter = 64 billion bytes64 billion bytes / (1024 * 1024 * 1024) = 60 GBNote:This calculation only considers the memory required to store the model parameters themselves.In reality, you’ll also need memory for:Activations during inferenceOptimizer states (if training)Intermediate calculationsSystem overheadThis means the actual memory requirements will be significantly higher.