Distributed Inference on GPT-2
I’m using this to do distributed inference on the GPT-2 modelmodel= AutoModelForCausalLM.from_pretrained(f"model-name", return_dict=True, device_map="auto", low_cpu_mem_usage=True, torch_dtype=torch.float16)However, I see that not all GPUs are being utilized to their maximum capacity when I runnvidia-smi. Is there any way to speed up the distributed inference? I’m probably missing some crucial things here and would love any thoughts!
2024-05-02T00:25:49Z
jaydeepb:I see that not all GPUs are being utilized to their maximum capacity when I runnvidia-smi. Is there any way to speed up the distributed inference? I’m probably missing some crucial things here and would love any thoughts!GPT2 is quite small, so it may not fully fit across all your GPUs. The output of nvidia-smi would be handy.In general though, you can try maximizing your batch size as much as possible
2024-05-02T01:21:32Z
Image 5-1-24 at 9.18 PM698×389 93.4 KBHere’s the nvidia-smi outputI’m running inference on the XL model with a batch size of 200 (more than that throws OOM). I’m generating 25k samples and it’s taking me around an hour. Which is not bad but I was curious if there are any other ways to decrease the inference time.