Starcoder: CUDA out of memory
I am trying to runbigcode/starcodermodel on Amazon EC2. I have 8 tesla T4 GPUs with 16GB RAM, but somehow I still encounter the“Cuda out of memory”error.What possible solutions can this issue have? Thank you.Code Snapshot:from transformers import AutoModelForCausalLM, AutoTokenizerfrom accelerate import Acceleratoraccelerator = Accelerator()checkpoint = “bigcode/starcoder”device = accelerator.device # for GPU usage or “cpu” for CPU usageprint(‘Reached device Selection.’)tokenizer = AutoTokenizer.from_pretrained(checkpoint)print(‘Tokens generated.’)model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)accelerator.prepare(model)inputs = tokenizer.encode(“def print_hello_world():”, return_tensors=“pt”).to(device)outputs = model.generate(inputs)print(tokenizer.decode(outputs[0]))Error:Screenshot 2023-11-27 1541251922×603 55.7 KB
2024-02-09T06:02:09Z
same issue here
2024-02-09T07:41:53Z
Hi!I searched the internet and it seems thataccelerator.devicealways usescuda:0, so the model is not distributed to the GPUs.Maybethishelps you.
2024-02-22T09:28:51Z
is it a limit for most of the LLM model which can’t scale into multiple GPUs?
2024-02-23T16:44:01Z
shrey-shah:model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)AutoModelForCausalLM.from_pretrained has device_map option, set it todevice_map="auto"it will split model layers into different available devices.Correct:model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto").to(device)