Accelerate on single GPU doesnt seem to work
Im new to the huggingface community and to ML and starting playing around withaccelerateand followed the instruction set out in the tutorials.The issue i seem to be having is that i have used the accelerate config and set my machine to use my GPU, but after looking at the resource monitor my GPU usage is only at 7% i dont think my training is using my GPU at all, i have a 3090TI.22960×1455 234 KBhere is the command that i used to start my trainingaccelerate launch train_unconditional.py–train_data_dir=“data”–resolution=256–center_crop–random_flip–output_dir=“output-256”–train_batch_size=2–save_model_epochs=5–num_epochs=100–gradient_accumulation_steps=1–use_ema–learning_rate=1e-4–lr_warmup_steps=500–mixed_precision=noIs there maybe a library that i am missing or some other config setting i need to change, because in the accelerate documentation i can only find info about multi gpu training.
2023-05-15T09:06:40Z
I expect my GPU usage to increase, thats how i assume that my training is using my GPU, i might be wrong which is why i am reaching out.Capture1182×1264 44.2 KB
2023-05-16T00:54:38Z
After many hours of trying to figure this out i found the solution to this.I uninstalled my version of Python (I was using version 3.11.3)I also uninstalled my version of Nvidia Cuda (I was using version 12.1).My new ApproachI went toPytorch Websiteand found the latest CUDA version to use (11.8)Installed Python Version 3.10.9installed Cuda Version 11.8 from Nvidia website.Created my new python environment and ran the following pip commands.pip install torch torchvision torchaudio --index-urlhttps://download.pytorch.org/whl/cu118pip install git+https://github.com/huggingface/diffuserspip install accelerate datasets tensorboardand the went through myaccelerate configand when i started training my GPU was being used.The important part was to ensure that the CUDA version from Nvidia matched the CUDA version from the PyTorch website, and that Nvidia CUDA was installed BEFORE installing CUDA through PIP.