Why is Trainer only using 1 (not 4) GPUs?
The Transformers Trainer is only using 1 out of 4 possible GPUs. Why is that?Hi, I’ve setCUDA_VISIBLE_DEVICES=0,1,2,3andtorch.cuda.device_count()shows 4.  But when I run my Trainer, nvtop shows that only GPU 0 is computing anything.  I would expect all 4 GPU usage bars in the following screenshot to be all the way up, but devices 1-3 show 0% usage:Screen Shot 2022-06-02 at 2.54.58 PM1320×472 219 KBI even tried manually settingtrainer.args._n_gpu = 4(instead of leaving it alone) but it had no effect.Someone will ask to see the full code for what I’m doing, which is understandable. It is a clone of this Kaggle notebook:Music Genre Classification with Wav2Vec2 | KagglePS- In searching for answers, I notice people askinghow to limit the number of GPUs to 1but I"m trying to get it to do the opposite – use all GPUs!  If the latter is supposed to be the default… ?Do the HuggingFace models not automatically invoke DistributedDataParallel?  If that omission is the issue, then…that would be understandable.  I don’t see any usage of “parallel” or “accelerate” anywhere in the code.  So perhaps I need to add that manually?
2022-06-02T21:08:27Z
Tried taking a look ataccelerateexample notebook for SimpleNLP, but it crashes with SIGSEV for me:Simple NLP Example not working🤗AccelerateI too am seeing the SIGSEV error when I run the Simple NLP example on an A100 pod: 
---------------------------------------------------------------------------
ProcessExitedException                    Traceback (most recent call last)
Input In [25], in <cell line: 3>()
      1 from accelerate import notebook_launcher
----> 3 notebook_launcher(training_function, num_processes=4)

File ~/envs/wav2vec/lib/python3.9/site-packages/accelerate/launchers.py:129, in notebook_launcher(function, args, num…