Out of memory error
Hi with my trainer I get an error and I am not able to fix it. Log as follows, just when 100% is reached:torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 11.39 GiB already allocated; 253.88 MiB free; 13.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFProgress:|                         |:   0% 1/1600 [00:08<3:59:07,  8.97s/it, loss=0.686, lr=1e-6]Traceback (most recent call last):File “/usr/local/bin/accelerate”, line 8, insys.exit(main())File “/usr/local/lib/python3.8/dist-packages/accelerate/commands/accelerate_cli.py”, line 43, in mainargs.func(args)File “/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py”, line 837, in launch_commandsimple_launcher(args)File “/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py”, line 354, in simple_launcherraise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)The problem gets when Generating class images:  100%How could I fix it? Thanks in advance