Running stable diffusion models
I am runnning the Custom dataset, i.e., toy example using 15 photos of my cat Miles (took ~40 minutes on Tesla V100): example fromGitHub - harrywang/finetune-sd: what I learned about fine-tuning stable diffusionbut getting errors can someone help meTraceback (most recent call last):File “/home/padamatinti.s/.conda/envs/StableDif/bin/accelerate”, line 8, insys.exit(main())File “/home/padamatinti.s/.conda/envs/StableDif/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py”, line 45, in mainargs.func(args)File “/home/padamatinti.s/.conda/envs/StableDif/lib/python3.9/site-packages/accelerate/commands/launch.py”, line 1097, in launch_commandsimple_launcher(args)File “/home/padamatinti.s/.conda/envs/StableDif/lib/python3.9/site-packages/accelerate/commands/launch.py”, line 552, in simple_launcherraise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)subprocess.CalledProcessError: Command ‘[’/home/padamatinti.s/.conda/envs/StableDif/bin/python3.9’, ‘train_text_to_image_lora.py’, ‘–pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5’, ‘–train_data_dir=./data/full-finetune/cat’, ‘–dataloader_num_workers=1’, ‘–resolution=512’, ‘–center_crop’, ‘–random_flip’, ‘–train_batch_size=1’, ‘–gradient_accumulation_steps=4’, ‘–max_train_steps=1500’, ‘–learning_rate=1e-04’, ‘–max_grad_norm=1’, ‘–lr_scheduler=cosine’, ‘–lr_warmup_steps=0’, ‘–output_dir=./models/lora/miles’, ‘–checkpointing_steps=500’, ‘–validation_prompt=A photo of a cat in a bucket’, ‘–validation_epochs=10’, ‘–seed=42’, ‘–report_to=wandb’]’ died with <Signals.SIGKILL: 9>.
2023-12-23T00:18:45Z
I get the same error when checkpointing