Error while training LORA in KOHYA_SS (stabilityai/stable-diffusion-xl-base-1.0)
HelloI have provided 1 image for training. It’s a simple JPG.Sample1250×200 11.3 KBI am getting the error below:Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\sdxl_train_network.py", line 185, in <module>
    trainer.train(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\train_network.py", line 272, in train
    train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 2325, in cache_latents
    dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 1146, in cache_latents
    cache_batch_latents(vae, cache_to_disk, batch, subset.flip_aug, subset.alpha_mask, subset.random_crop)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 2775, in cache_batch_latents
    raise RuntimeError(f"NaN detected in latents: {info.absolute_path}")
RuntimeError: NaN detected in latents: D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\trained-model\img\40_A Event of a Pharma Client A Event of a Pharma Client\Sample1.jpg
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module>
    sys.exit(main())
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
    args.func(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', 'D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model/config_lora-20250130-144011.toml']' returned non-zero exit status 1.
14:43:08-514022 INFO     Training has ended.The config file is as below:bucket_no_upscale = true
bucket_reso_steps = 64
cache_latents = true
caption_extension = ".txt"
clip_skip = 1
dynamo_backend = "no"
enable_bucket = true
epoch = 1
gradient_accumulation_steps = 1
huber_c = 0.1
huber_schedule = "snr"
learning_rate = 0.0001
logging_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\log"
loss_type = "l2"
lr_scheduler = "cosine"
lr_scheduler_args = []
lr_scheduler_num_cycles = 1
lr_scheduler_power = 1
lr_warmup_steps = 160
max_bucket_reso = 2048
max_data_loader_n_workers = 0
max_grad_norm = 1
max_timestep = 1000
max_token_length = 75
max_train_steps = 1600
min_bucket_reso = 256
mixed_precision = "fp16"
multires_noise_discount = 0.3
network_alpha = 1
network_args = []
network_dim = 8
network_module = "networks.lora"
noise_offset_type = "Original"
optimizer_args = []
optimizer_type = "AdamW8bit"
output_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model"
output_name = "last"
pretrained_model_name_or_path = "stabilityai/stable-diffusion-xl-base-1.0"
prior_loss_weight = 1
resolution = "512,512"
sample_prompts = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model\\prompt.txt"
sample_sampler = "euler_a"
save_every_n_epochs = 1
save_model_as = "safetensors"
save_precision = "fp16"
text_encoder_lr = 0.0001
train_batch_size = 1
train_data_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\img"
unet_lr = 0.0001
xformers = true
2025-01-30T09:40:38Z
Maybe this is it. There is a famous bug in the VAE of stabilityai/stable-diffusion-xl-base-1.0, and it seems that the VAE does not work properly with fp16 precision…github.com/bmaltais/kohya_ssSDXL Training Error - RuntimeError: NaN detected in latentsopened04:28AM - 29 Jul 24 UTCPENGUINADELIEI am trying to create an SDXL LoRA using Runpod. My dataset consists of 25 image…s of women, each with a size of 1024x1024 pixels. I keep encountering error logs indicating issues with the images. All images are 1024x1024 pixels, and I've tried using both PNG and JPG formats, but the issue persists. Does anyone know how to fix this?

**[Error log]**
  File "/workspace/kohya_ss/sd-scripts/sdxl_train_network.py", line 185, in <module>
    trainer.train(args)
  File "/workspace/kohya_ss/sd-scripts/train_network.py", line 272, in train
    train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process)
  File "/workspace/kohya_ss/sd-scripts/library/train_util.py", line 2324, in cache_latents
    dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process, file_suffix)
  File "/workspace/kohya_ss/sd-scripts/library/train_util.py", line 1146, in cache_latents
    cache_batch_latents(vae, cache_to_disk, batch, subset.flip_aug, subset.alpha_mask, subset.random_crop)
  File "/workspace/kohya_ss/sd-scripts/library/train_util.py", line 2772, in cache_batch_latents
    raise RuntimeError(f"NaN detected in latents: {info.absolute_path}")
RuntimeError: NaN detected in latents: /workspace/data/img_xyzminji/40_xyzminji woman/xyzminji_(1).jpg
Traceback (most recent call last):
  File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', '/workspace/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', '/workspace/data/model_lora/config_lora-20240729-035252.toml']' returned non-zero exit status 1.
03:53:26-826393 INFO     Training has ended.