Dreambooth finetuning does yield expected result
Hi,This is my first attempt at training a dreambooth checkpoint using the following script:github.comhuggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py#!/usr/bin/env python
# coding=utf-8
# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and

import argparse
import copy
import gc
import logging
import mathThis file has been truncated.show originalI followed the instructions at the following link to train on the dog dataset:github.comdiffusers/examples/dreambooth at main · huggingface/diffusers🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX. - huggingface/diffusersMy base SD model is: “runwayml/stable-diffusion-v1-5”I saved the checkpoint to my local folder and used it to generate T2I. But I dont see the dog that was used in the finetuning show up in the inferred result. My training config is as follows. All the other params are as in the default training file above.max_train_steps: tried between 400 to 1000enabled train_text_encoderlearning_rate: tried, 5e-4, 5e-6, 1e-6.I would appreciate any help on how I can get the model to generate the images from the fine-tuning dataset.My inference script is as follows:model_base = “runwayml/stable-diffusion-v1-5”pipe = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16)pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)db_ckpt = “lora_db_dogs”pipe.load_lora_weights(db_ckpt)pipe.to(“cuda”)prompt = “A photo of sks dog in a bucket”image = pipe(prompt, num_inference_steps=50, guidance_scale=7.5).images[0]image.save(“results/dog-bucket.png”)