FLUX.1-dev image generation on M1 Mac using GPU
I want to make use of GPU on my M1 Mac to generate images through prompt usingblack-forest-labs/FLUX.1-dev. Adding the code below.I’ve specifiedmpsfortorch.Generatorbut the script fails stating with errorlatents = torch.randn(shape, generator=generator, device=rand_device, dtype=dtype, layout=layout).to(device) RuntimeError: Placeholder storage has not been allocated on MPS device!How can I make use of GPU to generate image?prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=1,
    max_sequence_length=512,
    generator=torch.Generator("mps").manual_seed(0)
).images[0]
image.save("flux-dev.png")
2024-10-18T14:30:38Z
How about this?import torch
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
prompt = "A cat holding a sign that says hello world"
pipe.to(device)
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=1,
    max_sequence_length=512,
    generator=torch.Generator(device).manual_seed(0)
).images[0]
image.save("flux-dev.png")
2024-10-18T15:31:08Z
Although now it picks GPU but still fails adding error below. I’ve addedpipe.enable_model_cpu_offload()but still fails. Also I’ve executedPYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0still failsRuntimeError: MPS backend out of memory (MPS allocated: 36.21 GB, other allocations: 384.00 KB, max allowed: 36.27 GB). Tried to allocate 144.00 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
2024-10-18T15:35:22Z
How about this?pip install -U accelerateimport torch
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16, device_map="auto") # reweite it
pipe.enable_model_cpu_offload()
prompt = "A cat holding a sign that says hello world"
#pipe.to(device) # it causes OOM
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=1,
    max_sequence_length=512,
    generator=torch.Generator(device).manual_seed(0)
).images[0]
image.save("flux-dev.png")
2024-10-18T15:40:55Z
there is syntax error in this code. pipe is referenced before assignment
2024-10-18T15:44:11Z
Sorry, I failed to copy and paste.