MuhibBeekun/pickai-qwen2.5-vl-3b-layout-vision-lora

PickAI Layout Vision LoRA

Status: value gate passed · PickAI default for layout-from-image

QLoRA adapter for PickAI, an open-source WMS-adjacent pick-path optimization service. Given a warehouse floor-plan image, the adapter extracts labeled bay locations as structured JSON (facility_draft.locations) for facility onboarding — a domain-tuned vision capability, not prompt-only BYOK.

Base model: Qwen/Qwen2.5-VL-3B-Instruct

---

Evaluation (30-sample holdout)

Metric	Base Qwen2.5-VL-3B	This LoRA
Aggregate score	59.15%	93.24%
Location recall	54.72%	100.00%
Location coordinate match	0.00%	72.96%
JSON valid	100.00%	100.00%

Value gate: PASS (LoRA aggregate exceeds base on held-out synthetic floor plans).

Full eval log: fine-tune-eval-vision.md

---

What it does

flowchart LR
  IMG[Warehouse floor plan PNG] --> VLM[Qwen2.5-VL-3B + LoRA]
  VLM --> JSON[facility_draft.locations JSON]
  JSON --> Profile[PickAI FacilityProfile]
  Profile --> Opt[Pick-path optimizer]

Each location object:

Field	Type	Example
location_id	string	A1-L01 (exact label on image)
x, y	number	Bay coordinates (meters)
aisle	string | null	A1
level	string | null	2

Training mode: locations-only (shorter JSON targets; zones/docks inferred separately or merged later).

---

Training

Setting	Value
Base model	Qwen/Qwen2.5-VL-3B-Instruct
Method	QLoRA (4-bit NF4) + PEFT LoRA
Rank (r)	32
LoRA alpha	64
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Train rows	210 synthetic floor plans
Holdout	30 (hash split, excluded from training)
Max steps	600 (best checkpoint @ step 568, loss 0.0662)
Learning rate	2e-4
Image max side	448 px
Max sequence	4096 tokens
GPU	NVIDIA GeForce RTX 3090 (24 GB)

Synthetic data: generated locally via scripts/generate_layout_vision_jsonl.py (240 labeled floor-plan renders).

Training script: scripts/train_lora_layout_vision.py

---

Usage

PickAI runtime (recommended)

PICKAI_USE_VISION_LORA=1
PICKAI_VISION_TRAIN_MODE=locations_only
PICKAI_VISION_LORA_DIR=outputs/lora-vision

Docker Compose mounts this adapter read-only for the Facility Console layout upload flow.

Standalone (Transformers + PEFT)

import json
import torch
from peft import PeftModel
from transformers import AutoProcessor, BitsAndBytesConfig, Qwen2_5_VLForConditionalGeneration
from PIL import Image

base = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter = "MuhibBeekun/pickai-qwen2.5-vl-3b-layout-vision-lora"

processor = AutoProcessor.from_pretrained(adapter, trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base,
    trust_remote_code=True,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16,
    ),
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

image = Image.open("floor_plan.png").convert("RGB")
prompt = (
    'Return JSON only: {"facility_draft":{"locations":[...]}} extracted from the warehouse floor plan image.\n'
    "Copy location_id exactly as printed on bay labels.\n"
)

messages = [{"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)

decoded = processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(decoded)

Set PICKAI_VISION_TRAIN_MODE=locations_only when using PickAI inference helpers.

---

Limitations

• Trained on synthetic top-down floor plans (2–3 aisles, 3–4 bays per side). Real CAD/PDF layouts may need fine-tuning or human review.
• Locations-only output: aisles, docks, zones, and layout dimensions may need separate extraction or manual entry.
• Requires GPU with ~8 GB+ VRAM for 4-bit inference; training used 24 GB.
• Coordinate accuracy (~73% within tolerance on holdout) is good but not perfect — validate before production slotting.

---

Links

Resource	URL
PickAI repository	https://github.com/Muhib-Beekun/pickai
Eval report	https://github.com/Muhib-Beekun/pickai/blob/main/docs/fine-tune-eval-vision.md
NL-parse dataset (separate task)	https://huggingface.co/datasets/MuhibBeekun/pickai-synthetic-nl-parse-v1

---

Citation

If you use this adapter in research or a product demo, please link to the PickAI repo and this model card.