Example for Fine Tuning CLIP or BLIP2 for VQA
Hi,I wanted to fine tune CLIP and BLIP2 for a VQA task on custom dataset, but I was unsure how to do it. Are there any examples for fine tuning CLIP and BLIP2 for VQA?Thank you
2023-12-05T06:24:58Z
I have implemented a repohere. I hope this can help.
2024-02-16T12:46:53Z
This uses BLIP rather than BLIP2, no? Any pointers on BLIP-2? The architecture is slightly different
2024-04-11T16:54:03Z
Hello@swtb, did you find anything for finetuning vqa or blip2 or have implemented anything? Any help will be really appreciated.
2024-04-12T07:22:10Z
Yes though I moved on from it. You can find alot of clues from the inputs of the model and outputs of its forward pass.I’ll be at my pc later, will attach a code snippet from my training loop
2024-04-12T11:31:05Z
Tutorials for fine-tuning BLIP-2 are linked here:Transformers-Tutorials/BLIP-2 at master · NielsRogge/Transformers-Tutorials · GitHub. These include notebooks for both full fine-tuning (updating all parameters) as well as PEFT (parameter efficient fine-tuning using LoRa).
2024-05-24T11:46:36Z
Hi, have you implemented the use of lora to fine tune blip2 on top of vqa tasks?
2024-05-30T07:52:23Z
I’d recommend checking out therepolinked above, and then you just need to wrap the BLIP-2 model using PEFT:from transformers import Blip2ForConditionalGeneration
from peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training

model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", load_in_8bit=True)

lora_config = LoraConfig(
            r=8,
            lora_alpha=8,
            lora_dropout=0.1,
            target_modules="...",
            init_lora_weights="gaussian",
)

model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()Then proceed as usual. Thetarget_modulesare model-specific.
2024-10-02T14:57:13Z
Can anyone please share the code for Finetuning  BLIP-2 for VQA?? (PEFT)
2024-10-14T08:51:56Z
@NirmalML: were you able to find a suitable code?
2024-10-14T12:54:52Z
Hello@sashika, no not yet. I have a code but I’m not sure if it is correct
2024-10-14T13:01:41Z
I need it urgently, so it would be really kind and helpful if anyone helps me in this
2024-10-14T13:05:42Z
@not-lainHe seems to be in a hurry. Can you think of anyone who might know this code?code for Finetuning BLIP-2 for VQA?? (PEFT)
2024-10-15T06:18:45Z
Could you please share your code?
2024-10-15T06:47:49Z
I ended up using BLIP not BLIP2, so:my complete script for BLIP is here:github.comkreimanlab/HumorDB/blob/main/benchmarks/binary_classification/vqa/fine_tune.pyimport numpy as np


# Critical imports
import os
import numpy as np
import pandas as pd
from PIL import Image
import random
import torch
import torchvision
import torchvision.models as models
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
import torch.optim as optim
from torch.optim import lr_scheduler
import gc
import matplotlib.pyplot as plt
from typing import Dict, List, Optional, Set, Tuple, Union
from transformers import AutoImageProcessor, BlipForQuestionAnswering, BlipProcessorThis file has been truncated.show originalIf looking for a basic snippet,encoding = self.processor(images=image, text=question, padding="max_length", truncation=True, return_tensors="pt")
labels = self.processor(text=answer, return_tensors="pt").input_ids
encoding["labels"] = labels
batch = encoding
batch = {k: v.cuda() for k, v in batch.items()}
outputs = model(**batch)
loss = outputs.lossif you are interested in the loss used by the model by default, it is on line 899 ontransformers/src/transformers/models/blip/modeling_blip_text.py at main · huggingface/transformers · GitHubputting the small snippet here:if labels is not None:
            # we are doing next-token prediction; shift prediction scores and input ids by one
            shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
            labels = labels[:, 1:].contiguous().to(shifted_prediction_scores.device)
            loss_fct = CrossEntropyLoss(reduction=reduction, label_smoothing=self.label_smoothing)
            lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
            if reduction == "none":
                lm_loss = lm_loss.view(prediction_scores.size(0), -1).sum(1)
2024-10-15T06:50:57Z
The above basically calculates over all predictions so every token after the first one not just the answerbut empirically that is fine and it helps if one wants to ask specific kinds of questions as the model is better tuned for the question too.