LoRa fine tuning a chatbot on 6GB VRAM GPU
Hello, I am doing a chatbot working with data from IMDB.I was thinking of using meta-llama/Llama-3.2-1B-Instruct and LoRa fine tuning it on BrightData/IMDb-Media Dataset. I have a 6GB VRAM GPU and not much time.My questions are: Is this the right model to use for the task, or should I use a smaller one? Is it good paired with the dataset? Will 6GB VRAM be enough for LoRa fine tuning it? Also, should I split the dataset and LoRa tune it on a smaller scale? The dataset has 250K rows.
2025-01-21T09:30:37Z
I’m not very familiar with it, but training requires a lot of VRAM, so even if you use LoRA with fp16 precision at 1B, it might be just about possible with 6GB of VRAM.Well, I think it won’t crash if you have enough RAM…If you can use QLoRA, you can save a lot of VRAM, but I’ve never used it.github.comGitHub - OValery16/Tutorial-about-LLM-Finetuning-using-QLORA: Fine-tuning a Language Model (LLM) with QLORA on a...Fine-tuning a Language Model (LLM) with QLORA on a Single GPU