Pipeline vram problem with google colab
HelloI am using the llama 3 8b model in google colab.I am trying to run it in a vram 40g environment.I can upload the model with quantumization and LoRa, butthe moment I load the pipeline, the vram is exceeded and it becomes impossible to run.To solve this problem, should I write the code myself instead of loading the pipeline?If you have any code examples, I would really appreciate it if you could post them.0(This is a text-generation pipeline)If you have any other methods, please let me know.
2025-01-16T02:25:07Z
Normally, the majority of VRAM is consumed when loading models, but some VRAM is also consumed during inference. Also, although the reason is not well understood, there are cases where an abnormal amount of VRAM is consumed during pipeline execution. There are several code examples for when the pipeline is not used, but I will post the official HF sample for Llama.huggingface.coLLaMAWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2025-01-21T14:22:30Z
thank you sir