Does llama-2 need pro subscription?
I get the following error when trying to use meta-llama/Llama-2-7b-hf model. I didn’t find any pointers through web search, so asking here. Can someone please help?llm = HuggingFaceHub(repo_id = “meta-llama/Llama-2-7b-hf”,huggingfacehub_api_token=my_token, model_kwargs={“temperature”:0.5, “max_length”:512})Generates the following error. If I swap out the repo_id to “google/flan-t5-base”, the code runs fine.raise ValueError(f"Error raised by inference API: {response[‘error’]}")ValueError: Error raised by inference API: Model requires a Pro subscriptionThank you for your help!
2023-07-28T21:54:23Z
yes to use it with inference api, you need pro subscription since its too large(13gbish>10gb which is free api limit). Ofcourse you could run it locally without any error.
2023-07-28T22:15:36Z
Thank you,@YaTharThShaRma999! Are there quantized versions I can use through inference API? I see models by ‘TheBloke’ which are smaller than 10GB, but it appears that inference API is turned off for these.Thank you so much for your help!
2023-07-28T23:36:48Z
You cant use those since those are actually for different libraries and things. Like gptq version is for exllama and autogptq while ggml version are for llama cpp.If you really want to use a llama model and dont have some sort of gpu, try out llama cpp python.It uses like 4gb ram max for 7b model(avg phones have like 6gb ram).If you do have gpu, use autogptq or something which works with transformers as well.
2023-08-02T11:17:44Z
@YaTharThShaRma999hi sir can you please provide some resources for the mentionedi wanna implement llama cpp python but using ctransformer library does not provide some functions likeimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoTokenizermodel_name = “TheBloke/Llama-2-7B-GGML”bnb_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type=“nf4”,bnb_4bit_compute_dtype=torch.float16,)model = AutoModelForCausalLM.from_pretrained(model_name,quantization_config=bnb_config,trust_remote_code=True)model.config.use_cache = Falselike this code i am not able to use in ggmlso i am not able to understand what model should we pick for finetuning ???is ggml not for finetuning?if not then whats the uses of ggml ?can you please answer or give some links for these as i am not able to get
2023-08-02T21:05:02Z
Ggml is for inference but it is kinda possible to train a new model from scratch.Also, in your code for some reason, you try to load a ggml model(already 4 bit quantized) in 4 bit again?That is not possible. If you need a bit more information check out ctransformers docs? (Ctransformers is also just for inference and doesn’t have all the things like transformers)Ggml models are used because they use extremely low ram up and very fast inference in cpu.
2023-11-24T20:25:22Z
hi did you got any solution, found any other model or found any way to use API for free?