How to use gated model in inference
Hi, I have obtained access to Meta llama3 models, and I am trying to use it for inference using the sample code from model card. When I run my inference script, it gives me an error 'Cannot access gated repo for urlhuggingface..../meta-llama.....config.jsonSo my question is how can I access this model from my inference script? Do I need to pass any authientication/api/token key to ensure my script can access the model?Also how to get huggingface version of the model locally?
2024-09-26T23:27:38Z
You could do it by making the code look like this.huggingface.cometa-llama/Llama-3.1-8B-Instruct · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Sample codeimport transformers
import torch

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])Actual Codeimport transformers
import torch

REDACTED_TOKEN = "hf_*********" # When uploading code, never write directly!

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
    token=REDACTED_TOKEN,
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])You can make as many tokens as you want here. You should name your tokens in such a way that you can easily identify which is which.huggingface.coHugging Face – The AI community building the future.We’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-09-27T06:36:20Z
Make sure to authenticate withhuggingface-cli loginin the terminal before running the script.
2024-09-30T21:42:57Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.