How do I send system prompts using inference api serverless, llama3 8b instruct model
I’m using lama 3 8 b model, using the inference api serverless, like this:import requests

API_URL = "https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3-8B-Instruct"
headers = {"Authorization": "Bearer mytoken"}

def query(payload):
	response = requests.post(API_URL, headers=headers, json=payload)
	return response.json()
	
output = query({
	"inputs": "Can you please let us know more details about your ",
})
print(output)everything works great, but how do I use a system prompt, & do I need to send it with every query or only once, also is it possible to have it remember conversations or do I’ve to send old ones with every query, finally, how do I make it to complete or to chat, like on the huggingface chat ui
2024-04-28T15:38:08Z
Hey I found this post that helped me:Formatting Inference API call for LLama 2Inference Endpoints on the HubI solved it by inputting a single string using the official Llama 2 format (seeLlama 2 is here - get it on Hugging Face). I don’t know why the default Sagemaker Llama endpoint doesn’t work that way. But this works for me: 
import json
import requests
API_URL = "https://api-inference.huggingface.co/models/meta-llama/Llama-2-7b-chat-hf"
headers = {"Authorization": f"Bearer REDACTED_TOKEN",
        "Content-Type": "application/json",}

def query(payload):
    json_body = {
        "inputs":…
2024-04-29T12:27:16Z
Meta Llama 3 | Model Cards and Prompt formatsfor the appropriate tags of Llama 3