Using Langchain ChatHuggingface with Text Generation Inference: missing field `inputs`
I deployed Mistral 7B Instruct v0.3 with TGI using the official docker container:model=mistralai/Mistral-7B-Instruct-v0.3
# share a volume with the Docker container to avoid downloading weights every run
volume=$PWD/data
REDACTED_TOKEN=<your_REDACTED_TOKEN_token>

docker run --gpus all --shm-size 1g -e HF_TOKEN=$REDACTED_TOKEN -p 8080:80 -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:2.2.0 --model-id $model --quantize bitsandbytes-nf4I then created a HuggingFaceEndpoint and was able to interact with the Mistral model:llm = HuggingFaceEndpoint(
    endpoint_url="http://localhost:8080/",
    max_new_tokens=512
)
llm.invoke("What is the capital of France?")Working with theInferenceClientfromhuggingface_hubor directly viacurlorrequestsworks fine.Then I created a ChatHuggingFace, so that I don’t have to manually add the special tokens, etc.:llm_engine_hf = ChatHuggingFace(llm=llm, model_id="mistralai/Mistral-7B-Instruct-v0.3")   # model_id can't be inferred from the endpoint_url
llm_engine_hf.invoke("Hugging Face is")This fails because the json dict in the request does not have aninputsfield, but amessagesfield.The error description in the response is:'Failed to deserialize the JSON body into the target type: missing field `inputs` at line 1 column 400'Btw. the actual error message that is output in the console is not that helpfulhuggingface_hub.utils._errors.HfHubHTTPError: 422 Client Error: Unprocessable Entity for url: http://localhost:8080/What am I missing? Is there some parameter that I have to change when instantiating theHuggingFaceEndpoint/ChatHuggingFaceor when deploying the model via TGI? I couldn’t find anything in the official guides.
2024-08-08T14:36:02Z
Okay, the solution was pretty trivial.I changed the endpoint_url to “http://localhost:8080/v1/chat/completions”.It now works withmessages.I guess using the official Inference API from Huggingface chooses the correct url for you, but when you self-host you have to manually specify the url like that in order to use the Messages API. Otherwise it uses the “/generate” endpoint, which requires aninputsfield.
2024-08-09T02:36:02Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.