Error when deploying GPT4-Alpaca on Sagemaker via HF model hub
Hello, thanks for reading. I am having many issues deploying LLM models on sagemaker. I have been able to get the canned AWS foundation models deployed, but when I try to use one off of HF hub I always get a similar error. Here is the error I am getting trying to deploy anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g:from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g',
	'HF_TASK':'text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.17.0',
	pytorch_version='1.10.2',
	py_version='py38',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.g4dn.12xlarge' # ec2 instance type
)The endpoint deploys successfully, but when querying the endpoint I get the following error:ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "\u0027llama\u0027"
}The cloud watch logs don’t provide any useful information that I can see. Any thoughts?
2023-05-20T02:55:27Z
Curious to know if you found a fix
2023-05-20T03:12:16Z
not a fix, but an explanation. the current sagemaker instances dont support the transformers version the llama models were trained on, so cant support them in general. i dont think there is a fix, at least that i have been able to find
2023-05-27T11:37:08Z
I have faced the same problems. Does any one know how to fix?
2023-05-31T20:43:35Z
Any fix for this yet?
2023-06-07T10:28:04Z
@philschmidanswered my question which is similar to this issue. His suggested solution is this and it worked for me:can you try to use the new LLM container?Introducing the Hugging Face LLM Inference Container for Amazon SageMaker
2023-06-07T12:17:01Z
This will not work forgpt4-x-alpaca-13b-native-4bit-128gsince it requires the GPTQ package. Therefore you need to create a custominfernece.pyscript and add the latest transforemrs version + gptq with arequirements.txt
2023-06-08T02:55:47Z
this didnt work for me, i got an error in cloud logs like “huggingface_hub.utils._errors.LocalEntryNotFoundError: File aining_args.safetensors of model ehartford/Wizard-Vicuna-13B-Uncensored not found in /tmp. Please runtext-generation-server download-weights ehartford/Wizard-Vicuna-13B-Uncensoredfirst.”i did deploy the model in a hacky way using a pure ec2 instance, fastapi, llamacpp, and nginx
2023-07-11T00:51:45Z
Try a newer version of "transformers_version " compatible with your model:huggingface.coconfig.json · anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g at...We’re on a journey to advance and democratize artificial intelligence through open source and open science.Maybe your model needs more than this, but it’s a useful breadcrumb for folks with similar issues.