Deploy big model to AWS Sagemaker fails
Hello,I’m trying to deploy a model on AWS Sagemake (mosaicml/mpt-7b-chat · Hugging Face)I used the code generated by the model’s page for a quick start, it is as follow:from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'mosaicml/mpt-7b-chat',
    'HF_TASK':'text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    transformers_version='4.17.0',
    pytorch_version='1.10.2',
    py_version='py38',
    env=hub,
    role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
    initial_instance_count=1, # number of instances
    instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
    'inputs': "Can you please let us know more details about your "
})The code crashes at the “predict” call with this error: Loading /.sagemaker/mms/models/mosaicml__mpt-7b-chat requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the optiontrust_remote_code\u003dTrueto remove this error.I had the same issue locally that I fixed by adding the "  trust_remote_code=True" parameter to the AutoModelForCausalLM.from_pretrained constructor, but that doesn’t seem to exist with HuggingFaceModel constructor.Any way to fix this? thank you
2023-05-13T22:57:57Z
I had the same issue! Were you able to fix this?
2023-05-19T02:46:27Z
I am facing the same issue. Any input on how to fix it?
2023-05-28T09:14:08Z
Anyone with any solution for this? Seems like it is still not resolved
2023-06-05T15:27:51Z
this medium article allowed me to deploy successfullyDeploy MosaicML MPT-7B-Instruct on SageMaker | by Manoranjan Rajguru | May, 2023 | Medium
2023-07-31T23:17:43Z
Was anyone able to fix this issue? Looks like this issue exists for all mosaicml/mpt-7b-* models. Is that right?