Text Generation response truncation
I am having an issue where all responses are truncated, no matter which model or where they are hosted. Other questions on this topic have referenced required changes in langchain modules, but we are not using langchain.For reference, I am getting the same error from:databricks/dolly-v2-3b run on databricks ML RT 13.3 (56 ram 8cpu GCP)databricks/dolly-v2-3b run on HF free inference using huggingface_hub librarydatabricks/dolly-v2-7b huggingface inference endpoint Nvidia A10G using recommended sample inputtiiuae/falcon-7b huggingface inference endpoint Nvidia A10G using recommended sample inputIn case of HF inference endpoints: I get the same error using python requests as I do with the sample ui. Setting for inference endpoint are default: max input tokens: 1024 max tokens:1025. Screenshot below:truncation1248×657 41.6 KB
2023-09-01T15:04:58Z
This problem exists in other modes to. Here is a test on inference endpoint, queried from Databricks(GCP) using requests after setting the inference endpoint to text-to-text:Screenshot 2023-09-01 at 11.01.24 AM1215×539 39.6 KB
2023-09-01T15:05:30Z
and the same but after setting inference endpoint to summarization mode:Screenshot 2023-09-01 at 11.03.34 AM1215×539 42.3 KB
2023-11-07T07:03:15Z
Hey,@cCaldwell! Have you been able to solve this issue? I have started to have the same problem while using the Llama 2 70b model.
2023-11-14T04:25:51Z
same issue with 7b
2024-06-15T17:35:38Z
I’m also facing same issueon all the models I’ve tried.Does anyone have the solution for this??
2024-08-18T19:21:12Z
I am using chat completion with Microsofts SemanticKernel and get the same result. Usually the responses are truncated at 100 tokens.I have two questions, can I raise the bar above 100 tokens and is there a way to programmatically detect that the response are truncated?