Issue with structured output generation

#13
by fpaupier - opened

With the suggested config for vLLM, I face the issue when trying to generate structured output as json with guided decoding. Same client calling code worked for the small 3.1

ERROR 06-24 18:59:41 [backend_xgrammar.py:160] Failed to advance FSM for request cmpl-a6ccae43c35d4c58894e10ad1be2a273-0 for tokens 0. Please file an issue.

Any idea what could be the cause ?

Mistral AI_ org

Hi could you paste your entire code snippet please with the complete trace ?

Hi @juliendenize
Thanks for your feedback. Unfortunately I'm struggling to replicate this myself and I'm now facing a different issue / maybe somehow linked issue with applying the chat template.
cf vlLM log stack trace

An error occurred in `mistral_common` while applying chat template
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293] Traceback (most recent call last):
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 1275, in apply_mistral_chat_template
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]     return tokenizer.apply_chat_template(
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]   File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizers/mistral.py", line 388, in apply_chat_template
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]     request = make_mistral_chat_completion_request(messages, tools)
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]   File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizers/mistral.py", line 181, in make_mistral_chat_completion_request
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]     return ChatCompletionRequest(messages=messages,
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]   File "/usr/local/lib/python3.12/dist-packages/pydantic/main.py", line 253, in __init__
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293] pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatCompletionRequest
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293] tools.0.function.strict
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]   Extra inputs are not permitted [type=extra_forbidden, input_value=True, input_type=bool]
vllm-1  | ERROR 06-25 08:41:34 [chat_utils.py:1293]     For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden

On the application side calling the mistral3.2 we have

status_code: 400,  body: {'object': 'error', 'message': '1 validation error for ChatCompletionRequest\ntools.0.function.strict\n  Extra inputs are not permitted [type=extra_forbidden, input_value=True, input_type=bool]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden 1 validation error for ChatCompletionRequest\ntools.0.function.strict\n  Extra inputs are not permitted [type=extra_forbidden, input_value=True, input_type=bool]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden', 'type': 'BadRequestError', 'param': None, 'code': 400}

I'll come back to you if I find a clear n easy reproducible step for this issue

Sign up or log in to comment