Unable to Structured output
Hi, did anyone manage to use structured output with oss 20b? especially with ollama, if not ollama, with any other setup?
Hi, yea using vllm with xgrammar, but its really weird
It often generates completely random text or more random json after the (most of the time correct) actual json
without robust parsing its almost unusable.
I did use lm format enforcer before but it got removed from vllm in this newest version so this might also be a xgrammar / outlines issue.
I think the <|constrain|>json is important before the message, tho it still doesnt stop the garbage after the json / guarantee success
Same issue as @ztsvvstz . I'm able to get the generally expected output, but then it just keeps going. I'm experimenting with adding split words before and after the output template and parsing with Python, and having moderate success.
same issue here. Has anyone solved this problem? texts after call out message is so random.
Alright I have some information, no fix yet but some explanation atleast for VLLM:
vllm doesnt support structured output for gpt oss yet (mentioned in their docs) and therefore the output is always "unguided"
Now, before V1 you could use lm format enforcer which needs to setup a logits processor on VLLM to do the same job, but
with V1 they removed logits processor and didnt finish an alternative yet.
So basically, vllm is "deadlocked" in you need V1 to run gpt oss, V1 doesnt support neither structured out nor logit processors.
So ya'll best bet (atleast of the time being) is to manually parse the llm output
For anyone still having trouble:
https://github.com/vllm-project/vllm/issues/23120
Patching this function to always return true fixed it for me