Controlling bos, eos, etc in api-inference
Is there a way to control the beginning of sentence, end of sentence tokens through the inference api? I could not find it in the documentation.
2021-05-04T15:18:19Z
Hi@cristivlad,Currently there is no way to override those within the API.We are adding andend_sequenceparameter to enable stopping the generation when using prompt-like generation (for GPT-Neo for instance).For BOS, what did you want to do with it ?Cheers,Nicolas
2022-01-12T20:33:13Z
I triedend_sequenceand indeed works, I would suggest updating thedoc, thanks in any case!data = json.dumps(
        {
            "inputs": "1+1=2\n2+2=",
            "parameters": {
                "max_length": 50,
                "num_return_sequences": 1,
                "return_text": False,
                "return_full_text": False,
                "do_sample": True,
                "top_k": 50,
                "top_p": 0.95,
                "end_sequence": "\n"
            },
            "options": {
                "wait_for_model": True,
            },
        }
    )