Dynamically change max_new_tokens in a pipeline
Is there anyway to change the max_new_tokens used in a pipeline without having to load the whole pipeline again?I am using gemma27b-it and just playing around with the model, starting with a base prompt and then stringing the model’s answers with new prompts and so on. The only thing is that right now the max_new_tokens variable is static, and I want to control it every time I call the pipeline object such that for a question like ‘Paris is the capital of France: T or F?’ I want to set max_new_tokens = 1 and for more elaborate questions, to raise this limit. Is this possible?
2024-07-07T08:03:59Z
Specify max_new_tokens when you call the pipeline.Ex:Q = “Paris is the capital of France: T or F?”T_or_F = pipeline(Q, max_new_tokens=1)Elaborated = pipeline(f“Elaborate the response to ‘{Q}’: {T_or_F}.”, max_new_tokens=1024)