How does the pipeline deal with too long sequences?
I’ve been playing around with Text2Text Generation for generative Q&A, using theFLAN-T5. For example:generative_qa_t5 = pipeline(task="text2text-generation", model="google/flan-t5-base")

full_article = ... # Full "Attention is all you need" paper (around 10k tokens)
question = "What's a transformer?"
input_text = f"{full_article} Given this context, please answer the following question. {question}"

print(generative_qa_t5(input_text))
# Outputs: "a model architecture relying entirely on self-attention to compute representations of its input and"If I pass large inputs that are longer than the 512 max sequence length, I get a warning. Surprisingly, the generated answers are still quite good!What happens under the hood within thepipelineto handle this overly long sequence?I tried to split the full text into several parts of max 512 tokens and use them as separate contexts. The resulting answers were much worse than what was generated above. Is there a similar, maybe more sophisticated windowing logic implemented?
2025-01-10T11:39:35Z
It seems that this is the only pre-processing function in the pipeline. By default, it is set to pass the input to the Tokenizer without truncating it.github.comhuggingface/transformers/blob/main/src/transformers/pipelines/text2text_generation.py#L123forward_params["assistant_tokenizer"] = self.assistant_tokenizerreturn preprocess_params, forward_params, postprocess_paramsdef check_inputs(self, input_length: int, min_length: int, max_length: int):"""Checks whether there might be something wrong with given input with regard to the model."""return Truedef _parse_and_tokenize(self, *args, truncation):prefix = self.prefix if self.prefix is not None else ""if isinstance(args[0], list):if self.tokenizer.pad_token_id is None:raise ValueError("Please make sure that the tokenizer has a pad_token_id when using a batch input")args = ([prefix + arg for arg in args[0]],)padding = Trueelif isinstance(args[0], str):args = (prefix + args[0],)padding = False
2025-01-17T08:17:16Z
Do I see it correctly that_parse_and_tokenize()does not implement any special functionality to handle overly long sequences? It just passes the full sequence without truncation (by default) to the tokenizer. I suppose the tokenizer doesn’t care about the sequence length, but how can the model handle overly long tokenized sequence?I still don’t understand how/why passing a too long sequence works without crashing. I get a warning but still a sensible answer.
2025-01-17T09:02:46Z
Looking at the code in the pipeline, it seems like that… However, the pipeline contains everything that is needed, so it is quite difficult to accurately follow the code. There may be something I have missed.It seems like a truncation or similar process is needed somewhere, so the default settings of the tokenizer itself in that model may be excellent and be processing it in a good way…