Question about query_length in modeling_t5.py
github.comhuggingface/transformers/blob/0a057201a96565df29984d716f660fd8d634329a/src/transformers/models/t5/modeling_t5.py#L464# Mask is (batch_size, key_length) (non-causal) or (batch_size, key_length, key_length)# past_key_value[0] is (batch_size, n_heads, q_len - 1, dim_per_head)batch_size, seq_length = hidden_states.shape[:2]real_seq_length = seq_lengthif past_key_value is not None:assert (len(past_key_value) == 2), f"past_key_value should have 2 past states: keys and values. Got { len(past_key_value)} past states"real_seq_length += past_key_value[0].shape[2] if query_length is None else query_lengthkey_length = real_seq_length if key_value_states is None else key_value_states.shape[1]def shape(states):"""projection"""return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2)def unshape(states):"""reshape"""return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim)If thepast_key_valueis not None, thequery_lengthinduced fromSelfAttentioncan be the length ofpast_key_value+hidden_states, which is actually wrong.Luckily, this problematic behavior would not affect theEncDecAttetionsince theposition_biasare always zeros.Correct me if I am wrong.