ALiBi - Train Short, Test Long: Attention with linear biases enables input length extrapolation
alibi #transformers #attention Transformers are essentially set models that need additional inputs to make sense of sequence data ...