Issue with batching long sequences
So I had this question came to my mind when watching a LLM training tutorial: if the max sequence length of your training data is set to smaller than the sequence length of some of the sequences in your training set, then it means those sequences need to be broken into smaller pieces. My question is if this happens for some sequence of length n at index where m < n, say m = n/2, and m+1 < n, is the p(m+1|m) not going to be trained? But by right p(m+1|m) needs to be optimized right?