How to get log probs if we already have a generation?
Hi! I’m wondering if there’s any way I could get log probs for tokens in a generated sequence if I already have a generation. I was going throughthispost about getting log probs withgenerate().However, if I already have a sequence that was generated by the model using a specific decoding method (greedy, top-k) and I want to get log probs for each token in that generated sequence, is that possible?
2024-05-09T10:40:54Z
You could try just feeding your sequence into the model all at once with a single forward call. If you need to simulate the effects of top-k, etc you could then run the logits processors on the logits you get back from the forward call