When training Llama for sequence classification, should the final token be an EOS?
Thedoc stringsays:LlamaForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do.Should this “last token” be an EOS or simply the final token in the input without an EOS? My interpretation is that it shouldnotbe an EOS, because otherwise, it would probably say that explicitly. Plus many people use the EOS as the pad token, in which case it would be identical tonotusing the EOS as the “last token” for sequence classification.However, I’m not certain so I’d appreciate it if anyone knew!
2023-09-18T15:42:26Z
@ArthurZwould you happen to know the answer to this? Thanks!
2023-09-18T21:16:30Z
This is a standard module, Llama was not trained forLlamaForSequenceClassificationso you should use whatever you want no?