BLOOM for sentence classification
I was wondering whether it is sensible to use BLOOM for sentence classification tasks (ie using thelast_hidden_statesin combination with a classification layer).It seems to work for my test case. However, I’m curious whether this is only by chance (and I’m interpreting my results wrong) because BLOOM is specified for text generation and not for sentence classification in huggingface.
2022-09-02T08:15:28Z
Usually, one uses models with bidirectional attention (like BERT, RoBERTa) for text classification tasks (to benefit from context on both sides). Models like GPT-2 or BLOOM use a causal attention mask, making sure only context from the left is taken into account.They are more suited for text generation, rather than text classification/NER etc. (however you can just ask the model to generate a class, or named entities, for instance).