Longformer and sentiment analysis
I am trying to use longformer to do a sentiment analysis and I am wondering what the best way is to do it. I have the following code:from transformers import LongformerTokenizer, EncoderDecoderModelmodel = EncoderDecoderModel.from_pretrained(“patrickvonplaten/longformer2roberta-cnn_dailymail-fp16”)tokenizer = LongformerTokenizer.from_pretrained(“allenai/longformer-base-4096”)
2021-08-23T18:58:46Z
Hi,LongFormer itself is a Transformer encoder, and that’s more than sufficient to perform sentiment analysis. You can just useLongFormerForSequenceClassification, like so:from transformers import LongformerTokenizer, LongformerForSequenceClassification
import torch

tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
model = LongformerForSequenceClassification.from_pretrained('allenai/longformer-base-4096')

inputs = tokenizer("This text is positive", return_tensors="pt")
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(**inputs, labels=labels)
loss = outputs.loss
logits = outputs.logitsNote that this model will have a classification head that is randomly initialized, so you’ll need to fine-tune it on a custom dataset.
2021-08-23T19:08:05Z
This is great! I think I am getting it but I am just wondering how you get the label and socre then. I normally use pipeline and sentiment-analysis and I can get something like [{‘label’: ‘NEGATIVE’, ‘score’: 0.9982099533081055}].
2021-08-24T07:00:06Z
So you want to do inference with an already fine-tuned model?Looking on thehub, there are currently no LongFormer checkpoints fine-tuned on a sentiment analysis dataset. So feel free to upload the first LongFormer checkpoint fine-tuned on a sentiment analysis dataset to the hubOnce you’ve trained a model, you can plug it into the pipeline API for quick inference.Here’s an example with a RoBERTa model from the hub, fine-tuned on sentiment analysis:from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

model_name = "cardiffnlp/twitter-roberta-base-sentiment"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

nlp = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)

text = "this is a very positive text"

outputs = nlp(text)In this case, printing the outputs returns[{'label': 'LABEL_2', 'score': 0.9623518586158752}]. This is of course not that helpful, as we don’t know what LABEL_2 means. This is because the authors of this model have not updated theconfig.id2label. However, looking atthis file(mentioned in the code example on themodel page), it appears that LABEL_2 means “positive”.
2021-08-24T11:32:23Z
Thanks@nielsr! I think I’d need to train one and if I get to finish that, I’ll share it out. I guess the other way to tackle the longform text is to focus on the maximum limit of the text the pre-trained model can take. Do you happen to know what the limit is - like how much text I can push into the pipeline for the sentiment?
2021-08-24T19:00:48Z
Models like BERT, RoBERTa, etc. all take a max sequence length of 512 tokens. Note that these models usesubwordtokenization, which means that a given word might be tokenized into several tokens, so in practice these models can take in less than 500 words. So if you really want to use the pipeline API with a very long text, you can use models likeLongFormerorBigBird, which can handle 4096 tokens in a single forward pass.As an alternative (if you still want to use a model like BERT/RoBERTa), you can implement it using a sliding window approach. You can do this using the tokenizer, like so:from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "a very long text"
encoding = tokenizer(text, return_overflowing_tokens=True, stride=128)This will create multiple training examples (encodings) for a given text, by sliding a window (each having 128 tokens of overlap) across the text. You can then feed each training example through the model, and average the predictions.
2023-08-30T23:49:48Z
Hi nielsr,Sorry to bother you on this!  If I’m looking to utilize RoBERTa for long sequence sentiment analysis and I want to use tokenizer stride during training, do I need to write an entirely custom train() logic to average the predictions during the batch evaluation?  Or is there something already built-in to trainer.train() that I could leverage to take advantage of strides?