title: Huggingface GPT2 and T5 model APIs for sentence classification?
body: <p>I've successfully used the <a href="https://huggingface.co/transformers/model_doc/bert.html" rel="nofollow noreferrer">Huggingface Transformers BERT model</a> to do sentence classification  using the <a href="https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification" rel="nofollow noreferrer">BERTForSequenceClassification</a> class and API. I've used it for both 1-sentence sentiment analysis and 2-sentence NLI.</p>
  <p>I can see that other models have analogous classes, e.g. <a href="https://huggingface.co/transformers/model_doc/xlnet.html#xlnetforsequenceclassification" rel="nofollow noreferrer">XLNetForSequenceClassification</a> and <a href="https://huggingface.co/transformers/model_doc/roberta.html#robertaforsequenceclassification" rel="nofollow noreferrer">RobertaForSequenceClassification</a>. This type of sentence classification usually involves placing a classifier layer on top of a dense vector representing the entirety of the sentence.</p>
  <p>Now I'm trying to use the <a href="https://huggingface.co/transformers/model_doc/gpt2.html" rel="nofollow noreferrer">GPT2</a> and <a href="https://huggingface.co/transformers/model_doc/t5.html" rel="nofollow noreferrer">T5</a> models. However, when I look at the available classes and API for each one, there is no equivalent &quot;ForSequenceClassification&quot; class. For example, for GPT2 there are <a href="https://huggingface.co/transformers/model_doc/gpt2.html#gpt2model" rel="nofollow noreferrer">GPT2Model</a>, <a href="https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel" rel="nofollow noreferrer">GPT2LMHeadModel</a>, and <a href="https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel" rel="nofollow noreferrer">GPT2DoubleHeadsModel</a> classes. Perhaps I'm not familiar enough with the research for GPT2 and T5, but I'm certain that both models are capable of sentence classification.</p>
  <p>So my questions are:</p>
  <ol>
  <li><p>What Huggingface classes for GPT2 and T5 should I use for 1-sentence classification?</p>
  </li>
  <li><p>What classes should I use for 2-sentence (sentence pair) classification (like natural language inference)?</p>
  </li>
  </ol>
  <p>Thank you for any help.</p>