MNLI Inference on a fine-tuned model from hub
Hello,I tried to run various models such ashuggingface/distilbert-base-uncased-finetuned-mnlimicrosoft/deberta-v2-xxlarge-mnliroberta-large-mnlisqueezebert/squeezebert-mnliI can see the weights are loaded but the accuracy I get is about 7%.I use  “transformers_version”: “4.3.2”, and the following arguments:python run_glue.py   --model_name_or_path roberta-large-mnli  --task_name mnli    --do_eval   --max_seq_length 128    --output_dir /tmp/mnli/Any help would be appreciated,Thank you
2021-02-24T19:50:34Z
Hi Ali93H,I don’t understand what you are trying to do.   Are you wanting to do further fine-tuning (in which case you might want DistilBertForMaskedLM) or to classify your texts for example sentiment analysis (in which case you might want DistilBertForSequenceClassification).The transformers docs are hereQuick tour — transformers 4.3.0 documentation,Summary of the models — transformers 4.3.0 documentation,DistilBERT — transformers 4.3.0 documentation
2021-02-24T20:04:15Z
Thanks for your reply. I’m trying to do an evaluation on an already fine-tuned model (reproduce the accuracy metrics).If I try this with mrpc or stsb  (with fine-tuned models, of course) it works just fine, but I don’t know what is the issue with MNLI.
2021-02-25T15:57:57Z
Here is the source of the issue and a solution:[run_glue] Add MNLI compatible mode by JetRunner · Pull Request #10203 · huggingface/transformers · GitHub