Make predictions with the Dropout on
The default behavior ofTrainer(...)when evaluating model is disabling Dropout. Concretely,y_predforMruns will be exactly the samefor i in range(M):
    logits, labels, metrics = trainer.predict(tokenized_datasets["eval"])
    y_pred = np.argmax(logits, axis=2)
    ...Now I am trying to apply Monte Carlo Dropout trick introducedthis this answer. This requires to turn the Dropoutonwhile making predictions on the validation set.I am wondering how I achieve this goal. Any input is appreciated
2021-07-17T16:42:21Z
I don’t think this is possible with the Trainer class as it is, but you can derive this class and then change the relevant methods.In your case, I think you need to change theevaluation_loopmethod and delete the model.eval() line.I think it would be better to keep the model.eval() line and set only the Dropout layer to train mode like it is shown inthis post.There might be some more changes that you need to make.You can find the Trainer source codehere.Hope this helps.