Sample evaluation script on custom dataset
Hey, I  have a custom dataset. can you send a sample script to get the accuracy on such a dataset? I was going through examples and I couldn’t get a code that does that. Can someone send me a resource?my dataset is of the format-premise , hypothesis, label(0 or 1)and my model is debertaThanks@lewtun
2021-12-09T09:10:43Z
side note: i am using a zero-shot version of deberta-NDugar/ZSD-microsoft-v2xxlmnli · Hugging Face
2021-12-09T13:20:51Z
Hey@NDugar, the simplest way to do this would be with theaccuracymetric in thedatasetslibrary, e.g.from datasets import load_metric

# You might need to install scikit-learn
metric = load_metric("accuracy")
results = metric.compute(references=[0, 1], predictions=[0, 1])
results
# {'accuracy': 1.0}As you can see in this example, the main bit of work you need to do is create the arrays for thereferences(i.e. ground truth labels) andpredictions(predicted labels). If you’re using theTraineryou can get these arrays easily by callingTrainer.predict(test_dataset)
2021-12-09T13:56:37Z
In the sample code you wrote where do I mention the model and the dataset?
2021-12-09T14:11:09Z
If you’re after an end-to-end example, you could try adapting the official text classification examplehere
2021-12-09T14:13:42Z
ok thank you.
2021-12-09T17:32:43Z
Hey, I don’t think this code is useful for evaluation. is it possible you have another code for this purpose?
2021-12-10T20:19:34Z
The following columns in the evaluation set  don't have a corresponding arg
ument in `DebertaV2ForSequenceClassification.forward` and have been ignored
: premise, id, hypothesis.
***** Running Evaluation *****
  Num examples = 187
  Batch size = 4
100%|██████████████████████████████████████| 47/47 [00:03<00:00, 13.15it/s]Hey,I was able to run the evaluation in a different way but I am not getting the results in the end or in the logs. How do I get the result? Also it seems to ignore the key columns what do I do?@lewtun
2021-12-13T15:45:44Z
Hey@NDugarif you’re using theTrainermy suggestion would be to runTrainer.predict(your_test_dataset)so you can get all the predictions. Then you should be able to feed those into theaccuracymetric in a second step (or whatever metric you’re interested in).If you’re still having trouble, I suggest providing a minimal reproducible example, as explainedhere
2021-12-13T18:02:28Z
oh um small question what is the difference betweenTrainer.predict(your_test_dataset)andTrainer.evaluate(your_test_dataset)? I think that predict might solve my problem. i was using evaluate and getting errors previously.
2021-12-14T08:30:20Z
Trainer.predict()will return the model’s predicted labels for your dataset, whileTrainer.evaluate()will go one step further and compute the loss + anything else you’ve defined in thecompute_metrics()function