Evaluation loss depends on batch size
I train a token classification model on a private dataset.I noticed that the evaluation loss has different values if I change the value of per_device_eval_batch_size.Is it a known thing?The total test set size is not divisible with the batch size (per_device_eval_batch_size*gpu_num).When I put a batch size that divides the total test set size, I still get a different number from what I get when I calculate the loss directly with torch.nn.CrossEntropyLoss.Also, accuracy, recall, etc do not change with the batch size.Thanks for you help!
2024-10-14T23:46:30Z
Apparently a known but neglected issue for several years…Batch size during training vs batch size during evaluationBeginnersI am confused about the difference between batch size during training versus batch size during evaluation. I am trying to measure how batch size influences the inference time(speed of prediction) of different NLP models after they have  been trained using the Huggingface Trainer API. The code I used is below 
def print_summary(result):
    print(f"Time: {result.metrics['test_runtime']:.2f}")
    print(f"Samples/second: {result.metrics['test_samples_per_second']:.2f}")
    print(f"Test Loss: {res…