Make compute_metric / load_metric distributed and use multiple test sets on do_predict
I am using transformerPyTorch examplesto do Machine Translation. I am computing multiple metrics on large validation sets.how is it possible to Integrate distributed compute_metrics with withseq2seqTrainer? (with or withoutpredict_with_generate( just for duringdo_eval) ) I took a look at distributed usage ofload_metric, however not sure if it is possible to integrate it without implementing custom train/evaluation loop.how is it possible to have two sets ondo_predict?Thanks,
2022-02-14T16:08:05Z
As long as you launch your script with thetorch.distributedlauncher, the evaluation will automatically be distributed (each GPU will see a fraction of your dataset and make the predictions on it).To pass along two datasets for predictions, you will need to tweak the example script to run the predict method on those two datasets.
2023-08-25T13:11:50Z
@sguggerI am using DeepSpeed ZeroStage 3 and am passing a custom compute_metrics() to the trainer. I have 4 GPUs (devices). The compute_metrics function is being invoked by all the devices. Moreover, all the points (let’s say there N datapoints in the eval set) in the entire eval dataset seem to be sent to the compute_metrics of all the devices. Am I missing something here?