What's the best way to speed up inference on a large dataset?
Hi,I have been trying to do inference of a model I’ve finetuned for a large dataset.I’ve done it this way:Summary of the tasksIterating over all the questions and contexts but it’s too slow.This way from the course seems to be quite ok but I run into memory issues, assuming because the whole dataset is in a dict?batch = {k: eval_set_for_model[k].to(device) for k in eval_set_for_model.column_names}
trained_model = AutoModelForQuestionAnswering.from_pretrained(trained_checkpoint).to(
    device
)

with torch.no_grad():
    outputs = trained_model(**batch)Is there some way I can pass the dataset like I would in lightning directly and iterate over the batches dynamically?I.e. instead of getting the batch manually as above, do something likefor batch in iter(dataset):
 pred = model(**batch)?Thanks a lot in advance
2022-03-13T15:16:37Z
You may find the discussion onpipeline batchinguseful. I think batching is usually only worth it for running on GPU. If you are doing inference on CPU looking intoONNXmight make sense (probably it’s only worth the effort if you are going to be doing inference multiple times – if it’s a one-time thing you might just prefer to wait a bit longer!)
2022-03-13T15:53:08Z
thanks for the answer,yes, I’ve tried pipeline batching, but I seem to not be able to feed the dataset into the pipeline (just for qa, it works for classification etc, for q&a I get asked to makea  dict out of it).It’s running on GPU.
2022-03-13T15:53:51Z
I’m doing it for the kaggle student nlp project, so it’s just that I have the 9 hours inference limit.Do you know how to do pipeline batching for q&a?