title: How to create a custom PyTorch dataset when the order and the total number of training samples is not known in advance?
body: <p>I have a 42 GB jsonl file. Every element of this file is a json object. I create training samples from every json object. But the number of training samples from every json object that I extract can vary between 0 to 5 samples. What is the best way to create a custom PyTorch dataset without reading the entire jsonl file in memory?</p>
  
  <p>This is the dataset I am talking about - <a href="https://ai.google.com/research/NaturalQuestions/download" rel="nofollow noreferrer">Google Natural Questions</a>.</p>