title: Python processing items from list/queue and saving progress
body: <p>If I have about 10+ million little tasks to process in python (convert images or so), how can I create queue and <em>save progress</em> in case of crash in processing. To be clear, how can I save progress or stop process whatever I want and continue processing from the last point.</p>
  
  <p>Also how to deal with multiple threads in that case?</p>
  
  <p>In general question is how to save progress on processed data to file. Issue if it huge amount of very small files, saving file after each iteration will be longer than processing itself...</p>
  
  <p>Thanks! </p>
  
  <p>(sorry for my English if its not clear)</p>