title: pytorch - How to Save and load model from DistributedDataParallel learning
body: <p>I'm new to the Pytorch DstributedDataParallel(), but I found that most of the tutorials save the <strong>local rank 0</strong> model during training. Which means if I get 3 machine with 4 GPU on each of them, at the final I'll get 3 model that save from each machine.</p>
  
  <p>For example in pytorch <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="noreferrer">ImageNet</a> tutorial on line 252:</p>
  
  <pre class="lang-py prettyprint-override"><code>if not args.multiprocessing_distributed or (args.multiprocessing_distributed
                  and args.rank % ngpus_per_node == 0):
              save_checkpoint({...})
  </code></pre>
  
  <p>They save the model if <code>rank % ngpus_per_node == 0</code>.</p>
  
  <p>To the best of my knowledge, DistributedDataParallel() will automatic do all reduce to the loss on the backend, without doing any further job, every process can sync the loss automatically base on that. 
  All the model on each process will only have slightly different at the end of the process. That mean we only need to save one model is enough.</p>
  
  <p>So why don't we just save model on <code>rank == 0</code>, but <code>rank % ngpus_per_node == 0</code> ?</p>
  
  <p>And which model should I used for if I get multiple model? </p>
  
  <p>If this is the right way for saving model in distribute learning, should I merge them, used one of them, or inference the result base on all three models?</p>
  
  <p>Please let me know if I'm wrong.</p>