title: Understanding distributing deep learning with tensorflow
body: <p>I have written an RNN with tensorflow that trains on some time series data. I am training the model on moving window taking in 2 time slices and predicting for the third. </p>
  
  <p>I batch my inputs and labels and run the entire set of batches over 30 epochs. The model runs fine as a single TF application. </p>
  
  <p>I am now looking to mature my approach and distribute computation. I have this working by using DataSet for batching and MonitoredTrainingSession. My cluster consists of one ps and two workers. </p>
  
  <p>I have noticed that the distributed training was <strong>much</strong> slower than on a single machine. I attribute this down to the network (without any scientific measurements) however I am also concerned that I may have misunderstood the set up - and my distributed model is not behaving in the same way as a single self contained process. </p>
  
  <p>I have provided a local copy of the input data to each worker and the output (plot) was different on each worker. I feel the data should be loaded by the ps instead and distributed to the workers? This way it will <em>split</em> the data equally between the workers? Is this the correct approach? However, does this then break the RNN’s ability to train on sequential data when each worker is not seeing the entire picture?</p>
  
  <p><strong>Edit</strong></p>
  
  <p>On closer inspection, does the ps collect individual gradients from each worker and update them all on each other’s progress so that they each are aware of one another?</p>