title: RMSprop, Adam, AdaDelta test accuracy does not improve using Caffe
body: <p>I am <code>finetuning</code> using <code>Caffe</code> on an image dataset on a <code>Tesla K40</code>. Using a <code>batch size=47</code>, <code>solver_type=SGD</code>, <code>base_lr=0.001</code>, <code>lr_policy="step"</code>, <code>momentum=0.9</code>, <code>gamma=0.1</code>, the <code>training loss</code> decreases and <code>test accuracy</code> goes from <code>2%-50%</code> in <code>100</code> iterations which is quite good.</p>
  
  <p>When using other optimisers such as <code>RMSPROP</code>, <code>ADAM</code> and <code>ADADELTA</code>, the <code>training loss</code> remains almost the same even and no improvement in <code>test accuracy</code> after <code>1000</code> iterations.</p>
  
  <p>For <code>RMSPROP</code>, I have changed the respective parameters as mentioned <a href="https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_solver_rmsprop.prototxt" rel="noreferrer">here</a>.</p>
  
  <p>For <code>ADAM</code>, I have changed the respective parameters as mentioned <a href="https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_solver_adam.prototxt" rel="noreferrer">here</a></p>
  
  <p>For <code>ADADELTA</code>, I have changed the respective parameters as mentioned <a href="https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_adadelta_solver.prototxt" rel="noreferrer">here</a></p>
  
  <p>Can someone please tell me what i am doing wrong?</p>