title: Standby NN RPC Latency Issues
body: <p>I am currently facing an issue where i see RPC latency issues on my secondary NameNode from time to time. Log event at that instance looks like this:</p>
  
  <pre><code>The health test result for NAME_NODE_RPC_LATENCY has become bad: The moving average of the RPC latency is 6 second(s) over the previous 5 minute(s). The moving average of the queue time is 0 second(s). The moving average of the processing time is 6 second(s). Critical threshold: 5 second(s). 
  Time: Sep 25, 2015 5:52:02 AM 
  </code></pre>
  
  <p>We see these RPC errors from time to time. I looked at the logs and don't see anything different.</p>
  
  <p>I checked the logs at the time the issue has occurred and found nothing unusual </p>
  
  <pre><code>Call#0 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby
  </code></pre>
  
  <p>This is usually because the client does not know where he is connecting to and as this node is standby it seamlessly connects to the active NN. </p>
  
  <p>I checked the RPC avg queue time and processing time, at one point I saw that there was a burst in connections and we got an alert, however the other time when this went bad there were no burst in requests. </p>
  
  <p>Any advise ? anything else I can check ?</p>