title: MongoDB Replica Set is really slow when a member becomes unavailable
body: <p>we have faced a problem with our production system yesterday which I
  am unable to explain when following the official documentation.</p>
  
  <p><strong>The Setup:</strong></p>
  
  <ul>
  <li>MongoDB 2.0.1</li>
  <li>Replica Set spanning over 5 Servers with one preferred Primary</li>
  <li>PHP Application using V 1.2.6 of the PHP Driver</li>
  <li>One collection with roughly 3 million entries</li>
  <li>SlaveOkay set to true for every connection</li>
  <li>The connection string for MongoDB includes all five servers from the replica set</li>
  </ul>
  
  <p><strong>The Problem:</strong></p>
  
  <p>Yesterday one of the secondaries suddenly died (hardware crash) and
  became totally unavailable. From that point onwards many read
  operations carried out from the PHP driver took over 30 seconds to
  complete (before it rarely took over 0.1 second).</p>
  
  <ul>
  <li>rs.status() was clearly reporting that the failed secondary was
  NOK and not accessible.</li>
  <li>all same queries sent directly via the console to the primary or
  any secondary were processed in under 0.1 seconds (as expected)</li>
  </ul>
  
  <p>In the beginning I have taken the reference to the failed secondary
  from the connection string in PHP, yet this yielded no change to the
  overall performance. Only when I did an explicit
  rs.remove(hostname_port) from the replica set did the performance
  scale back to normal.</p>
  
  <p>I suppose this is not expected behavior? Can we shield ourselves from
  anything like this in the future?</p>