title: Why does my Perl TCP server script hang with many TCP connections?
body: <p>I've got a strange issue with a server accepting TCP connections. Even though there are normally some processes waiting, at some volume of connections it hangs.</p>
  
  <p>Long version:</p>
  
  <p>The server is written in Perl and binds a <code>$srv</code> socket with the reuse flag and listen == 5. Afterwards, it forks into 10 processes with a loop of <code>$clt=$srv-&gt;accept(); do_processing($clt); $clt-&gt;shutdown(2);</code></p>
  
  <p>The client written in C is also very simple - it sends some lines, then receives all lines available and does a <code>shutdown(sockfd, 2);</code> There's nothing async going on and at the end both send and receive queues are empty (as reported by <code>netstat</code>).</p>
  
  <p>Connections last only ~20ms. All clients behave the same way, are the same implementation, etc. Now let's say I'm accepting <code>X</code> connections from client 1 and another <code>X</code> from client 2. Processes still report that they're idle all the time. If I add another <code>X</code> connections from client 3, suddenly the server processes start hanging just after accepting. The first blocking thing they do after <code>accept();</code> is <code>while (&lt;$clt&gt;) ...</code> - but they don't get any data (on the first try already). Suddenly all 10 processes are in this state and do not stop waiting. On <code>strace</code>, the server processes seem to hang on <code>read()</code>, which makes sense.</p>
  
  <p>There are loads of connections in <code>TIME_WAIT</code> state belonging to that server (~100 when the problem starts to manifest), but this might be a red herring.</p>
  
  <p>What could be happening here?</p>
  
  <hr>
  
  <p>After some more analysis: It turned out that the client was at fault, not closing previous connections properly before trying the next one. The servers at the beginning of the load-balancing list were left stale connections.</p>