title: Linux TCP weirdly unresponsive when under heavy load
body: <p>I'm trying to get an HTTP server I'm writing on to behave well when under heavy load, but I'm getting some weird behavior that I cannot quite understand.</p>
  
  <p>My testing consists of using <code>ab</code> (the Apache benchmark program) over the loopback interface at a concurrency level of 1000 (<code>ab -n 50000 -c 1000 http://localhost:8080/apa</code>), while straceing the server process. Strace both slows processing down well enough for the problem to be readily reproducible and allows me to debug the server internals post completion to some extent. I also capture the network traffic with <code>tcpdump</code> while the test is running.</p>
  
  <p>What happens is that <code>ab</code> stops running a while into the test, complaining that a connection returned <code>ECONNRESET</code>, which I find a bit weird. I could easily buy into a connection timing out since the server might simply not have the bandwidth to process them all, but shouldn't that reasonably return <code>ETIMEDOUT</code> or even <code>ECONNREFUSED</code> if not all connections can be accepted?</p>
  
  <p>I used Wireshark to extract the packets constituting the first connection to return <code>ECONNRESET</code>, and its brief packet list looks like this:
  <img src="https://i.stack.imgur.com/HMo7i.png" alt="packet list">
  (The entire <code>tcpdump</code> file of this connection is available <a href="http://www.dolda2000.com/~fredrik/tmp/rstdump" rel="nofollow noreferrer">here</a>.)</p>
  
  <p>As you can see from this dump, the connection is accepted (after a few <code>SYN</code> retransmissions), and then the request is retransmitted a few times, and then the server resets the connection. I'm wondering, what could cause this to happen? Normally, Linux' TCP implementation ACKs data before the reading process even chooses to receive it so long as their is space in the TCP window, so why doesn't it do that here? Are there some kind of shared buffers that are running out? Most importantly, why is the kernel responding with a <code>RST</code> packet all of a sudden instead of simply waiting and letting the client re-transmit further?</p>
  
  <p>For the record, the strace of the process indicates that it never even accepts a connection from the port in this connection (port 56946), so this seems to be something Linux does on its own. It is also worth noting that the server works perfectly well as long as ab's concurrency level is low enough (it works perfectly well up to about 100, and then starts failing intermittently somewhere between 100-500), and that its request throughput is rather constant regardless of the concurrency level (it processes somewhere between 6000-7000 requests per second as long as it isn't being straced). I have not found any particular correlation between the frequency of the problem occurring and my backlog setting to <code>listen()</code> (I'm currently using 128, but I've tried up to 1024 without it seeming to make a difference).</p>
  
  <p>In case it matters, I'm running Linux 3.2.0 on this AMD64 box.</p>