title: Why there is so many connections in LAST_ACK state?
body: <p>I've written a simple HTTP server to study how <code>select</code> function works.</p>
  
  <p>There is the code:</p>
  
  <pre><code>#include &lt;sys/types.h&gt;
  #include &lt;sys/socket.h&gt;
  #include &lt;netinet/in.h&gt;
  #include &lt;arpa/inet.h&gt;
  #include &lt;sys/select.h&gt;
  #include &lt;sys/capsicum.h&gt;
  #include &lt;unistd.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;string.h&gt;
  #include &lt;errno.h&gt;
  #include &lt;fcntl.h&gt;
  
  static const char ok_response[] =
      "HTTP/1.1 200 OK\r\n"
      "Content-Type: text/html\r\n"
      "Content-Length: 76\r\n"
      "\r\n"
      "&lt;html&gt;"
       "&lt;head&gt;"
        "&lt;title&gt; Kokoko &lt;/title&gt;"
       "&lt;/head&gt;"
       "&lt;body&gt;"
         "Hello kokoko"
       "&lt;/body&gt;"
      "&lt;/html&gt;"
      "\r\n";
  
  struct write_state
  {
      const char *ptr;
      size_t position;
      size_t length;
  };
  
  static struct write_state states[FD_SETSIZE];
  
  int main ()
  {
      int res;
      int acceptor = socket (PF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
      if (acceptor == -1) {
          perror ("Error creating socket");
          goto end;
      }
  
      int optval = 1;
      setsockopt(acceptor, SOL_SOCKET, SO_REUSEADDR, &amp;optval, sizeof (optval));
  
      struct sockaddr_in addr;
      memset (&amp;addr, 0, sizeof (struct sockaddr_in));
      addr.sin_family = AF_INET;
      addr.sin_port = htons(8080);
      inet_pton(AF_INET, "127.0.0.1", &amp;addr.sin_addr);
  
      res = bind (acceptor, (struct sockaddr*) &amp;addr, sizeof (struct sockaddr));
      if (res == -1) {
          perror ("bind() error");
          goto end;
      }
  
      res = listen (acceptor, 50);
      if (res == -1) {
          perror ("listen() error");
          goto end;
      }
  
      fd_set acceptor_set;
      fd_set connection_set;
      int i;
  
      FD_ZERO (&amp;acceptor_set);
      FD_ZERO (&amp;connection_set);
  
      FD_SET (acceptor, &amp;acceptor_set);
  
      res = cap_enter();
      if (res == -1) {
          perror ("cap_enter() error");
          goto end;
      }
  
      cap_rights_t acc_rights;
      cap_rights_limit (acceptor, cap_rights_init (&amp;acc_rights, CAP_ACCEPT,
                        CAP_READ, CAP_WRITE, CAP_EVENT,
                        /* FIXME */ CAP_FCNTL, CAP_SHUTDOWN));
  
      while (1) {
          fd_set read_set = acceptor_set;
          fd_set write_set = connection_set;
          res = select (FD_SETSIZE, &amp;read_set, &amp;write_set, NULL, NULL);
          if (res == -1 &amp;&amp; errno != EINTR) {
              perror ("select() error");
              goto end;
          }
          if (res &gt; 0) {
              if (FD_ISSET (acceptor, &amp;read_set)) {
                  int fd = accept (acceptor, NULL, NULL);
                  if (fd == -1) {
                      perror ("accept() error");
                      goto end;
                  }
                  int flags = fcntl (fd, F_GETFL);
                  if (flags == -1) {
                      perror ("fcntl() error");
                      goto end;
                  }
                  flags = fcntl (fd, F_SETFL, flags | O_NONBLOCK);
                  if (flags == -1) {
                      perror ("fcntl() error");
                      goto end;
                  }
                  states[fd].ptr = ok_response;
                  states[fd].position = 0;
                  states[fd].length = strlen (ok_response);
                  FD_SET (fd, &amp;connection_set);
              }
              for (i=0; i&lt;FD_SETSIZE; i++) {
                  if (FD_ISSET (i, &amp;write_set)) {
                      int length = write (i, states[i].ptr + states[i].position,
                                          states[i].length - states[i].position);
                      if (length == -1) {
                          perror ("write error");
                          goto end;
                      }
                      states[i].position += length;
                      if (states[i].position == states[i].length) {
                          res = shutdown (i, SHUT_RDWR);
                          if (res == -1) {
                              perror ("shutdown() error");
                              goto end;
                          }
                          char buffer[32];
                          int length;
                          for (;;) {
                              length = read (i, buffer, sizeof (buffer));
                              if (length == -1)
                              {
                                  perror ("read");
                                  break;
                              }
                              if (length == 0) break;
                          }
                          close (i);
                          FD_CLR (i, &amp;connection_set);
                      }
                  }
              }
          }
      }
  
  end:
      if (acceptor != -1) close (acceptor);
      for (i=0; i&lt;FD_SETSIZE; i++) {
          if (FD_ISSET (i, &amp;connection_set)) close (i);
      }
  
      return 0;
  }
  </code></pre>
  
  <p>It listens for a connection at address <code>localhost:8080</code> until you hit C-c. When I test this server with httperf like so: </p>
  
  <pre><code>httperf --hog --client=0/1 --server=localhost --port=8080 --uri=/ --rate=9000 --send-buffer=4096 --recv-buffer=16384 --ssl-protocol=auto --num-conns=10000 --num-calls=1
  Maximum connect burst length: 10
  
  Total: connections 10000 requests 10000 replies 10000 test-duration 1.111 s
  
  Connection rate: 8998.0 conn/s (0.1 ms/conn, &lt;=10 concurrent connections)
  Connection time [ms]: min 0.1 avg 0.2 max 0.4 median 0.5 stddev 0.0
  Connection time [ms]: connect 0.1
  Connection length [replies/conn]: 1.000
  
  Request rate: 8998.0 req/s (0.1 ms/req)
  Request size [B]: 62.0
  
  Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
  Reply time [ms]: response 0.1 transfer 0.0
  Reply size [B]: header 64.0 content 76.0 footer 0.0 (total 140.0)
  Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0
  
  CPU time [s]: user 0.04 system 0.30 (user 3.3% system 27.0% total 30.3%)
  Net I/O: 1775.0 KB/s (14.5*10^6 bps)
  
  Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
  Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
  </code></pre>
  
  <p>I get too many (thousands) sockets in LAST_ACK state. Netstat output looks like:<br>
  <code>tcp4     0      62 127.0.0.1.1696      127.0.0.1.8080      LAST_ACK</code><br>
  When server is not heavily loaded, there are no connections in <code>LAST_ACK</code> state.</p>
  
  <p>From what I have read about <code>LAST_ACK</code> state, I can assume what a connection is being closed, <code>FIN</code> is sent, and you wait for the last <code>ACK</code> to close a connection. I also found this diagram very useful: <a href="http://www4.cs.fau.de/Projects/JX/Projects/TCP/tcpstate.html" rel="nofollow noreferrer">http://www4.cs.fau.de/Projects/JX/Projects/TCP/tcpstate.html</a></p>
  
  <p>I still do not understand what is happening and if it is normal or not. I think maybe I close a connection in the wrong way. Am I doing this right in the provided code? Can I safely close TCP connection when I have data to read from it? As you can see I ignore HTTP request completely and send the same response all the time.</p>
  
  <p>P.S. Testing nginx with httperf in the same way leaves lots of connections in <code>TIME_WAIT</code> state, but I believe this is normal.</p>