title: Kubernetes: slow DNS
body: <p>I have a Kubernetes cluster that was initialized using the <code>kube-up.sh</code> script inside AWS, and occasionally there's a very slow DNS lookup when finding one service from inside another pod. Here's the basic picture:</p>
  
  <pre><code>    (browser)
          |
          V
        (ELB)
          |
          V
  (front-end service)
          |
          V
    (front-end pod)
          |
          V
   (back-end service)
          |
          V
    (back-end pod)
          |
          V
      (database)
  </code></pre>
  
  <p>I have timing logging installed at the front-end and back-end levels, and their numbers are wildly divergent for <strong>some requests</strong>. Occasionally we'll see a request that the FE nginx logging says takes 8.3 seconds, but the back-end gunicorn process says takes 30ms.</p>
  
  <p>I can <code>exec</code> into the FE pod and do a <code>curl</code> to the backend endpoint to get timing data according to the example in <a href="https://josephscott.org/archives/2011/10/timing-details-with-curl/" rel="nofollow">this blog post</a>, and it looks like this:</p>
  
  <pre><code>        time_namelookup:  3.513
             time_connect:  3.513
          time_appconnect:  0.000
         time_pretransfer:  3.513
            time_redirect:  0.000
       time_starttransfer:  3.520
                          ----------
               time_total:  3.520
  </code></pre>
  
  <p>So the slowness seems to be coming from DNS. We have a separate cluster set up for staging, and this sort of thing doesn't seem to be happening there, so I'm not sure what to make of it. Most requests happen in a reasonable amount of time, less than 50ms, but every tenth one or so takes multiple seconds to resolve.</p>
  
  <p>I found <a href="https://groups.google.com/forum/#!topic/google-containers/oxVw35elJj8" rel="nofollow">this thread</a> that made it sound like SkyDNS's use of etcd might be the problem, but I'm not sure how to verify that or fix it. And this is happening <em>way</em> too often to be periodic missing configuration values (our traffic isn't that high).</p>