title: kubectl commands timeout without details
body: <p>I'm running a Kubernetes cluster, which has worked fine for several months. Now, today, when I was about to deploy some updates, I get timeouts from the server.</p>
  
  <p>Running <code>$ kubectl get nodes</code> yields</p>
  
  <pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)
  </code></pre>
  
  <p>Running <code>$ kubectl get pods --all-namespaces</code> yields</p>
  
  <pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)
  </code></pre>
  
  <p>Running <code>$ kubectl get deployments</code> yields</p>
  
  <pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get deployments.extensions)
  </code></pre>
  
  <p>Running <code>$ kubectl get svc</code> yields</p>
  
  <pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get services)
  </code></pre>
  
  <p>Running <code>$ kubectl cluster-info</code> yields (note no output after the master)</p>
  
  <pre><code>Kubernetes master is running at https://cluster.mysite.com
  
  To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
  </code></pre>
  
  <p>As I get these timeouts for every command, troubleshooting is impossible. </p>
  
  <p>How can I continue from here to access my servers? I'm using <code>kube-aws</code>, and an AWS CloudFormation VPC.</p>
  
  <p>Thanks for your time.</p>
  
  <p><strong>EDIT</strong>:</p>
  
  <p>As per request, I ran <code>$ kubectl get pods -v 7</code> and after a bunch of cache returns got this:</p>
  
  <pre><code>I0103 16:51:32.196859 25644 round_trippers.go:414] GET cluster.mysite.com/api/v1/nodes
  I0103 16:51:32.196888 25644 round_trippers.go:421] Request Headers: 
  I0103 16:51:32.196894 25644 round_trippers.go:424]     Accept: application/json
  I0103 16:51:32.196899 25644 round_trippers.go:424]     User-Agent: kubectl/v1.8.3 (darwin/amd64) kubernetes/f0efb3c
  I0103 16:52:32.239841 25644 round_trippers.go:439]     Response Status: 504 Gateway Timeout in 60044 milliseconds
  </code></pre>
  
  <p>I also ran <code>$ kubectl cluster-info dump -v 7</code> and got:</p>
  
  <pre><code>I0103 16:51:32.196888   25644 round_trippers.go:421] Request Headers:
  I0103 16:51:32.196894   25644 round_trippers.go:424]     Accept: application/json
  I0103 16:51:32.196899   25644 round_trippers.go:424]     User-Agent: kubectl/v1.8.3 (darwin/amd64) kubernetes/f0efb3c
  I0103 16:52:32.239841   25644 round_trippers.go:439] Response Status: 504 Gateway Timeout in 60044 milliseconds
  I0103 16:52:32.242362   25644 helpers.go:207] server response object: [{
    "metadata": {},
    "status": "Failure",
    "message": "the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)",
    "reason": "Timeout",
    "details": {
      "kind": "nodes",
      "causes": [
        {
          "reason": "UnexpectedServerResponse",
          "message": "{\"metadata\":{},\"status\":\"Failure\",\"message\":\"The list operation against nodes could not be completed at this time, please try again.\",\"reason\":\"ServerTimeout\",\"details\":{\"name\":\"list\",\"kind\":\"nodes\"},\"code\":500}"
        }
      ]
    },
    "code": 504
  }]
  </code></pre>
  
  <p><strong>EDIT 2:</strong>
  Okay, now I'm just getting <code>Unable to connect to the server: EOF</code> on every request and I'm starting to get scared. This is a production cluster and I can't even access it to try to troubleshoot. Anyone have a hint on how to proceed?</p>
  
  <p><strong>EDIT 3:</strong>
  I've gotten as far as realizing that the etcd cluster was not working properly, with 2/3 nodes out of sync. Restarting one node had it properly joining the cluster again, but the second one can't start the services. The services that don't start are:</p>
  
  <ul>
  <li>etcdadm-check.service</li>
  <li>etcdadm-save.service</li>
  <li>etcdadm-update-status.service</li>
  <li>user@0.service</li>
  </ul>
  
  <p>The three former ones all give the error <code>etcdadm-check.service: Control process exited, code=exited status=3</code> and the last one gives <code>user@0.service: Start request repeated too quickly.</code>.</p>
  
  <p>Any hints on how to handle this?</p>
  
  <p>Also, after restoring the second etcd, I get <code>Unable to connect to the server: x509: certificate signed by unknown authority</code> when running any <code>kubectl</code> commands. Does this signify data loss? My certificates are still valid for over half a year, and I haven't changed anything about them.</p>
  
  <p><strong>EDIT 4</strong>:
  I still have the etcd-issue, but am following the instructions in camil's answer at this time, will update with the result. However, I solved the issue with the certificates not being valid simply by re-running <code>$ kube-aws render credentials</code> with the proper paths to my intermediate root CA, so that issue is solved.</p>