{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39626","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39626/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39626/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39626/events","html_url":"https://github.com/kubernetes/kubernetes/issues/39626","id":199613470,"number":39626,"title":"Kubectl 1.51 much slower to execute requests","user":{"login":"Raffo","id":683988,"avatar_url":"https://avatars.githubusercontent.com/u/683988?v=3","gravatar_id":"","url":"https://api.github.com/users/Raffo","html_url":"https://github.com/Raffo","followers_url":"https://api.github.com/users/Raffo/followers","following_url":"https://api.github.com/users/Raffo/following{/other_user}","gists_url":"https://api.github.com/users/Raffo/gists{/gist_id}","starred_url":"https://api.github.com/users/Raffo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Raffo/subscriptions","organizations_url":"https://api.github.com/users/Raffo/orgs","repos_url":"https://api.github.com/users/Raffo/repos","events_url":"https://api.github.com/users/Raffo/events{/privacy}","received_events_url":"https://api.github.com/users/Raffo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-01-09T17:33:42Z","updated_at":"2017-01-17T19:12:10Z","closed_at":null,"body":"**Is this a BUG REPORT or FEATURE REQUEST?** (choose one): BUG REPORT\r\n\r\n\r\n**Kubernetes version** (use `kubectl version`): \r\n\r\n````\r\nClient Version: version.Info{Major:\"1\", Minor:\"5\", GitVersion:\"v1.5.1\", GitCommit:\"82450d03cb057bab0950214ef122b67c83fb11df\", GitTreeState:\"clean\", BuildDate:\"2016-12-14T00:57:05Z\", GoVersion:\"go1.7.4\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"5\", GitVersion:\"v1.5.1+coreos.0\", GitCommit:\"cc65f5321f9230bf9a3fa171155c1213d6e3480e\", GitTreeState:\"clean\", BuildDate:\"2016-12-14T04:08:28Z\", GoVersion:\"go1.7.4\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n````\r\n\r\n**Environment**:\r\n- **Cloud provider or hardware configuration**: AWS, GKE\r\n- **OS** (e.g. from /etc/os-release): Container Linux by CoreOS stable (1235.5.0) \r\n- **Kernel** (e.g. `uname -a`): Linux ip-172-31-12-46.eu-central-1.compute.internal 4.7.3-coreos-r2 #1 SMP Sun Jan 8 00:32:25 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz GenuineIntel GNU/Linux\r\n\r\n\r\n**What happened**:\r\nKubectl 1.5.1 is much slower. If I try the following command `kubectl get pods --v=9` I can see: \r\n\r\n````\r\nI0109 18:25:44.695300   40723 round_trippers.go:299] curl -k -v -XGET  -H \"Accept: application/json, */*\" -H \"User-Agent: kubectl/v1.5.1 (darwin/amd64) kubernetes/82450d0\" -H \"Authorization: Bearer $TOKEN\" https://$MY_CLUSTER/api\r\nI0109 18:25:48.129106   40723 round_trippers.go:318] GET https://$MY_CLUSTER/api 200 OK in 3433 milliseconds\r\nI0109 18:25:48.129180   40723 round_trippers.go:324] Response Headers:\r\nI0109 18:25:48.129194   40723 round_trippers.go:327]     Content-Length: 131\r\nI0109 18:25:48.129206   40723 round_trippers.go:327]     Content-Type: application/json\r\nI0109 18:25:48.129217   40723 round_trippers.go:327]     Date: Mon, 09 Jan 2017 17:25:48 GMT\r\n````\r\n\r\nBut when running the `curl` command I get: \r\n\r\n````\r\ntime curl -k -XGET  -H \"Accept: application/json, */*\" -H \"User-Agent: kubectl/v1.5.1 (darwin/amd64) kubernetes/82450d0\" -H \"Authorization: Bearer $TOKEN\" https://$MY_CLUSTER/api\r\n{\"kind\":\"APIVersions\",\"versions\":[\"v1\"],\"serverAddressByClientCIDRs\":[{\"clientCIDR\":\"0.0.0.0/0\",\"serverAddress\":\"10.10.10.0\"}]}\r\ncurl -k -XGET -H \"Accept: application/json, */*\" -H  -H    0.03s user 0.01s system 5% cpu 0.825 total\r\n````\r\nwhich is very fast. \r\n\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\nI can reproduce it with all clusters I have in AWS when using 1.5.1 (both on client and serveR). I reproduced the same in GKE too. \r\n\r\nI am obviously available to provide more data if needed.\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/273267924","html_url":"https://github.com/kubernetes/kubernetes/issues/39626#issuecomment-273267924","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/39626","id":273267924,"user":{"login":"philk","id":29562,"avatar_url":"https://avatars.githubusercontent.com/u/29562?v=3","gravatar_id":"","url":"https://api.github.com/users/philk","html_url":"https://github.com/philk","followers_url":"https://api.github.com/users/philk/followers","following_url":"https://api.github.com/users/philk/following{/other_user}","gists_url":"https://api.github.com/users/philk/gists{/gist_id}","starred_url":"https://api.github.com/users/philk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philk/subscriptions","organizations_url":"https://api.github.com/users/philk/orgs","repos_url":"https://api.github.com/users/philk/repos","events_url":"https://api.github.com/users/philk/events{/privacy}","received_events_url":"https://api.github.com/users/philk/received_events","type":"User","site_admin":false},"created_at":"2017-01-17T19:12:10Z","updated_at":"2017-01-17T19:12:10Z","body":"I'm also seeing this.\r\n\r\nkubectl 1.5.2\r\n```\r\nI0117 10:49:39.028965   73942 round_trippers.go:299] curl -k -v -XGET  -H \"Accept: application/json\" -H \"User-Agent: kubectl-1.5.2/v1.5.2 (darwin/amd64) kubernetes/08e0995\" -H \"Authorization: Bearer [redact]\" https://[redact]/api/v1/nodes\r\nI0117 10:49:47.257350   73942 round_trippers.go:318] GET https://[redact]/api/v1/nodes 200 OK in 8228 milliseconds\r\nI0117 10:49:47.257387   73942 round_trippers.go:324] Response Headers:\r\nI0117 10:49:47.257393   73942 round_trippers.go:327]     Content-Type: application/json\r\nI0117 10:49:47.257397   73942 round_trippers.go:327]     Date: Tue, 17 Jan 2017 18:49:47 GMT\r\nI0117 10:49:47.319431   73942 request.go:904] Response Body: [actual json response]\r\n```\r\n\r\nkubectl 1.4.6\r\n```\r\nI0117 10:40:15.644083   68072 round_trippers.go:299] curl -k -v -XGET  -H \"Accept: application/json, */*\" -H \"User-Agent: kubectl-1.4.6/v1.4.6 (darwin/amd64) kubernetes/e569a27\" -H \"Authorization: Bearer [redact]\" https://[redact]/api/v1/nodes\r\nI0117 10:40:15.680027   68072 round_trippers.go:318] GET https://[redact]/api/v1/nodes 200 OK in 35 milliseconds\r\nI0117 10:40:15.680053   68072 round_trippers.go:324] Response Headers:\r\nI0117 10:40:15.680061   68072 round_trippers.go:327]     Content-Type: application/json\r\nI0117 10:40:15.680067   68072 round_trippers.go:327]     Date: Tue, 17 Jan 2017 18:40:15 GMT\r\nI0117 10:40:15.775953   68072 request.go:908] Response Body: [actual json response]\r\n```\r\n\r\nThe only difference I see is a bunch of cache stuff at the beginning of the 1.5.x command. Removing the `~/.kube/cache` causes a similar multi-second hang but on an `/api` call:\r\n\r\n```\r\nI0117 10:57:43.300426   76533 round_trippers.go:299] curl -k -v -XGET  -H \"Accept: application/json, */*\" -H \"User-Agent: kubectl-1.5.2/v1.5.2 (darwin/amd64) kubernetes/08e0995\" -H \"Authorization: Bearer [redact]\" https://[redact]/api\r\nI0117 10:57:50.640742   76533 round_trippers.go:318] GET https://[redact]/api 200 OK in 7340 milliseconds\r\nI0117 10:57:50.640773   76533 round_trippers.go:324] Response Headers:\r\nI0117 10:57:50.640779   76533 round_trippers.go:327]     Content-Type: application/json\r\nI0117 10:57:50.640783   76533 round_trippers.go:327]     Date: Tue, 17 Jan 2017 18:57:50 GMT\r\nI0117 10:57:50.640787   76533 round_trippers.go:327]     Content-Length: 132\r\nI0117 10:57:50.640871   76533 request.go:904] Response Body: {\"kind\":\"APIVersions\",\"versions\":[\"v1\"],\"serverAddressByClientCIDRs\":[{\"clientCIDR\":\"0.0.0.0/0\",\"serverAddress\":\"10.0.38.98:443\"}]}\r\n```\r\n\r\nI tried running kubectl from the hyperkube docker image on the same system (well, the docker VM) and it's normal (<1 second response) so it's probably something OSX related.\r\n\r\nI set up an SSH tunnel to the insecure port on an api server and that responded at normal speed. So it feels to me like some bad golang / SSL / OSX interaction."}}
{"actor":{"display_login":"philk"}}