{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44776","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44776/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44776/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44776/events","html_url":"https://github.com/kubernetes/kubernetes/issues/44776","id":223465742,"number":44776,"title":"Figure out how to reduce any watch or http errors in schduler_perf","user":{"login":"jayunit100","id":826111,"avatar_url":"https://avatars3.githubusercontent.com/u/826111?v=4","gravatar_id":"","url":"https://api.github.com/users/jayunit100","html_url":"https://github.com/jayunit100","followers_url":"https://api.github.com/users/jayunit100/followers","following_url":"https://api.github.com/users/jayunit100/following{/other_user}","gists_url":"https://api.github.com/users/jayunit100/gists{/gist_id}","starred_url":"https://api.github.com/users/jayunit100/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jayunit100/subscriptions","organizations_url":"https://api.github.com/users/jayunit100/orgs","repos_url":"https://api.github.com/users/jayunit100/repos","events_url":"https://api.github.com/users/jayunit100/events{/privacy}","received_events_url":"https://api.github.com/users/jayunit100/received_events","type":"User","site_admin":false},"labels":[{"id":125550211,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scheduling","name":"sig/scheduling","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-04-21T18:48:42Z","updated_at":"2017-12-23T12:02:40Z","closed_at":null,"author_association":"MEMBER","body":"**Bug**\r\n\r\nLots of errors happen due to various write/create operations during scheduler_perf runs.\r\n\r\nExample 1: Failed Retry spam in scheduler, the same error occurs > 10 times/sec\r\n```\r\nE0421 18:38:32.025907   28746 runners.go:978] Error while creating pod, maybe retry: <nil>\r\nE0421 18:38:32.070228   28746 runners.go:978] Error while creating pod, maybe retry: <nil>\r\nE0421 18:38:32.097123   28746 runners.go:978] Error while creating pod, maybe retry: <nil> \r\n... and so on (several more)\r\n```\r\nExample 2: Various other sorts of failures, also happening at the subsecond level, spamming the results.\r\n```\r\nE0421 18:38:41.949430   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r\nE0421 18:38:45.384900   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r\nE0421 18:38:49.736259   28746 event.go:209] Unable to write event: 'Post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused' (may retry after sleeping)\r\nE0421 18:38:51.196751   28746 event.go:209] Unable to write event: 'Post http://127.0.0.1:42185/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:42185: getsockopt: connection refused' (may retry after sleeping)\r\nE0421 18:38:51.950079   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r\nE0421 18:38:55.386130   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r\nE0421 18:38:59.736754   28746 event.go:209] Unable to write event: 'Post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused' (may retry after sleeping)\r\nE0421 18:39:01.197541   28746 event.go:209] Unable to write event: 'Post http://127.0.0.1:42185/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:42185: getsockopt: connection refused' (may retry after sleeping)\r\nE0421 18:39:01.950814   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r\nE0421 18:39:05.381689   28746 controller.go:149] unable to create required kubernetes system namespace kube-system: Post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r\nE0421 18:39:05.382180   28746 controller.go:149] unable to create required kubernetes system namespace kube-public: Post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r\nE0421 18:39:05.386905   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r\nE0421 18:39:09.737250   28746 event.go:209] Unable to write event: 'Post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused' (may retry after sleeping)\r\nE0421 18:39:11.203499   28746 event.go:209] Unable to write event: 'Post http://127.0.0.1:42185/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:42185: getsockopt: connection refused' (may retry after sleeping)\r\nE0421 18:39:11.943570   28746 controller.go:149] unable to create required kubernetes system namespace kube-system: Post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r\nE0421 18:39:11.943998   28746 controller.go:149] unable to create required kubernetes system namespace kube-public: Post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r\nE0421 18:39:11.951339   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r\nE0421 18:39:15.388153   28746 controller.go:162] unable to sync kubernetes service: Post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r\nE0421 18:39:19.737843   28746 event.go:209] Unable to write event: 'Post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused' (may retry after sleeping)\r\n```\r\n**version**\r\n \r\nmaster \r\n\r\n**What happened**:\r\n\r\nLots of write http failures for creating, events, and so on.  \r\n\r\n**What you expected to happen**:\r\n\r\nScheduler_perf may have a few sporadic failures, but they should backoff quickly and/or be clearly be related to something that we know is a acceptable system limitation, like apiserver throughput.  \r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\n\r\n`test/integration/scheduler_perf/test-performance.sh`\r\n\r\n**Anything else we need to know**:\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/353722639","html_url":"https://github.com/kubernetes/kubernetes/issues/44776#issuecomment-353722639","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/44776","id":353722639,"user":{"login":"fejta-bot","id":30488982,"avatar_url":"https://avatars1.githubusercontent.com/u/30488982?v=4","gravatar_id":"","url":"https://api.github.com/users/fejta-bot","html_url":"https://github.com/fejta-bot","followers_url":"https://api.github.com/users/fejta-bot/followers","following_url":"https://api.github.com/users/fejta-bot/following{/other_user}","gists_url":"https://api.github.com/users/fejta-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/fejta-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fejta-bot/subscriptions","organizations_url":"https://api.github.com/users/fejta-bot/orgs","repos_url":"https://api.github.com/users/fejta-bot/repos","events_url":"https://api.github.com/users/fejta-bot/events{/privacy}","received_events_url":"https://api.github.com/users/fejta-bot/received_events","type":"User","site_admin":false},"created_at":"2017-12-23T12:02:40Z","updated_at":"2017-12-23T12:02:40Z","author_association":"NONE","body":"Issues go stale after 90d of inactivity.\nMark the issue as fresh with `/remove-lifecycle stale`.\nStale issues rot after an additional 30d of inactivity and eventually close.\n\nPrevent issues from auto-closing with an `/lifecycle frozen` comment.\n\nIf this issue is safe to close now please do so with `/close`.\n\nSend feedback to sig-testing, kubernetes/test-infra and/or `@fejta`.\n/lifecycle stale"}}
{"actor":{"display_login":"fejta-bot"}}