{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51899","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51899/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51899/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51899/events","html_url":"https://github.com/kubernetes/kubernetes/issues/51899","id":255030243,"number":51899,"title":"API call latencies shot up on large cluster tests","user":{"login":"shyamjvs","id":4333971,"avatar_url":"https://avatars3.githubusercontent.com/u/4333971?v=4","gravatar_id":"","url":"https://api.github.com/users/shyamjvs","html_url":"https://github.com/shyamjvs","followers_url":"https://api.github.com/users/shyamjvs/followers","following_url":"https://api.github.com/users/shyamjvs/following{/other_user}","gists_url":"https://api.github.com/users/shyamjvs/gists{/gist_id}","starred_url":"https://api.github.com/users/shyamjvs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shyamjvs/subscriptions","organizations_url":"https://api.github.com/users/shyamjvs/orgs","repos_url":"https://api.github.com/users/shyamjvs/repos","events_url":"https://api.github.com/users/shyamjvs/events{/privacy}","received_events_url":"https://api.github.com/users/shyamjvs/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":681622079,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/milestone-labels-complete","name":"milestone-labels-complete","color":"ededed","default":false},{"id":114528068,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/critical-urgent","name":"priority/critical-urgent","color":"e11d21","default":false},{"id":435667038,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-blocker","name":"release-blocker","color":"d93f0b","default":false},{"id":173493835,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/api-machinery","name":"sig/api-machinery","color":"d2b48c","default":false},{"id":125010198,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scalability","name":"sig/scalability","color":"d2b48c","default":false},{"id":611767149,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/status/approved-for-milestone","name":"status/approved-for-milestone","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":{"login":"shyamjvs","id":4333971,"avatar_url":"https://avatars3.githubusercontent.com/u/4333971?v=4","gravatar_id":"","url":"https://api.github.com/users/shyamjvs","html_url":"https://github.com/shyamjvs","followers_url":"https://api.github.com/users/shyamjvs/followers","following_url":"https://api.github.com/users/shyamjvs/following{/other_user}","gists_url":"https://api.github.com/users/shyamjvs/gists{/gist_id}","starred_url":"https://api.github.com/users/shyamjvs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shyamjvs/subscriptions","organizations_url":"https://api.github.com/users/shyamjvs/orgs","repos_url":"https://api.github.com/users/shyamjvs/repos","events_url":"https://api.github.com/users/shyamjvs/events{/privacy}","received_events_url":"https://api.github.com/users/shyamjvs/received_events","type":"User","site_admin":false},"assignees":[{"login":"shyamjvs","id":4333971,"avatar_url":"https://avatars3.githubusercontent.com/u/4333971?v=4","gravatar_id":"","url":"https://api.github.com/users/shyamjvs","html_url":"https://github.com/shyamjvs","followers_url":"https://api.github.com/users/shyamjvs/followers","following_url":"https://api.github.com/users/shyamjvs/following{/other_user}","gists_url":"https://api.github.com/users/shyamjvs/gists{/gist_id}","starred_url":"https://api.github.com/users/shyamjvs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shyamjvs/subscriptions","organizations_url":"https://api.github.com/users/shyamjvs/orgs","repos_url":"https://api.github.com/users/shyamjvs/repos","events_url":"https://api.github.com/users/shyamjvs/events{/privacy}","received_events_url":"https://api.github.com/users/shyamjvs/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/36","html_url":"https://github.com/kubernetes/kubernetes/milestone/36","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/36/labels","id":2545392,"number":36,"title":"v1.8","description":"Kubernetes 1.8 release milestone, date is TBD.","creator":{"login":"timothysc","id":169553,"avatar_url":"https://avatars0.githubusercontent.com/u/169553?v=4","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":742,"state":"open","created_at":"2017-05-26T17:18:43Z","updated_at":"2017-09-23T19:40:43Z","due_on":null,"closed_at":null},"comments":66,"created_at":"2017-09-04T12:39:04Z","updated_at":"2017-09-23T22:16:11Z","closed_at":null,"author_association":"MEMBER","body":"From 5k-node density test run (no. 26) on last friday:\r\n\r\n```\r\nI0901 10:11:29.395] Sep  1 10:11:29.395: INFO: WARNING Top latency metric: {Resource:pods Subresource: Verb:LIST Latency:{Perc50:66.805ms Perc90:375.804ms Perc99:8.111217s Perc100:0s} Count:11911}\r\nI0901 10:11:29.396] Sep  1 10:11:29.395: INFO: Top latency metric: {Resource:nodes Subresource: Verb:LIST Latency:{Perc50:417.103ms Perc90:1.090847s Perc99:2.336106s Perc100:0s} Count:1261}\r\nI0901 10:11:29.396] Sep  1 10:11:29.395: INFO: WARNING Top latency metric: {Resource:nodes Subresource:status Verb:PATCH Latency:{Perc50:5.633ms Perc90:279.15ms Perc99:1.04109s Perc100:0s} Count:5543730}\r\nI0901 10:11:29.396] Sep  1 10:11:29.395: INFO: Top latency metric: {Resource:pods Subresource:binding Verb:POST Latency:{Perc50:1.442ms Perc90:15.171ms Perc99:921.523ms Perc100:0s} Count:155096}\r\nI0901 10:11:29.396] Sep  1 10:11:29.395: INFO: Top latency metric: {Resource:replicationcontrollers Subresource: Verb:POST Latency:{Perc50:3.13ms Perc90:218.472ms Perc99:886.065ms Perc100:0s} Count:5049}\r\n```\r\n\r\nAnd from the last healthy run (no. 23) of the test:\r\n\r\n```\r\nI0826 03:49:44.301] Aug 26 03:49:44.301: INFO: Top latency metric: {Resource:pods Subresource: Verb:LIST Latency:{Perc50:830.411ms Perc90:1.468316s Perc99:2.627309s Perc100:0s} Count:10615}\r\nI0826 03:49:44.302] Aug 26 03:49:44.301: INFO: Top latency metric: {Resource:nodes Subresource: Verb:LIST Latency:{Perc50:337.423ms Perc90:427.443ms Perc99:998.426ms Perc100:0s} Count:1800}\r\nI0826 03:49:44.302] Aug 26 03:49:44.301: INFO: Top latency metric: {Resource:apiservices Subresource: Verb:PUT Latency:{Perc50:954Âµs Perc90:5.726ms Perc99:138.528ms Perc100:0s} Count:840}\r\nI0826 03:49:44.302] Aug 26 03:49:44.301: INFO: Top latency metric: {Resource:namespaces Subresource: Verb:GET Latency:{Perc50:445Âµs Perc90:2.641ms Perc99:83.57ms Perc100:0s} Count:1608}\r\nI0826 03:49:44.302] Aug 26 03:49:44.301: INFO: Top latency metric: {Resource:nodes Subresource:status Verb:PATCH Latency:{Perc50:3.22ms Perc90:9.449ms Perc99:55.588ms Perc100:0s} Count:7033941}\r\n```\r\n\r\nThis is a huge increase we're seeing:\r\nLIST pods : 2.6s -> 8.1s\r\nLIST nodes: 1s -> 2.3s\r\nPATCH node-status: 56ms -> 1s\r\n...\r\n\r\ncc @kubernetes/sig-api-machinery-bugs @kubernetes/sig-scalability-misc @smarterclayton @wojtek-t @gmarek \r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/331673343","html_url":"https://github.com/kubernetes/kubernetes/issues/51899#issuecomment-331673343","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51899","id":331673343,"user":{"login":"shyamjvs","id":4333971,"avatar_url":"https://avatars3.githubusercontent.com/u/4333971?v=4","gravatar_id":"","url":"https://api.github.com/users/shyamjvs","html_url":"https://github.com/shyamjvs","followers_url":"https://api.github.com/users/shyamjvs/followers","following_url":"https://api.github.com/users/shyamjvs/following{/other_user}","gists_url":"https://api.github.com/users/shyamjvs/gists{/gist_id}","starred_url":"https://api.github.com/users/shyamjvs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shyamjvs/subscriptions","organizations_url":"https://api.github.com/users/shyamjvs/orgs","repos_url":"https://api.github.com/users/shyamjvs/repos","events_url":"https://api.github.com/users/shyamjvs/events{/privacy}","received_events_url":"https://api.github.com/users/shyamjvs/received_events","type":"User","site_admin":false},"created_at":"2017-09-23T22:16:11Z","updated_at":"2017-09-23T22:16:11Z","author_association":"MEMBER","body":"Commit 99a9ee5a3c is unhealthy:\r\n\r\n```\r\nSep 23 18:14:04.314: INFO: WARNING Top latency metric: {Resource:pods Subresource: Verb:LIST Latency:{Perc50:1.1407s Perc90:3.400196s Perc99:5.078185s Perc100:0s} Count:14832}\r\nSep 23 18:14:04.314: INFO: Top latency metric: {Resource:nodes Subresource: Verb:LIST Latency:{Perc50:408.691ms Perc90:793.079ms Perc99:1.831115s Perc100:0s} Count:1703}\r\nSep 23 18:14:04.314: INFO: WARNING Top latency metric: {Resource:nodes Subresource:status Verb:PATCH Latency:{Perc50:4.134ms Perc90:103.428ms Perc99:1.139633s Perc100:0s} Count:7396342}\r\nSep 23 18:14:04.314: INFO: Top latency metric: {Resource:services Subresource: Verb:LIST Latency:{Perc50:1.057ms Perc90:5.84ms Perc99:474.047ms Perc100:0s} Count:511}\r\nSep 23 18:14:04.314: INFO: Top latency metric: {Resource:replicationcontrollers Subresource: Verb:LIST Latency:{Perc50:869µs Perc90:14.563ms Perc99:436.187ms Perc100:0s} Count:405}\r\n```\r\n\r\nNew range - https://github.com/kubernetes/kubernetes/compare/11299e363c538c...99a9ee5a3c52"}}
{"actor":{"display_login":"shyamjvs"}}