{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/community/pulls/comments/103855853","pull_request_review_id":24657447,"id":103855853,"diff_hunk":"@@ -0,0 +1,40 @@\n+# Kubernetes Testing Flakiness SLA\n+\n+This document captures the expectations of the community about flakiness in\n+our tests and our test infrastructure. It sets out an SLA (Service Level \n+Agreement) for flakiness in our tests, as well as actions that we will\n+take when we are out of SLA.\n+\n+\n+## Definition of a \"Flake\"\n+\n+We'll start by the definition of a _flake_. A test result is considered to\n+be a flake if it both fails and passes without any changes in the code\n+or environment being tested. Changes in the environment are\n+notably hard to measure or control for, but we'll do our best. Changes in\n+code are obviously quite easy to control.\n+\n+## Flakiness SLA\n+We will measure flakiness based on pull requests that are run through e2e\n+testing. The metric that we will monitor is: Flakes per PR. This metric\n+will be calculated on a daily and weekly basis using:\n+\n+Sum(Flakes) / Total(PRs)\n+\n+Our current SLA is that this metric will be less than 0.01\n+(1% of all PRs have flakes).\n+\n+## Activities on SLA violation\n+When the project is out of SLA for flakiness, we will stop the merge queue.\n+Only PRs with the label \"flake-fix\" will be allowed to merge. The merge queue\n+will remain blocked until a sufficient number of flake-fixes have been","path":"sig-testing/flakiness-sla.md","position":30,"original_position":30,"commit_id":"1124e2191b90dabdd9eefabf1f3fe052d8cd2310","original_commit_id":"1124e2191b90dabdd9eefabf1f3fe052d8cd2310","user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars0.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"body":"Does this mean we restore sla on a daily measurement interval or a weekly measurement interval?","created_at":"2017-03-02T05:55:09Z","updated_at":"2017-03-02T05:59:17Z","html_url":"https://github.com/kubernetes/community/pull/422#discussion_r103855853","pull_request_url":"https://api.github.com/repos/kubernetes/community/pulls/422","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/community/pulls/comments/103855853"},"html":{"href":"https://github.com/kubernetes/community/pull/422#discussion_r103855853"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/community/pulls/422"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/community/pulls/422","id":108689420,"html_url":"https://github.com/kubernetes/community/pull/422","diff_url":"https://github.com/kubernetes/community/pull/422.diff","patch_url":"https://github.com/kubernetes/community/pull/422.patch","issue_url":"https://api.github.com/repos/kubernetes/community/issues/422","number":422,"state":"open","locked":false,"title":"Add a document on the flakiness SLA for Kubernetes.","user":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars1.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"body":"@calebamiles @pwittrock @kubernetes/sig-testing-pr-reviews ","created_at":"2017-03-02T05:26:02Z","updated_at":"2017-03-02T05:59:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bb8b797d24127836551f883cb14436dec2f9670d","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/community/pulls/422/commits","review_comments_url":"https://api.github.com/repos/kubernetes/community/pulls/422/comments","review_comment_url":"https://api.github.com/repos/kubernetes/community/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/community/issues/422/comments","statuses_url":"https://api.github.com/repos/kubernetes/community/statuses/1124e2191b90dabdd9eefabf1f3fe052d8cd2310","head":{"label":"brendandburns:master","ref":"master","sha":"1124e2191b90dabdd9eefabf1f3fe052d8cd2310","user":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars1.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"repo":{"id":83636631,"name":"community","full_name":"brendandburns/community","owner":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars1.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/brendandburns/community","description":"Kubernetes community content","fork":true,"url":"https://api.github.com/repos/brendandburns/community","forks_url":"https://api.github.com/repos/brendandburns/community/forks","keys_url":"https://api.github.com/repos/brendandburns/community/keys{/key_id}","collaborators_url":"https://api.github.com/repos/brendandburns/community/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/brendandburns/community/teams","hooks_url":"https://api.github.com/repos/brendandburns/community/hooks","issue_events_url":"https://api.github.com/repos/brendandburns/community/issues/events{/number}","events_url":"https://api.github.com/repos/brendandburns/community/events","assignees_url":"https://api.github.com/repos/brendandburns/community/assignees{/user}","branches_url":"https://api.github.com/repos/brendandburns/community/branches{/branch}","tags_url":"https://api.github.com/repos/brendandburns/community/tags","blobs_url":"https://api.github.com/repos/brendandburns/community/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/brendandburns/community/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/brendandburns/community/git/refs{/sha}","trees_url":"https://api.github.com/repos/brendandburns/community/git/trees{/sha}","statuses_url":"https://api.github.com/repos/brendandburns/community/statuses/{sha}","languages_url":"https://api.github.com/repos/brendandburns/community/languages","stargazers_url":"https://api.github.com/repos/brendandburns/community/stargazers","contributors_url":"https://api.github.com/repos/brendandburns/community/contributors","subscribers_url":"https://api.github.com/repos/brendandburns/community/subscribers","subscription_url":"https://api.github.com/repos/brendandburns/community/subscription","commits_url":"https://api.github.com/repos/brendandburns/community/commits{/sha}","git_commits_url":"https://api.github.com/repos/brendandburns/community/git/commits{/sha}","comments_url":"https://api.github.com/repos/brendandburns/community/comments{/number}","issue_comment_url":"https://api.github.com/repos/brendandburns/community/issues/comments{/number}","contents_url":"https://api.github.com/repos/brendandburns/community/contents/{+path}","compare_url":"https://api.github.com/repos/brendandburns/community/compare/{base}...{head}","merges_url":"https://api.github.com/repos/brendandburns/community/merges","archive_url":"https://api.github.com/repos/brendandburns/community/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/brendandburns/community/downloads","issues_url":"https://api.github.com/repos/brendandburns/community/issues{/number}","pulls_url":"https://api.github.com/repos/brendandburns/community/pulls{/number}","milestones_url":"https://api.github.com/repos/brendandburns/community/milestones{/number}","notifications_url":"https://api.github.com/repos/brendandburns/community/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/brendandburns/community/labels{/name}","releases_url":"https://api.github.com/repos/brendandburns/community/releases{/id}","deployments_url":"https://api.github.com/repos/brendandburns/community/deployments","created_at":"2017-03-02T05:05:25Z","updated_at":"2017-03-02T05:05:27Z","pushed_at":"2017-03-02T05:24:28Z","git_url":"git://github.com/brendandburns/community.git","ssh_url":"git@github.com:brendandburns/community.git","clone_url":"https://github.com/brendandburns/community.git","svn_url":"https://github.com/brendandburns/community","homepage":"","size":7847,"stargazers_count":0,"watchers_count":0,"language":"Makefile","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"a8a8b85f37e1c71510ff47a04d9882d39058efab","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars1.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":57939112,"name":"community","full_name":"kubernetes/community","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars1.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/community","description":"Kubernetes community content","fork":false,"url":"https://api.github.com/repos/kubernetes/community","forks_url":"https://api.github.com/repos/kubernetes/community/forks","keys_url":"https://api.github.com/repos/kubernetes/community/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/community/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/community/teams","hooks_url":"https://api.github.com/repos/kubernetes/community/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/community/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/community/events","assignees_url":"https://api.github.com/repos/kubernetes/community/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/community/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/community/tags","blobs_url":"https://api.github.com/repos/kubernetes/community/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/community/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/community/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/community/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/community/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/community/languages","stargazers_url":"https://api.github.com/repos/kubernetes/community/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/community/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/community/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/community/subscription","commits_url":"https://api.github.com/repos/kubernetes/community/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/community/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/community/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/community/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/community/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/community/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/community/merges","archive_url":"https://api.github.com/repos/kubernetes/community/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/community/downloads","issues_url":"https://api.github.com/repos/kubernetes/community/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/community/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/community/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/community/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/community/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/community/releases{/id}","deployments_url":"https://api.github.com/repos/kubernetes/community/deployments","created_at":"2016-05-03T03:36:05Z","updated_at":"2017-02-28T18:40:35Z","pushed_at":"2017-03-02T05:35:50Z","git_url":"git://github.com/kubernetes/community.git","ssh_url":"git@github.com:kubernetes/community.git","clone_url":"https://github.com/kubernetes/community.git","svn_url":"https://github.com/kubernetes/community","homepage":"","size":7845,"stargazers_count":211,"watchers_count":211,"language":"Makefile","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":224,"mirror_url":null,"open_issues_count":95,"forks":224,"open_issues":95,"watchers":211,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/community/pulls/422"},"html":{"href":"https://github.com/kubernetes/community/pull/422"},"issue":{"href":"https://api.github.com/repos/kubernetes/community/issues/422"},"comments":{"href":"https://api.github.com/repos/kubernetes/community/issues/422/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/community/pulls/422/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/community/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/community/pulls/422/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/community/statuses/1124e2191b90dabdd9eefabf1f3fe052d8cd2310"}}}}
{"actor":{"display_login":"derekwaynecarr"}}