{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/comments/151881637","pull_request_review_id":77659032,"id":151881637,"diff_hunk":"@@ -154,11 +168,15 @@ func (server *Server) Listen(port int) {\n }\n \n func loggingMiddleware(logger *Logger) gin.HandlerFunc {\n+\tvar requestID int64\n \treturn func(c *gin.Context) {\n+\t\treqID := strconv.FormatInt(atomic.AddInt64(&requestID, 1), 10)","path":"pkg/chartmuseum/server.go","position":89,"original_position":79,"commit_id":"8a7e989c501d22bd59993d2d52f9542554eba462","original_commit_id":"0e9d2e36eaeee752ca26fc4f9ea4e5f151fe3635","user":{"login":"jdolitsky","id":393494,"avatar_url":"https://avatars3.githubusercontent.com/u/393494?v=4","gravatar_id":"","url":"https://api.github.com/users/jdolitsky","html_url":"https://github.com/jdolitsky","followers_url":"https://api.github.com/users/jdolitsky/followers","following_url":"https://api.github.com/users/jdolitsky/following{/other_user}","gists_url":"https://api.github.com/users/jdolitsky/gists{/gist_id}","starred_url":"https://api.github.com/users/jdolitsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdolitsky/subscriptions","organizations_url":"https://api.github.com/users/jdolitsky/orgs","repos_url":"https://api.github.com/users/jdolitsky/repos","events_url":"https://api.github.com/users/jdolitsky/events{/privacy}","received_events_url":"https://api.github.com/users/jdolitsky/received_events","type":"User","site_admin":false},"body":"What do these end up looking like?  like `\"Request ID\": 455` ? Will this result in collisions?","created_at":"2017-11-19T23:04:27Z","updated_at":"2017-11-19T23:04:27Z","html_url":"https://github.com/kubernetes-helm/chartmuseum/pull/19#discussion_r151881637","pull_request_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/comments/151881637"},"html":{"href":"https://github.com/kubernetes-helm/chartmuseum/pull/19#discussion_r151881637"},"pull_request":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19","id":153154651,"html_url":"https://github.com/kubernetes-helm/chartmuseum/pull/19","diff_url":"https://github.com/kubernetes-helm/chartmuseum/pull/19.diff","patch_url":"https://github.com/kubernetes-helm/chartmuseum/pull/19.patch","issue_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/issues/19","number":19,"state":"open","locked":false,"title":"Fix index response times increasing with many uploads","user":{"login":"davidovich","id":1440370,"avatar_url":"https://avatars0.githubusercontent.com/u/1440370?v=4","gravatar_id":"","url":"https://api.github.com/users/davidovich","html_url":"https://github.com/davidovich","followers_url":"https://api.github.com/users/davidovich/followers","following_url":"https://api.github.com/users/davidovich/following{/other_user}","gists_url":"https://api.github.com/users/davidovich/gists{/gist_id}","starred_url":"https://api.github.com/users/davidovich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidovich/subscriptions","organizations_url":"https://api.github.com/users/davidovich/orgs","repos_url":"https://api.github.com/users/davidovich/repos","events_url":"https://api.github.com/users/davidovich/events{/privacy}","received_events_url":"https://api.github.com/users/davidovich/received_events","type":"User","site_admin":false},"body":"Index regeneration before this PR was causing a concurrency pile-up when\r\nmany requests were coming in at a fast pace.  This was causing the time\r\nto return /index to always increase. The behavior was observed on a high\r\nlatency connection to Google Cloud Storage, but would also occur with\r\nother network stores.\r\n\r\nWith this PR, the regeneration time is constant and thus the serving of\r\nindex.yaml is also constant.\r\n\r\nThe implementation takes a general approach to first make initial\r\nnetwork requests (to satisfy the object list from storage) and to pile\r\nup subsequent requests while the initial one is being completed. The\r\nsame algorithm is used to update the in-memory cached index.  Note that\r\nthese requests need not be protected by a lock because they are only\r\never executing on their own. See server.getChartList() for the\r\nimplementaion of this idea.\r\n\r\nA refactor was needed to separate the fetch from the diff calculation to\r\nallow separate calling of the network heavy operations. While doing so,\r\nwe also removed redundant calls to storage file list update.\r\n\r\nAlso made small low-hanging fruit style optimisations to index\r\nmanipulations.\r\n\r\nAdded request ID to all requests for better debugging. This will be\r\nvisible with the --debug flag. This was indispensable to diagnose\r\ncomplex concurrent processing.\r\n\r\nTo test the before and after state, we have added the use of the\r\nlocusti.io loadtesting engine. A simple README in the loadtesting/\r\ndirectory shows how to install locust (with pipenv) and loadtest\r\nchartmuseum. This will prove useful in the future.\r\n\r\nFixes #18","created_at":"2017-11-16T21:26:58Z","updated_at":"2017-11-19T23:04:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6126d9068f20e2138b782568d0897301c8e5533b","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19/commits","review_comments_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19/comments","review_comment_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/issues/19/comments","statuses_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/statuses/8a7e989c501d22bd59993d2d52f9542554eba462","head":{"label":"davidovich:fix-index-response-times-increasing-with-many-uploads","ref":"fix-index-response-times-increasing-with-many-uploads","sha":"8a7e989c501d22bd59993d2d52f9542554eba462","user":{"login":"davidovich","id":1440370,"avatar_url":"https://avatars0.githubusercontent.com/u/1440370?v=4","gravatar_id":"","url":"https://api.github.com/users/davidovich","html_url":"https://github.com/davidovich","followers_url":"https://api.github.com/users/davidovich/followers","following_url":"https://api.github.com/users/davidovich/following{/other_user}","gists_url":"https://api.github.com/users/davidovich/gists{/gist_id}","starred_url":"https://api.github.com/users/davidovich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidovich/subscriptions","organizations_url":"https://api.github.com/users/davidovich/orgs","repos_url":"https://api.github.com/users/davidovich/repos","events_url":"https://api.github.com/users/davidovich/events{/privacy}","received_events_url":"https://api.github.com/users/davidovich/received_events","type":"User","site_admin":false},"repo":{"id":109761669,"name":"chartmuseum","full_name":"davidovich/chartmuseum","owner":{"login":"davidovich","id":1440370,"avatar_url":"https://avatars0.githubusercontent.com/u/1440370?v=4","gravatar_id":"","url":"https://api.github.com/users/davidovich","html_url":"https://github.com/davidovich","followers_url":"https://api.github.com/users/davidovich/followers","following_url":"https://api.github.com/users/davidovich/following{/other_user}","gists_url":"https://api.github.com/users/davidovich/gists{/gist_id}","starred_url":"https://api.github.com/users/davidovich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidovich/subscriptions","organizations_url":"https://api.github.com/users/davidovich/orgs","repos_url":"https://api.github.com/users/davidovich/repos","events_url":"https://api.github.com/users/davidovich/events{/privacy}","received_events_url":"https://api.github.com/users/davidovich/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/davidovich/chartmuseum","description":"Helm Chart Repository with support for Amazon S3 and Google Cloud Storage","fork":true,"url":"https://api.github.com/repos/davidovich/chartmuseum","forks_url":"https://api.github.com/repos/davidovich/chartmuseum/forks","keys_url":"https://api.github.com/repos/davidovich/chartmuseum/keys{/key_id}","collaborators_url":"https://api.github.com/repos/davidovich/chartmuseum/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/davidovich/chartmuseum/teams","hooks_url":"https://api.github.com/repos/davidovich/chartmuseum/hooks","issue_events_url":"https://api.github.com/repos/davidovich/chartmuseum/issues/events{/number}","events_url":"https://api.github.com/repos/davidovich/chartmuseum/events","assignees_url":"https://api.github.com/repos/davidovich/chartmuseum/assignees{/user}","branches_url":"https://api.github.com/repos/davidovich/chartmuseum/branches{/branch}","tags_url":"https://api.github.com/repos/davidovich/chartmuseum/tags","blobs_url":"https://api.github.com/repos/davidovich/chartmuseum/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/davidovich/chartmuseum/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/davidovich/chartmuseum/git/refs{/sha}","trees_url":"https://api.github.com/repos/davidovich/chartmuseum/git/trees{/sha}","statuses_url":"https://api.github.com/repos/davidovich/chartmuseum/statuses/{sha}","languages_url":"https://api.github.com/repos/davidovich/chartmuseum/languages","stargazers_url":"https://api.github.com/repos/davidovich/chartmuseum/stargazers","contributors_url":"https://api.github.com/repos/davidovich/chartmuseum/contributors","subscribers_url":"https://api.github.com/repos/davidovich/chartmuseum/subscribers","subscription_url":"https://api.github.com/repos/davidovich/chartmuseum/subscription","commits_url":"https://api.github.com/repos/davidovich/chartmuseum/commits{/sha}","git_commits_url":"https://api.github.com/repos/davidovich/chartmuseum/git/commits{/sha}","comments_url":"https://api.github.com/repos/davidovich/chartmuseum/comments{/number}","issue_comment_url":"https://api.github.com/repos/davidovich/chartmuseum/issues/comments{/number}","contents_url":"https://api.github.com/repos/davidovich/chartmuseum/contents/{+path}","compare_url":"https://api.github.com/repos/davidovich/chartmuseum/compare/{base}...{head}","merges_url":"https://api.github.com/repos/davidovich/chartmuseum/merges","archive_url":"https://api.github.com/repos/davidovich/chartmuseum/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/davidovich/chartmuseum/downloads","issues_url":"https://api.github.com/repos/davidovich/chartmuseum/issues{/number}","pulls_url":"https://api.github.com/repos/davidovich/chartmuseum/pulls{/number}","milestones_url":"https://api.github.com/repos/davidovich/chartmuseum/milestones{/number}","notifications_url":"https://api.github.com/repos/davidovich/chartmuseum/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/davidovich/chartmuseum/labels{/name}","releases_url":"https://api.github.com/repos/davidovich/chartmuseum/releases{/id}","deployments_url":"https://api.github.com/repos/davidovich/chartmuseum/deployments","created_at":"2017-11-06T23:21:36Z","updated_at":"2017-11-06T23:21:38Z","pushed_at":"2017-11-19T02:11:21Z","git_url":"git://github.com/davidovich/chartmuseum.git","ssh_url":"git@github.com:davidovich/chartmuseum.git","clone_url":"https://github.com/davidovich/chartmuseum.git","svn_url":"https://github.com/davidovich/chartmuseum","homepage":null,"size":6570,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes-helm:master","ref":"master","sha":"52183fcab12edc538cf4443ab4a4b6af7cf6cfd0","user":{"login":"kubernetes-helm","id":29493517,"avatar_url":"https://avatars2.githubusercontent.com/u/29493517?v=4","gravatar_id":"","url":"https://api.github.com/users/kubernetes-helm","html_url":"https://github.com/kubernetes-helm","followers_url":"https://api.github.com/users/kubernetes-helm/followers","following_url":"https://api.github.com/users/kubernetes-helm/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes-helm/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes-helm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes-helm/subscriptions","organizations_url":"https://api.github.com/users/kubernetes-helm/orgs","repos_url":"https://api.github.com/users/kubernetes-helm/repos","events_url":"https://api.github.com/users/kubernetes-helm/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes-helm/received_events","type":"Organization","site_admin":false},"repo":{"id":104032019,"name":"chartmuseum","full_name":"kubernetes-helm/chartmuseum","owner":{"login":"kubernetes-helm","id":29493517,"avatar_url":"https://avatars2.githubusercontent.com/u/29493517?v=4","gravatar_id":"","url":"https://api.github.com/users/kubernetes-helm","html_url":"https://github.com/kubernetes-helm","followers_url":"https://api.github.com/users/kubernetes-helm/followers","following_url":"https://api.github.com/users/kubernetes-helm/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes-helm/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes-helm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes-helm/subscriptions","organizations_url":"https://api.github.com/users/kubernetes-helm/orgs","repos_url":"https://api.github.com/users/kubernetes-helm/repos","events_url":"https://api.github.com/users/kubernetes-helm/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes-helm/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes-helm/chartmuseum","description":"Helm Chart Repository with support for Amazon S3 and Google Cloud Storage","fork":false,"url":"https://api.github.com/repos/kubernetes-helm/chartmuseum","forks_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/forks","keys_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/teams","hooks_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/hooks","issue_events_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/events","assignees_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/tags","blobs_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/languages","stargazers_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/stargazers","contributors_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/contributors","subscribers_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/subscribers","subscription_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/subscription","commits_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/merges","archive_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/downloads","issues_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/releases{/id}","deployments_url":"https://api.github.com/repos/kubernetes-helm/chartmuseum/deployments","created_at":"2017-09-19T05:48:21Z","updated_at":"2017-11-17T13:26:14Z","pushed_at":"2017-11-19T02:11:22Z","git_url":"git://github.com/kubernetes-helm/chartmuseum.git","ssh_url":"git@github.com:kubernetes-helm/chartmuseum.git","clone_url":"https://github.com/kubernetes-helm/chartmuseum.git","svn_url":"https://github.com/kubernetes-helm/chartmuseum","homepage":null,"size":6528,"stargazers_count":121,"watchers_count":121,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"archived":false,"open_issues_count":2,"forks":8,"open_issues":2,"watchers":121,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19"},"html":{"href":"https://github.com/kubernetes-helm/chartmuseum/pull/19"},"issue":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/issues/19"},"comments":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/issues/19/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/pulls/19/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes-helm/chartmuseum/statuses/8a7e989c501d22bd59993d2d52f9542554eba462"}},"author_association":"NONE"}}
{"actor":{"display_login":"jdolitsky"}}