{"action":"created","comment":{"url":"https://api.github.com/repos/resque/resque/pulls/comments/155765999","pull_request_review_id":82141027,"id":155765999,"diff_hunk":"@@ -275,6 +276,10 @@ def all_heartbeats\n         @redis.hgetall(HEARTBEAT_KEY)\n       end\n \n+      def acquire_pruning_dead_worker_lock(worker, expiry)\n+        @redis.set(redis_key_for_worker_pruning, worker.to_s, :ex => expiry, :nx => true)","path":"lib/resque/data_store.rb","position":13,"original_position":13,"commit_id":"8f07bfd223ef276acd395fc0358c0f94c4e90494","original_commit_id":"8f07bfd223ef276acd395fc0358c0f94c4e90494","user":{"login":"fw42","id":2072686,"avatar_url":"https://avatars1.githubusercontent.com/u/2072686?v=4","gravatar_id":"","url":"https://api.github.com/users/fw42","html_url":"https://github.com/fw42","followers_url":"https://api.github.com/users/fw42/followers","following_url":"https://api.github.com/users/fw42/following{/other_user}","gists_url":"https://api.github.com/users/fw42/gists{/gist_id}","starred_url":"https://api.github.com/users/fw42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fw42/subscriptions","organizations_url":"https://api.github.com/users/fw42/orgs","repos_url":"https://api.github.com/users/fw42/repos","events_url":"https://api.github.com/users/fw42/events{/privacy}","received_events_url":"https://api.github.com/users/fw42/received_events","type":"User","site_admin":false},"body":"Yea good point. The Redis `INFO` command returns the server version. But I'm not sure how expensive that command is (probably much less expensive than doing the entire pruning though, but the concern remains).","created_at":"2017-12-08T12:22:45Z","updated_at":"2017-12-08T12:22:45Z","html_url":"https://github.com/resque/resque/pull/1594#discussion_r155765999","pull_request_url":"https://api.github.com/repos/resque/resque/pulls/1594","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/resque/resque/pulls/comments/155765999"},"html":{"href":"https://github.com/resque/resque/pull/1594#discussion_r155765999"},"pull_request":{"href":"https://api.github.com/repos/resque/resque/pulls/1594"}},"in_reply_to_id":155745296},"pull_request":{"url":"https://api.github.com/repos/resque/resque/pulls/1594","id":157005600,"html_url":"https://github.com/resque/resque/pull/1594","diff_url":"https://github.com/resque/resque/pull/1594.diff","patch_url":"https://github.com/resque/resque/pull/1594.patch","issue_url":"https://api.github.com/repos/resque/resque/issues/1594","number":1594,"state":"open","locked":false,"title":"Prevent thundering herd when pruning dead workers","user":{"login":"fw42","id":2072686,"avatar_url":"https://avatars1.githubusercontent.com/u/2072686?v=4","gravatar_id":"","url":"https://api.github.com/users/fw42","html_url":"https://github.com/fw42","followers_url":"https://api.github.com/users/fw42/followers","following_url":"https://api.github.com/users/fw42/following{/other_user}","gists_url":"https://api.github.com/users/fw42/gists{/gist_id}","starred_url":"https://api.github.com/users/fw42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fw42/subscriptions","organizations_url":"https://api.github.com/users/fw42/orgs","repos_url":"https://api.github.com/users/fw42/repos","events_url":"https://api.github.com/users/fw42/events{/privacy}","received_events_url":"https://api.github.com/users/fw42/received_events","type":"User","site_admin":false},"body":"We've recently had a small service disruption where a large number of workers died and were dead for longer than `Resque.prune_interval`. Once the problem was fixed, all of our (thousands of) workers were able to boot again around the same time, meaning that all of them tried to prune all of the dead old workers at once (see [Thundering herd problem](https://en.wikipedia.org/wiki/Thundering_herd_problem)), therefore causing a huge load spike (pruning workers is a somewhat expensive operation, involving multiple server roundtrips to Redis).\r\n\r\nThis PR fixes that by making it so that only one worker can be pruning and all subsequent attempts will be skipped for the next `Resque.prune_interval` seconds.\r\n\r\n@es @pushrax @xldenis @hkdsun @sirupsen\r\ncc @chrisccerami ","created_at":"2017-12-07T14:17:56Z","updated_at":"2017-12-08T12:22:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c54c9180d2b54601c9d88abb113b00b1cbb9c7b1","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/resque/resque/pulls/1594/commits","review_comments_url":"https://api.github.com/repos/resque/resque/pulls/1594/comments","review_comment_url":"https://api.github.com/repos/resque/resque/pulls/comments{/number}","comments_url":"https://api.github.com/repos/resque/resque/issues/1594/comments","statuses_url":"https://api.github.com/repos/resque/resque/statuses/8f07bfd223ef276acd395fc0358c0f94c4e90494","head":{"label":"resque:thundering-herd-pruning","ref":"thundering-herd-pruning","sha":"8f07bfd223ef276acd395fc0358c0f94c4e90494","user":{"login":"resque","id":3876747,"avatar_url":"https://avatars0.githubusercontent.com/u/3876747?v=4","gravatar_id":"","url":"https://api.github.com/users/resque","html_url":"https://github.com/resque","followers_url":"https://api.github.com/users/resque/followers","following_url":"https://api.github.com/users/resque/following{/other_user}","gists_url":"https://api.github.com/users/resque/gists{/gist_id}","starred_url":"https://api.github.com/users/resque/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/resque/subscriptions","organizations_url":"https://api.github.com/users/resque/orgs","repos_url":"https://api.github.com/users/resque/repos","events_url":"https://api.github.com/users/resque/events{/privacy}","received_events_url":"https://api.github.com/users/resque/received_events","type":"Organization","site_admin":false},"repo":{"id":275604,"name":"resque","full_name":"resque/resque","owner":{"login":"resque","id":3876747,"avatar_url":"https://avatars0.githubusercontent.com/u/3876747?v=4","gravatar_id":"","url":"https://api.github.com/users/resque","html_url":"https://github.com/resque","followers_url":"https://api.github.com/users/resque/followers","following_url":"https://api.github.com/users/resque/following{/other_user}","gists_url":"https://api.github.com/users/resque/gists{/gist_id}","starred_url":"https://api.github.com/users/resque/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/resque/subscriptions","organizations_url":"https://api.github.com/users/resque/orgs","repos_url":"https://api.github.com/users/resque/repos","events_url":"https://api.github.com/users/resque/events{/privacy}","received_events_url":"https://api.github.com/users/resque/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/resque/resque","description":"Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.","fork":false,"url":"https://api.github.com/repos/resque/resque","forks_url":"https://api.github.com/repos/resque/resque/forks","keys_url":"https://api.github.com/repos/resque/resque/keys{/key_id}","collaborators_url":"https://api.github.com/repos/resque/resque/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/resque/resque/teams","hooks_url":"https://api.github.com/repos/resque/resque/hooks","issue_events_url":"https://api.github.com/repos/resque/resque/issues/events{/number}","events_url":"https://api.github.com/repos/resque/resque/events","assignees_url":"https://api.github.com/repos/resque/resque/assignees{/user}","branches_url":"https://api.github.com/repos/resque/resque/branches{/branch}","tags_url":"https://api.github.com/repos/resque/resque/tags","blobs_url":"https://api.github.com/repos/resque/resque/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/resque/resque/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/resque/resque/git/refs{/sha}","trees_url":"https://api.github.com/repos/resque/resque/git/trees{/sha}","statuses_url":"https://api.github.com/repos/resque/resque/statuses/{sha}","languages_url":"https://api.github.com/repos/resque/resque/languages","stargazers_url":"https://api.github.com/repos/resque/resque/stargazers","contributors_url":"https://api.github.com/repos/resque/resque/contributors","subscribers_url":"https://api.github.com/repos/resque/resque/subscribers","subscription_url":"https://api.github.com/repos/resque/resque/subscription","commits_url":"https://api.github.com/repos/resque/resque/commits{/sha}","git_commits_url":"https://api.github.com/repos/resque/resque/git/commits{/sha}","comments_url":"https://api.github.com/repos/resque/resque/comments{/number}","issue_comment_url":"https://api.github.com/repos/resque/resque/issues/comments{/number}","contents_url":"https://api.github.com/repos/resque/resque/contents/{+path}","compare_url":"https://api.github.com/repos/resque/resque/compare/{base}...{head}","merges_url":"https://api.github.com/repos/resque/resque/merges","archive_url":"https://api.github.com/repos/resque/resque/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/resque/resque/downloads","issues_url":"https://api.github.com/repos/resque/resque/issues{/number}","pulls_url":"https://api.github.com/repos/resque/resque/pulls{/number}","milestones_url":"https://api.github.com/repos/resque/resque/milestones{/number}","notifications_url":"https://api.github.com/repos/resque/resque/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/resque/resque/labels{/name}","releases_url":"https://api.github.com/repos/resque/resque/releases{/id}","deployments_url":"https://api.github.com/repos/resque/resque/deployments","created_at":"2009-08-12T00:27:35Z","updated_at":"2017-12-08T01:49:06Z","pushed_at":"2017-12-07T14:19:09Z","git_url":"git://github.com/resque/resque.git","ssh_url":"git@github.com:resque/resque.git","clone_url":"https://github.com/resque/resque.git","svn_url":"https://github.com/resque/resque","homepage":"http://resque.github.io/","size":2313,"stargazers_count":8040,"watchers_count":8040,"language":"Ruby","has_issues":true,"has_projects":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":1492,"mirror_url":null,"archived":false,"open_issues_count":74,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":1492,"open_issues":74,"watchers":8040,"default_branch":"master"}},"base":{"label":"resque:master","ref":"master","sha":"f54466f9d80505bdfcc1044e59ad7d52ead371b7","user":{"login":"resque","id":3876747,"avatar_url":"https://avatars0.githubusercontent.com/u/3876747?v=4","gravatar_id":"","url":"https://api.github.com/users/resque","html_url":"https://github.com/resque","followers_url":"https://api.github.com/users/resque/followers","following_url":"https://api.github.com/users/resque/following{/other_user}","gists_url":"https://api.github.com/users/resque/gists{/gist_id}","starred_url":"https://api.github.com/users/resque/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/resque/subscriptions","organizations_url":"https://api.github.com/users/resque/orgs","repos_url":"https://api.github.com/users/resque/repos","events_url":"https://api.github.com/users/resque/events{/privacy}","received_events_url":"https://api.github.com/users/resque/received_events","type":"Organization","site_admin":false},"repo":{"id":275604,"name":"resque","full_name":"resque/resque","owner":{"login":"resque","id":3876747,"avatar_url":"https://avatars0.githubusercontent.com/u/3876747?v=4","gravatar_id":"","url":"https://api.github.com/users/resque","html_url":"https://github.com/resque","followers_url":"https://api.github.com/users/resque/followers","following_url":"https://api.github.com/users/resque/following{/other_user}","gists_url":"https://api.github.com/users/resque/gists{/gist_id}","starred_url":"https://api.github.com/users/resque/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/resque/subscriptions","organizations_url":"https://api.github.com/users/resque/orgs","repos_url":"https://api.github.com/users/resque/repos","events_url":"https://api.github.com/users/resque/events{/privacy}","received_events_url":"https://api.github.com/users/resque/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/resque/resque","description":"Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.","fork":false,"url":"https://api.github.com/repos/resque/resque","forks_url":"https://api.github.com/repos/resque/resque/forks","keys_url":"https://api.github.com/repos/resque/resque/keys{/key_id}","collaborators_url":"https://api.github.com/repos/resque/resque/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/resque/resque/teams","hooks_url":"https://api.github.com/repos/resque/resque/hooks","issue_events_url":"https://api.github.com/repos/resque/resque/issues/events{/number}","events_url":"https://api.github.com/repos/resque/resque/events","assignees_url":"https://api.github.com/repos/resque/resque/assignees{/user}","branches_url":"https://api.github.com/repos/resque/resque/branches{/branch}","tags_url":"https://api.github.com/repos/resque/resque/tags","blobs_url":"https://api.github.com/repos/resque/resque/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/resque/resque/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/resque/resque/git/refs{/sha}","trees_url":"https://api.github.com/repos/resque/resque/git/trees{/sha}","statuses_url":"https://api.github.com/repos/resque/resque/statuses/{sha}","languages_url":"https://api.github.com/repos/resque/resque/languages","stargazers_url":"https://api.github.com/repos/resque/resque/stargazers","contributors_url":"https://api.github.com/repos/resque/resque/contributors","subscribers_url":"https://api.github.com/repos/resque/resque/subscribers","subscription_url":"https://api.github.com/repos/resque/resque/subscription","commits_url":"https://api.github.com/repos/resque/resque/commits{/sha}","git_commits_url":"https://api.github.com/repos/resque/resque/git/commits{/sha}","comments_url":"https://api.github.com/repos/resque/resque/comments{/number}","issue_comment_url":"https://api.github.com/repos/resque/resque/issues/comments{/number}","contents_url":"https://api.github.com/repos/resque/resque/contents/{+path}","compare_url":"https://api.github.com/repos/resque/resque/compare/{base}...{head}","merges_url":"https://api.github.com/repos/resque/resque/merges","archive_url":"https://api.github.com/repos/resque/resque/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/resque/resque/downloads","issues_url":"https://api.github.com/repos/resque/resque/issues{/number}","pulls_url":"https://api.github.com/repos/resque/resque/pulls{/number}","milestones_url":"https://api.github.com/repos/resque/resque/milestones{/number}","notifications_url":"https://api.github.com/repos/resque/resque/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/resque/resque/labels{/name}","releases_url":"https://api.github.com/repos/resque/resque/releases{/id}","deployments_url":"https://api.github.com/repos/resque/resque/deployments","created_at":"2009-08-12T00:27:35Z","updated_at":"2017-12-08T01:49:06Z","pushed_at":"2017-12-07T14:19:09Z","git_url":"git://github.com/resque/resque.git","ssh_url":"git@github.com:resque/resque.git","clone_url":"https://github.com/resque/resque.git","svn_url":"https://github.com/resque/resque","homepage":"http://resque.github.io/","size":2313,"stargazers_count":8040,"watchers_count":8040,"language":"Ruby","has_issues":true,"has_projects":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":1492,"mirror_url":null,"archived":false,"open_issues_count":74,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":1492,"open_issues":74,"watchers":8040,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/resque/resque/pulls/1594"},"html":{"href":"https://github.com/resque/resque/pull/1594"},"issue":{"href":"https://api.github.com/repos/resque/resque/issues/1594"},"comments":{"href":"https://api.github.com/repos/resque/resque/issues/1594/comments"},"review_comments":{"href":"https://api.github.com/repos/resque/resque/pulls/1594/comments"},"review_comment":{"href":"https://api.github.com/repos/resque/resque/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/resque/resque/pulls/1594/commits"},"statuses":{"href":"https://api.github.com/repos/resque/resque/statuses/8f07bfd223ef276acd395fc0358c0f94c4e90494"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"fw42"}}