{"action":"created","comment":{"url":"https://api.github.com/repos/DataDog/integrations-core/pulls/comments/118068368","pull_request_review_id":39827237,"id":118068368,"diff_hunk":"@@ -68,61 +205,99 @@ def check(self, instance):\n                 zk_conn.close()\n             except Exception:\n                 self.log.exception('Error cleaning up Zookeeper connection')\n+        return zk_consumer_offsets\n \n-        # Connect to Kafka\n-        kafka_conn = KafkaClient(kafka_host_ports, timeout=self.kafka_timeout)\n+    def check(self, instance):\n+        # For calculating lag, we have to fetch offsets from both kafka and\n+        # zookeeper. There's a potential race condition because whichever one we\n+        # check first may be outdated by the time we check the other. Better to\n+        # check consumer offset before checking broker offset because worst case\n+        # is that overstates consumer lag a little. Doing it the other way can\n+        # understate consumer lag to the point of having negative consumer lag,","path":"kafka_consumer/check.py","position":231,"original_position":231,"commit_id":"f5b85c93d0edf4d98ab9595110cab86f31393e5e","original_commit_id":"f5b85c93d0edf4d98ab9595110cab86f31393e5e","user":{"login":"jeffwidman","id":483314,"avatar_url":"https://avatars1.githubusercontent.com/u/483314?v=3","gravatar_id":"","url":"https://api.github.com/users/jeffwidman","html_url":"https://github.com/jeffwidman","followers_url":"https://api.github.com/users/jeffwidman/followers","following_url":"https://api.github.com/users/jeffwidman/following{/other_user}","gists_url":"https://api.github.com/users/jeffwidman/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffwidman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffwidman/subscriptions","organizations_url":"https://api.github.com/users/jeffwidman/orgs","repos_url":"https://api.github.com/users/jeffwidman/repos","events_url":"https://api.github.com/users/jeffwidman/events{/privacy}","received_events_url":"https://api.github.com/users/jeffwidman/received_events","type":"User","site_admin":false},"body":"\"understate\" --> \"may understate\"","created_at":"2017-05-23T18:21:46Z","updated_at":"2017-05-23T18:21:46Z","html_url":"https://github.com/DataDog/integrations-core/pull/271#discussion_r118068368","pull_request_url":"https://api.github.com/repos/DataDog/integrations-core/pulls/271","_links":{"self":{"href":"https://api.github.com/repos/DataDog/integrations-core/pulls/comments/118068368"},"html":{"href":"https://github.com/DataDog/integrations-core/pull/271#discussion_r118068368"},"pull_request":{"href":"https://api.github.com/repos/DataDog/integrations-core/pulls/271"}}},"pull_request":{"url":"https://api.github.com/repos/DataDog/integrations-core/pulls/271","id":110729532,"html_url":"https://github.com/DataDog/integrations-core/pull/271","diff_url":"https://github.com/DataDog/integrations-core/pull/271.diff","patch_url":"https://github.com/DataDog/integrations-core/pull/271.patch","issue_url":"https://api.github.com/repos/DataDog/integrations-core/issues/271","number":271,"state":"open","locked":false,"title":"[kafka_consumer] Make args optional in kafka_consumer.py","user":{"login":"jeffwidman","id":483314,"avatar_url":"https://avatars1.githubusercontent.com/u/483314?v=3","gravatar_id":"","url":"https://api.github.com/users/jeffwidman","html_url":"https://github.com/jeffwidman","followers_url":"https://api.github.com/users/jeffwidman/followers","following_url":"https://api.github.com/users/jeffwidman/following{/other_user}","gists_url":"https://api.github.com/users/jeffwidman/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffwidman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffwidman/subscriptions","organizations_url":"https://api.github.com/users/jeffwidman/orgs","repos_url":"https://api.github.com/users/jeffwidman/repos","events_url":"https://api.github.com/users/jeffwidman/events{/privacy}","received_events_url":"https://api.github.com/users/jeffwidman/received_events","type":"User","site_admin":false},"body":"### What does this PR do?\r\n\r\nThe consumer_groups, topics, and partitions arguments are each now optional. Any omitted values will be fetched from Zookeeper. If a value is omitted, the parent value must still be it's expected type (typically a dict). The `consumer_groups` argument can only be omitted if the`monitor_unlisted_consumer_groups` flag is set to True. The topics/partitions do not require this flag as I thought that specifying it at that lower level is redundant. Most users will specify a few consumer groups and leave it at that. \r\n\r\nModern Kafka consumers store their offset in Kafka rather than Zookeeper, but this PR does not add support for this. Instead, I will be submitting a second kafka consumer offset check that will support offsets stored in Kafka. Originally I tried implementing that as part of this check, but the YAML interface got a bit clunky because it'd require a user to specify whether to fetch the offset from Zookeeper or Kafka. And if a user didn't specify a consumer group but instead set `monitor_unlisted_consumer_groups: True`, then it was totally ambiguous whether to check Kafka or Zookeeper. \r\n\r\nAdditionally, since modern Kafka consumers have no dependency on Zookeeper, it seems like the better thing to do is keep them as two separate checks--one for zookeeper offsets and one for kafka offsets. This way the new check doesn't even concern itself with talking to Zookeeper at all, and it can do a more optimized handling of the Kafka connection since it's used for both broker and consumer offsets.\r\n\r\n","created_at":"2017-03-14T22:14:58Z","updated_at":"2017-05-23T18:21:46Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3a5ba42affa3307f13e5c3782f99734fbc6b660a","assignee":null,"assignees":[],"requested_reviewers":[{"login":"remh","id":1311477,"avatar_url":"https://avatars1.githubusercontent.com/u/1311477?v=3","gravatar_id":"","url":"https://api.github.com/users/remh","html_url":"https://github.com/remh","followers_url":"https://api.github.com/users/remh/followers","following_url":"https://api.github.com/users/remh/following{/other_user}","gists_url":"https://api.github.com/users/remh/gists{/gist_id}","starred_url":"https://api.github.com/users/remh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/remh/subscriptions","organizations_url":"https://api.github.com/users/remh/orgs","repos_url":"https://api.github.com/users/remh/repos","events_url":"https://api.github.com/users/remh/events{/privacy}","received_events_url":"https://api.github.com/users/remh/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/DataDog/integrations-core/milestones/8","html_url":"https://github.com/DataDog/integrations-core/milestone/8","labels_url":"https://api.github.com/repos/DataDog/integrations-core/milestones/8/labels","id":2453400,"number":8,"title":"5.14","description":"","creator":{"login":"masci","id":7241,"avatar_url":"https://avatars0.githubusercontent.com/u/7241?v=3","gravatar_id":"","url":"https://api.github.com/users/masci","html_url":"https://github.com/masci","followers_url":"https://api.github.com/users/masci/followers","following_url":"https://api.github.com/users/masci/following{/other_user}","gists_url":"https://api.github.com/users/masci/gists{/gist_id}","starred_url":"https://api.github.com/users/masci/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/masci/subscriptions","organizations_url":"https://api.github.com/users/masci/orgs","repos_url":"https://api.github.com/users/masci/repos","events_url":"https://api.github.com/users/masci/events{/privacy}","received_events_url":"https://api.github.com/users/masci/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":19,"state":"open","created_at":"2017-04-12T17:11:57Z","updated_at":"2017-05-23T14:35:50Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/DataDog/integrations-core/pulls/271/commits","review_comments_url":"https://api.github.com/repos/DataDog/integrations-core/pulls/271/comments","review_comment_url":"https://api.github.com/repos/DataDog/integrations-core/pulls/comments{/number}","comments_url":"https://api.github.com/repos/DataDog/integrations-core/issues/271/comments","statuses_url":"https://api.github.com/repos/DataDog/integrations-core/statuses/f5b85c93d0edf4d98ab9595110cab86f31393e5e","head":{"label":"jeffwidman:fix-kafka-consumer","ref":"fix-kafka-consumer","sha":"f5b85c93d0edf4d98ab9595110cab86f31393e5e","user":{"login":"jeffwidman","id":483314,"avatar_url":"https://avatars1.githubusercontent.com/u/483314?v=3","gravatar_id":"","url":"https://api.github.com/users/jeffwidman","html_url":"https://github.com/jeffwidman","followers_url":"https://api.github.com/users/jeffwidman/followers","following_url":"https://api.github.com/users/jeffwidman/following{/other_user}","gists_url":"https://api.github.com/users/jeffwidman/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffwidman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffwidman/subscriptions","organizations_url":"https://api.github.com/users/jeffwidman/orgs","repos_url":"https://api.github.com/users/jeffwidman/repos","events_url":"https://api.github.com/users/jeffwidman/events{/privacy}","received_events_url":"https://api.github.com/users/jeffwidman/received_events","type":"User","site_admin":false},"repo":{"id":82793929,"name":"integrations-core","full_name":"jeffwidman/integrations-core","owner":{"login":"jeffwidman","id":483314,"avatar_url":"https://avatars1.githubusercontent.com/u/483314?v=3","gravatar_id":"","url":"https://api.github.com/users/jeffwidman","html_url":"https://github.com/jeffwidman","followers_url":"https://api.github.com/users/jeffwidman/followers","following_url":"https://api.github.com/users/jeffwidman/following{/other_user}","gists_url":"https://api.github.com/users/jeffwidman/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffwidman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffwidman/subscriptions","organizations_url":"https://api.github.com/users/jeffwidman/orgs","repos_url":"https://api.github.com/users/jeffwidman/repos","events_url":"https://api.github.com/users/jeffwidman/events{/privacy}","received_events_url":"https://api.github.com/users/jeffwidman/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jeffwidman/integrations-core","description":"Core integrations of the datadog agent","fork":true,"url":"https://api.github.com/repos/jeffwidman/integrations-core","forks_url":"https://api.github.com/repos/jeffwidman/integrations-core/forks","keys_url":"https://api.github.com/repos/jeffwidman/integrations-core/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jeffwidman/integrations-core/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jeffwidman/integrations-core/teams","hooks_url":"https://api.github.com/repos/jeffwidman/integrations-core/hooks","issue_events_url":"https://api.github.com/repos/jeffwidman/integrations-core/issues/events{/number}","events_url":"https://api.github.com/repos/jeffwidman/integrations-core/events","assignees_url":"https://api.github.com/repos/jeffwidman/integrations-core/assignees{/user}","branches_url":"https://api.github.com/repos/jeffwidman/integrations-core/branches{/branch}","tags_url":"https://api.github.com/repos/jeffwidman/integrations-core/tags","blobs_url":"https://api.github.com/repos/jeffwidman/integrations-core/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jeffwidman/integrations-core/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jeffwidman/integrations-core/git/refs{/sha}","trees_url":"https://api.github.com/repos/jeffwidman/integrations-core/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jeffwidman/integrations-core/statuses/{sha}","languages_url":"https://api.github.com/repos/jeffwidman/integrations-core/languages","stargazers_url":"https://api.github.com/repos/jeffwidman/integrations-core/stargazers","contributors_url":"https://api.github.com/repos/jeffwidman/integrations-core/contributors","subscribers_url":"https://api.github.com/repos/jeffwidman/integrations-core/subscribers","subscription_url":"https://api.github.com/repos/jeffwidman/integrations-core/subscription","commits_url":"https://api.github.com/repos/jeffwidman/integrations-core/commits{/sha}","git_commits_url":"https://api.github.com/repos/jeffwidman/integrations-core/git/commits{/sha}","comments_url":"https://api.github.com/repos/jeffwidman/integrations-core/comments{/number}","issue_comment_url":"https://api.github.com/repos/jeffwidman/integrations-core/issues/comments{/number}","contents_url":"https://api.github.com/repos/jeffwidman/integrations-core/contents/{+path}","compare_url":"https://api.github.com/repos/jeffwidman/integrations-core/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jeffwidman/integrations-core/merges","archive_url":"https://api.github.com/repos/jeffwidman/integrations-core/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jeffwidman/integrations-core/downloads","issues_url":"https://api.github.com/repos/jeffwidman/integrations-core/issues{/number}","pulls_url":"https://api.github.com/repos/jeffwidman/integrations-core/pulls{/number}","milestones_url":"https://api.github.com/repos/jeffwidman/integrations-core/milestones{/number}","notifications_url":"https://api.github.com/repos/jeffwidman/integrations-core/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jeffwidman/integrations-core/labels{/name}","releases_url":"https://api.github.com/repos/jeffwidman/integrations-core/releases{/id}","deployments_url":"https://api.github.com/repos/jeffwidman/integrations-core/deployments","created_at":"2017-02-22T10:53:15Z","updated_at":"2017-02-22T10:53:18Z","pushed_at":"2017-05-20T22:35:48Z","git_url":"git://github.com/jeffwidman/integrations-core.git","ssh_url":"git@github.com:jeffwidman/integrations-core.git","clone_url":"https://github.com/jeffwidman/integrations-core.git","svn_url":"https://github.com/jeffwidman/integrations-core","homepage":null,"size":2905,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"DataDog:master","ref":"master","sha":"0a1117b0ca1a07aa71d017027a1248e1877d6356","user":{"login":"DataDog","id":365230,"avatar_url":"https://avatars2.githubusercontent.com/u/365230?v=3","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"repo":{"id":47203045,"name":"integrations-core","full_name":"DataDog/integrations-core","owner":{"login":"DataDog","id":365230,"avatar_url":"https://avatars2.githubusercontent.com/u/365230?v=3","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DataDog/integrations-core","description":"Core integrations of the datadog agent","fork":false,"url":"https://api.github.com/repos/DataDog/integrations-core","forks_url":"https://api.github.com/repos/DataDog/integrations-core/forks","keys_url":"https://api.github.com/repos/DataDog/integrations-core/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DataDog/integrations-core/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DataDog/integrations-core/teams","hooks_url":"https://api.github.com/repos/DataDog/integrations-core/hooks","issue_events_url":"https://api.github.com/repos/DataDog/integrations-core/issues/events{/number}","events_url":"https://api.github.com/repos/DataDog/integrations-core/events","assignees_url":"https://api.github.com/repos/DataDog/integrations-core/assignees{/user}","branches_url":"https://api.github.com/repos/DataDog/integrations-core/branches{/branch}","tags_url":"https://api.github.com/repos/DataDog/integrations-core/tags","blobs_url":"https://api.github.com/repos/DataDog/integrations-core/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DataDog/integrations-core/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DataDog/integrations-core/git/refs{/sha}","trees_url":"https://api.github.com/repos/DataDog/integrations-core/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DataDog/integrations-core/statuses/{sha}","languages_url":"https://api.github.com/repos/DataDog/integrations-core/languages","stargazers_url":"https://api.github.com/repos/DataDog/integrations-core/stargazers","contributors_url":"https://api.github.com/repos/DataDog/integrations-core/contributors","subscribers_url":"https://api.github.com/repos/DataDog/integrations-core/subscribers","subscription_url":"https://api.github.com/repos/DataDog/integrations-core/subscription","commits_url":"https://api.github.com/repos/DataDog/integrations-core/commits{/sha}","git_commits_url":"https://api.github.com/repos/DataDog/integrations-core/git/commits{/sha}","comments_url":"https://api.github.com/repos/DataDog/integrations-core/comments{/number}","issue_comment_url":"https://api.github.com/repos/DataDog/integrations-core/issues/comments{/number}","contents_url":"https://api.github.com/repos/DataDog/integrations-core/contents/{+path}","compare_url":"https://api.github.com/repos/DataDog/integrations-core/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DataDog/integrations-core/merges","archive_url":"https://api.github.com/repos/DataDog/integrations-core/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DataDog/integrations-core/downloads","issues_url":"https://api.github.com/repos/DataDog/integrations-core/issues{/number}","pulls_url":"https://api.github.com/repos/DataDog/integrations-core/pulls{/number}","milestones_url":"https://api.github.com/repos/DataDog/integrations-core/milestones{/number}","notifications_url":"https://api.github.com/repos/DataDog/integrations-core/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DataDog/integrations-core/labels{/name}","releases_url":"https://api.github.com/repos/DataDog/integrations-core/releases{/id}","deployments_url":"https://api.github.com/repos/DataDog/integrations-core/deployments","created_at":"2015-12-01T16:41:45Z","updated_at":"2017-05-15T23:37:41Z","pushed_at":"2017-05-23T17:32:31Z","git_url":"git://github.com/DataDog/integrations-core.git","ssh_url":"git@github.com:DataDog/integrations-core.git","clone_url":"https://github.com/DataDog/integrations-core.git","svn_url":"https://github.com/DataDog/integrations-core","homepage":null,"size":3461,"stargazers_count":22,"watchers_count":22,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":53,"mirror_url":null,"open_issues_count":48,"forks":53,"open_issues":48,"watchers":22,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/DataDog/integrations-core/pulls/271"},"html":{"href":"https://github.com/DataDog/integrations-core/pull/271"},"issue":{"href":"https://api.github.com/repos/DataDog/integrations-core/issues/271"},"comments":{"href":"https://api.github.com/repos/DataDog/integrations-core/issues/271/comments"},"review_comments":{"href":"https://api.github.com/repos/DataDog/integrations-core/pulls/271/comments"},"review_comment":{"href":"https://api.github.com/repos/DataDog/integrations-core/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/DataDog/integrations-core/pulls/271/commits"},"statuses":{"href":"https://api.github.com/repos/DataDog/integrations-core/statuses/f5b85c93d0edf4d98ab9595110cab86f31393e5e"}}}}
{"actor":{"display_login":"jeffwidman"}}