{"action":"created","comment":{"url":"https://api.github.com/repos/DataDog/datadog-agent/pulls/comments/150192403","pull_request_review_id":75705307,"id":150192403,"diff_hunk":"@@ -213,6 +214,51 @@ func (c *Config) IsTemplate() bool {\n \treturn false\n }\n \n+// IsJMX returns if the config is for a JMX check\n+func (c *Config) IsJMX() bool {\n+\treturn IsConfigJMX(c.String(), c.InitConfig)\n+}\n+\n+// AddMetrics adds metrics to a check configuration\n+func (c *Config) AddMetrics(metrics ConfigData) error {\n+\tvar rawInitConfig map[interface{}]interface{}\n+\terr := yaml.Unmarshal(c.InitConfig, &rawInitConfig)\n+\tif err != nil {\n+\t\treturn err\n+\t}\n+\n+\tvar rawMetricsConfig []interface{}\n+\terr = yaml.Unmarshal(metrics, &rawMetricsConfig)\n+\tif err != nil {\n+\t\treturn err\n+\t}\n+\n+\t// JMX fetch expects the metrics to be a part of init_config, under \"conf\"\n+\t// Preserve any metrics currently in init_config","path":"pkg/collector/check/check.go","position":32,"original_position":32,"commit_id":"3e7913a48c1c2946d660cab80bea8d38bb047ce1","original_commit_id":"3e7913a48c1c2946d660cab80bea8d38bb047ce1","user":{"login":"xvello","id":6241083,"avatar_url":"https://avatars2.githubusercontent.com/u/6241083?v=4","gravatar_id":"","url":"https://api.github.com/users/xvello","html_url":"https://github.com/xvello","followers_url":"https://api.github.com/users/xvello/followers","following_url":"https://api.github.com/users/xvello/following{/other_user}","gists_url":"https://api.github.com/users/xvello/gists{/gist_id}","starred_url":"https://api.github.com/users/xvello/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xvello/subscriptions","organizations_url":"https://api.github.com/users/xvello/orgs","repos_url":"https://api.github.com/users/xvello/repos","events_url":"https://api.github.com/users/xvello/events{/privacy}","received_events_url":"https://api.github.com/users/xvello/received_events","type":"User","site_admin":false},"body":"Is there a way to completely override the `.metrics` file from the template? Looks like we unconditionality merge them, but I think having the option of disabling the common metrics might be requested down the road.","created_at":"2017-11-10T09:50:47Z","updated_at":"2017-11-10T09:57:27Z","html_url":"https://github.com/DataDog/datadog-agent/pull/791#discussion_r150192403","pull_request_url":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/DataDog/datadog-agent/pulls/comments/150192403"},"html":{"href":"https://github.com/DataDog/datadog-agent/pull/791#discussion_r150192403"},"pull_request":{"href":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791"}}},"pull_request":{"url":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791","id":151738621,"html_url":"https://github.com/DataDog/datadog-agent/pull/791","diff_url":"https://github.com/DataDog/datadog-agent/pull/791.diff","patch_url":"https://github.com/DataDog/datadog-agent/pull/791.patch","issue_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/791","number":791,"state":"open","locked":false,"title":"Collect JMX check metrics for all check config providers","user":{"login":"ISauve","id":11351618,"avatar_url":"https://avatars3.githubusercontent.com/u/11351618?v=4","gravatar_id":"","url":"https://api.github.com/users/ISauve","html_url":"https://github.com/ISauve","followers_url":"https://api.github.com/users/ISauve/followers","following_url":"https://api.github.com/users/ISauve/following{/other_user}","gists_url":"https://api.github.com/users/ISauve/gists{/gist_id}","starred_url":"https://api.github.com/users/ISauve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ISauve/subscriptions","organizations_url":"https://api.github.com/users/ISauve/orgs","repos_url":"https://api.github.com/users/ISauve/repos","events_url":"https://api.github.com/users/ISauve/events{/privacy}","received_events_url":"https://api.github.com/users/ISauve/received_events","type":"User","site_admin":false},"body":"### What does this PR do?\r\n\r\nCurrently, JMX fetch checks only have 1 configuration file (the `<check>.yaml` file) which have a `conf` section configuring all the metrics collected by that check.\r\n\r\nThe problem is that when Autodiscovery uses check templates to configure a check, these check templates might not automatically have all the default metric configurations in them. For example, if a client uses a key/value store or Kubernetes Pod Annotations to configure their check, then it is up to them to add the metrics to their configuration file in the format required by that check provider. Some checks collect a lot of metrics by default (ie Kafka) so adding all these metrics can really clutter up their configuration file (as well as be tedious to configure)\r\n\r\nThis PR decouples the \"metrics collected\" section of a JMX check configuration file from the rest of the file - it replaces the one `<check>.yaml` file with two files: `<check>.yaml` and `<check>.yaml.metrics`. That way, no matter which config provider is used (that is, whether the main configuration file came from a `<check>.yaml` file, an auto_conf file, a key-value store, etc), we just add the metrics associated with that check to the configuration file before sending the file to JMX_fetch to run the check. This mimics the way non-JMX checks work in the sense that the metric collection would be separate from the check configuration.\r\n\r\nFor backwards compatibility, if users do add a `conf` section with metrics to collect in the `init_config` portion of their check configuration file, we will still read that & collect those metrics as usual.\r\n\r\n### Motivation\r\n\r\nA user was running Kafka on Kubernetes and used Pod Annotations to configure the Kafka check - which meant he add to add all the default metrics to his annotations file by hand, which ended up being around ~400 lines of JSON. After this, he asked Support to raise a feature request for us to find a fix for this.\r\n\r\n### Additional Notes\r\n\r\nAnything else we should know when reviewing?\r\n","created_at":"2017-11-09T19:16:39Z","updated_at":"2017-11-10T09:57:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6b852d598b2f0fd0d3520d087a9f1fae440e62ee","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791/commits","review_comments_url":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791/comments","review_comment_url":"https://api.github.com/repos/DataDog/datadog-agent/pulls/comments{/number}","comments_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/791/comments","statuses_url":"https://api.github.com/repos/DataDog/datadog-agent/statuses/3e7913a48c1c2946d660cab80bea8d38bb047ce1","head":{"label":"DataDog:isabelle/jmx","ref":"isabelle/jmx","sha":"3e7913a48c1c2946d660cab80bea8d38bb047ce1","user":{"login":"DataDog","id":365230,"avatar_url":"https://avatars1.githubusercontent.com/u/365230?v=4","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"repo":{"id":49970739,"name":"datadog-agent","full_name":"DataDog/datadog-agent","owner":{"login":"DataDog","id":365230,"avatar_url":"https://avatars1.githubusercontent.com/u/365230?v=4","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DataDog/datadog-agent","description":"Datadog Agent ","fork":false,"url":"https://api.github.com/repos/DataDog/datadog-agent","forks_url":"https://api.github.com/repos/DataDog/datadog-agent/forks","keys_url":"https://api.github.com/repos/DataDog/datadog-agent/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DataDog/datadog-agent/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DataDog/datadog-agent/teams","hooks_url":"https://api.github.com/repos/DataDog/datadog-agent/hooks","issue_events_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/events{/number}","events_url":"https://api.github.com/repos/DataDog/datadog-agent/events","assignees_url":"https://api.github.com/repos/DataDog/datadog-agent/assignees{/user}","branches_url":"https://api.github.com/repos/DataDog/datadog-agent/branches{/branch}","tags_url":"https://api.github.com/repos/DataDog/datadog-agent/tags","blobs_url":"https://api.github.com/repos/DataDog/datadog-agent/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DataDog/datadog-agent/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DataDog/datadog-agent/git/refs{/sha}","trees_url":"https://api.github.com/repos/DataDog/datadog-agent/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DataDog/datadog-agent/statuses/{sha}","languages_url":"https://api.github.com/repos/DataDog/datadog-agent/languages","stargazers_url":"https://api.github.com/repos/DataDog/datadog-agent/stargazers","contributors_url":"https://api.github.com/repos/DataDog/datadog-agent/contributors","subscribers_url":"https://api.github.com/repos/DataDog/datadog-agent/subscribers","subscription_url":"https://api.github.com/repos/DataDog/datadog-agent/subscription","commits_url":"https://api.github.com/repos/DataDog/datadog-agent/commits{/sha}","git_commits_url":"https://api.github.com/repos/DataDog/datadog-agent/git/commits{/sha}","comments_url":"https://api.github.com/repos/DataDog/datadog-agent/comments{/number}","issue_comment_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/comments{/number}","contents_url":"https://api.github.com/repos/DataDog/datadog-agent/contents/{+path}","compare_url":"https://api.github.com/repos/DataDog/datadog-agent/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DataDog/datadog-agent/merges","archive_url":"https://api.github.com/repos/DataDog/datadog-agent/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DataDog/datadog-agent/downloads","issues_url":"https://api.github.com/repos/DataDog/datadog-agent/issues{/number}","pulls_url":"https://api.github.com/repos/DataDog/datadog-agent/pulls{/number}","milestones_url":"https://api.github.com/repos/DataDog/datadog-agent/milestones{/number}","notifications_url":"https://api.github.com/repos/DataDog/datadog-agent/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DataDog/datadog-agent/labels{/name}","releases_url":"https://api.github.com/repos/DataDog/datadog-agent/releases{/id}","deployments_url":"https://api.github.com/repos/DataDog/datadog-agent/deployments","created_at":"2016-01-19T17:40:41Z","updated_at":"2017-11-09T19:26:37Z","pushed_at":"2017-11-10T09:44:40Z","git_url":"git://github.com/DataDog/datadog-agent.git","ssh_url":"git@github.com:DataDog/datadog-agent.git","clone_url":"https://github.com/DataDog/datadog-agent.git","svn_url":"https://github.com/DataDog/datadog-agent","homepage":"https://docs.datadoghq.com/","size":14816,"stargazers_count":32,"watchers_count":32,"language":"Go","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":16,"forks":3,"open_issues":16,"watchers":32,"default_branch":"master"}},"base":{"label":"DataDog:master","ref":"master","sha":"83b571651d707433272527aab0140b4cd176c204","user":{"login":"DataDog","id":365230,"avatar_url":"https://avatars1.githubusercontent.com/u/365230?v=4","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"repo":{"id":49970739,"name":"datadog-agent","full_name":"DataDog/datadog-agent","owner":{"login":"DataDog","id":365230,"avatar_url":"https://avatars1.githubusercontent.com/u/365230?v=4","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DataDog/datadog-agent","description":"Datadog Agent ","fork":false,"url":"https://api.github.com/repos/DataDog/datadog-agent","forks_url":"https://api.github.com/repos/DataDog/datadog-agent/forks","keys_url":"https://api.github.com/repos/DataDog/datadog-agent/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DataDog/datadog-agent/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DataDog/datadog-agent/teams","hooks_url":"https://api.github.com/repos/DataDog/datadog-agent/hooks","issue_events_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/events{/number}","events_url":"https://api.github.com/repos/DataDog/datadog-agent/events","assignees_url":"https://api.github.com/repos/DataDog/datadog-agent/assignees{/user}","branches_url":"https://api.github.com/repos/DataDog/datadog-agent/branches{/branch}","tags_url":"https://api.github.com/repos/DataDog/datadog-agent/tags","blobs_url":"https://api.github.com/repos/DataDog/datadog-agent/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DataDog/datadog-agent/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DataDog/datadog-agent/git/refs{/sha}","trees_url":"https://api.github.com/repos/DataDog/datadog-agent/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DataDog/datadog-agent/statuses/{sha}","languages_url":"https://api.github.com/repos/DataDog/datadog-agent/languages","stargazers_url":"https://api.github.com/repos/DataDog/datadog-agent/stargazers","contributors_url":"https://api.github.com/repos/DataDog/datadog-agent/contributors","subscribers_url":"https://api.github.com/repos/DataDog/datadog-agent/subscribers","subscription_url":"https://api.github.com/repos/DataDog/datadog-agent/subscription","commits_url":"https://api.github.com/repos/DataDog/datadog-agent/commits{/sha}","git_commits_url":"https://api.github.com/repos/DataDog/datadog-agent/git/commits{/sha}","comments_url":"https://api.github.com/repos/DataDog/datadog-agent/comments{/number}","issue_comment_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/comments{/number}","contents_url":"https://api.github.com/repos/DataDog/datadog-agent/contents/{+path}","compare_url":"https://api.github.com/repos/DataDog/datadog-agent/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DataDog/datadog-agent/merges","archive_url":"https://api.github.com/repos/DataDog/datadog-agent/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DataDog/datadog-agent/downloads","issues_url":"https://api.github.com/repos/DataDog/datadog-agent/issues{/number}","pulls_url":"https://api.github.com/repos/DataDog/datadog-agent/pulls{/number}","milestones_url":"https://api.github.com/repos/DataDog/datadog-agent/milestones{/number}","notifications_url":"https://api.github.com/repos/DataDog/datadog-agent/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DataDog/datadog-agent/labels{/name}","releases_url":"https://api.github.com/repos/DataDog/datadog-agent/releases{/id}","deployments_url":"https://api.github.com/repos/DataDog/datadog-agent/deployments","created_at":"2016-01-19T17:40:41Z","updated_at":"2017-11-09T19:26:37Z","pushed_at":"2017-11-10T09:44:40Z","git_url":"git://github.com/DataDog/datadog-agent.git","ssh_url":"git@github.com:DataDog/datadog-agent.git","clone_url":"https://github.com/DataDog/datadog-agent.git","svn_url":"https://github.com/DataDog/datadog-agent","homepage":"https://docs.datadoghq.com/","size":14816,"stargazers_count":32,"watchers_count":32,"language":"Go","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":16,"forks":3,"open_issues":16,"watchers":32,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791"},"html":{"href":"https://github.com/DataDog/datadog-agent/pull/791"},"issue":{"href":"https://api.github.com/repos/DataDog/datadog-agent/issues/791"},"comments":{"href":"https://api.github.com/repos/DataDog/datadog-agent/issues/791/comments"},"review_comments":{"href":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791/comments"},"review_comment":{"href":"https://api.github.com/repos/DataDog/datadog-agent/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/DataDog/datadog-agent/pulls/791/commits"},"statuses":{"href":"https://api.github.com/repos/DataDog/datadog-agent/statuses/3e7913a48c1c2946d660cab80bea8d38bb047ce1"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"xvello"}}