{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/3259","repository_url":"https://api.github.com/repos/hashicorp/consul","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/3259/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/3259/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/3259/events","html_url":"https://github.com/hashicorp/consul/issues/3259","id":242147587,"number":3259,"title":"Healthchecks resynced every ~2mins, causes consul index to grow","user":{"login":"shays10","id":610830,"avatar_url":"https://avatars2.githubusercontent.com/u/610830?v=4","gravatar_id":"","url":"https://api.github.com/users/shays10","html_url":"https://github.com/shays10","followers_url":"https://api.github.com/users/shays10/followers","following_url":"https://api.github.com/users/shays10/following{/other_user}","gists_url":"https://api.github.com/users/shays10/gists{/gist_id}","starred_url":"https://api.github.com/users/shays10/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shays10/subscriptions","organizations_url":"https://api.github.com/users/shays10/orgs","repos_url":"https://api.github.com/users/shays10/repos","events_url":"https://api.github.com/users/shays10/events{/privacy}","received_events_url":"https://api.github.com/users/shays10/received_events","type":"User","site_admin":false},"labels":[{"id":64219391,"url":"https://api.github.com/repos/hashicorp/consul/labels/bug","name":"bug","color":"fc2929","default":true},{"id":612589707,"url":"https://api.github.com/repos/hashicorp/consul/labels/theme/health-checks","name":"theme/health-checks","color":"c5def5","default":false}],"state":"open","locked":false,"assignee":{"login":"magiconair","id":229005,"avatar_url":"https://avatars3.githubusercontent.com/u/229005?v=4","gravatar_id":"","url":"https://api.github.com/users/magiconair","html_url":"https://github.com/magiconair","followers_url":"https://api.github.com/users/magiconair/followers","following_url":"https://api.github.com/users/magiconair/following{/other_user}","gists_url":"https://api.github.com/users/magiconair/gists{/gist_id}","starred_url":"https://api.github.com/users/magiconair/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magiconair/subscriptions","organizations_url":"https://api.github.com/users/magiconair/orgs","repos_url":"https://api.github.com/users/magiconair/repos","events_url":"https://api.github.com/users/magiconair/events{/privacy}","received_events_url":"https://api.github.com/users/magiconair/received_events","type":"User","site_admin":false},"assignees":[{"login":"magiconair","id":229005,"avatar_url":"https://avatars3.githubusercontent.com/u/229005?v=4","gravatar_id":"","url":"https://api.github.com/users/magiconair","html_url":"https://github.com/magiconair","followers_url":"https://api.github.com/users/magiconair/followers","following_url":"https://api.github.com/users/magiconair/following{/other_user}","gists_url":"https://api.github.com/users/magiconair/gists{/gist_id}","starred_url":"https://api.github.com/users/magiconair/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magiconair/subscriptions","organizations_url":"https://api.github.com/users/magiconair/orgs","repos_url":"https://api.github.com/users/magiconair/repos","events_url":"https://api.github.com/users/magiconair/events{/privacy}","received_events_url":"https://api.github.com/users/magiconair/received_events","type":"User","site_admin":false}],"milestone":null,"comments":14,"created_at":"2017-07-11T18:49:33Z","updated_at":"2017-10-10T07:08:31Z","closed_at":null,"author_association":"NONE","body":"Hey,\r\n\r\nThis is very much related to #1244\r\nI'm using blocking HTTP queries and I encountered a strange behavior where consul index is incremented even when healthchecks didn't change their status (or their output)\r\nSo I looked at the value of X-Consul-Index, searched for an offending healthcheck that had that exact same number (via /health/checks/any) and presumed that I'll find that the check state was changed.\r\nBut all I could find in the log is \r\nhttps://gist.github.com/shays10/d0bdb43e0ae3664af1686aa139f18965\r\n\r\nAll checks are synced again, and this happens every 90-120 seconds. I'm guessing that it happens when antiEntropy kicks in and it finds that the service is synced but the checks are not.\r\nI managed to find a correlation between the timestamp of the sync (according to the log) and the time where the ModifyIndex was bumped in all the checks.\r\nAny idea on how to find the root cause? I'm having lots of checks on a lot of nodes and this causes the index to go wild.\r\n\r\nI tried to make sure as much as I can that there isn't a flaky health check that fails randomly or returns a different output.\r\n\r\nClient: 0.8.3\r\nServer: 0.8.3\r\n\r\nClient:\r\n```\r\nagent:\r\n\tcheck_monitors = 0\r\n\tcheck_ttls = 0\r\n\tchecks = 11\r\n\tservices = 11\r\nbuild:\r\n\tprerelease = \r\n\trevision = ea2a82b\r\n\tversion = 0.8.3\r\nconsul:\r\n\tknown_servers = 3\r\n\tserver = false\r\nruntime:\r\n\tarch = amd64\r\n\tcpu_count = 8\r\n\tgoroutines = 60\r\n\tmax_procs = 2\r\n\tos = linux\r\n\tversion = go1.8.1\r\nserf_lan:\r\n\tencrypted = false\r\n\tevent_queue = 0\r\n\tevent_time = 243\r\n\tfailed = 0\r\n\thealth_score = 0\r\n\tintent_queue = 0\r\n\tleft = 0\r\n\tmember_time = 1496\r\n\tmembers = 32\r\n\tquery_queue = 0\r\n\tquery_time = 1\r\n\r\n```\r\n\r\nServer:\r\n```\r\nagent:\r\n\tcheck_monitors = 3\r\n\tcheck_ttls = 0\r\n\tchecks = 3\r\n\tservices = 1\r\nbuild:\r\n\tprerelease = \r\n\trevision = ea2a82b\r\n\tversion = 0.8.3\r\nconsul:\r\n\tbootstrap = false\r\n\tknown_datacenters = 1\r\n\tleader = true\r\n\tleader_addr = 172.31.9.104:8300\r\n\tserver = true\r\nraft:\r\n\tapplied_index = 2553095\r\n\tcommit_index = 2553095\r\n\tfsm_pending = 0\r\n\tlast_contact = 0\r\n\tlast_log_index = 2553095\r\n\tlast_log_term = 435\r\n\tlast_snapshot_index = 2551334\r\n\tlast_snapshot_term = 435\r\n\tlatest_configuration = [{Suffrage:Voter ID:172.31.13.71:8300 Address:172.31.13.71:8300} {Suffrage:Voter ID:172.31.11.117:8300 Address:172.31.11.117:8300} {Suffrage:Voter ID:172.31.9.104:8300 Address:172.31.9.104:8300}]\r\n\tlatest_configuration_index = 1180420\r\n\tnum_peers = 2\r\n\tprotocol_version = 2\r\n\tprotocol_version_max = 3\r\n\tprotocol_version_min = 0\r\n\tsnapshot_version_max = 1\r\n\tsnapshot_version_min = 0\r\n\tstate = Leader\r\n\tterm = 435\r\nruntime:\r\n\tarch = amd64\r\n\tcpu_count = 2\r\n\tgoroutines = 247\r\n\tmax_procs = 2\r\n\tos = linux\r\n\tversion = go1.8.1\r\nserf_lan:\r\n\tencrypted = false\r\n\tevent_queue = 0\r\n\tevent_time = 243\r\n\tfailed = 0\r\n\thealth_score = 0\r\n\tintent_queue = 0\r\n\tleft = 0\r\n\tmember_time = 1496\r\n\tmembers = 32\r\n\tquery_queue = 0\r\n\tquery_time = 1\r\nserf_wan:\r\n\tencrypted = false\r\n\tevent_queue = 0\r\n\tevent_time = 1\r\n\tfailed = 0\r\n\thealth_score = 0\r\n\tintent_queue = 0\r\n\tleft = 0\r\n\tmember_time = 92\r\n\tmembers = 3\r\n\tquery_queue = 0\r\n\tquery_time = 1\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/335381323","html_url":"https://github.com/hashicorp/consul/issues/3259#issuecomment-335381323","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/3259","id":335381323,"user":{"login":"magiconair","id":229005,"avatar_url":"https://avatars3.githubusercontent.com/u/229005?v=4","gravatar_id":"","url":"https://api.github.com/users/magiconair","html_url":"https://github.com/magiconair","followers_url":"https://api.github.com/users/magiconair/followers","following_url":"https://api.github.com/users/magiconair/following{/other_user}","gists_url":"https://api.github.com/users/magiconair/gists{/gist_id}","starred_url":"https://api.github.com/users/magiconair/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magiconair/subscriptions","organizations_url":"https://api.github.com/users/magiconair/orgs","repos_url":"https://api.github.com/users/magiconair/repos","events_url":"https://api.github.com/users/magiconair/events{/privacy}","received_events_url":"https://api.github.com/users/magiconair/received_events","type":"User","site_admin":false},"created_at":"2017-10-10T07:08:31Z","updated_at":"2017-10-10T07:08:31Z","author_association":"CONTRIBUTOR","body":"@ponimas If I can find the root cause then probably. However, usually this is triggered by the `Output` field being different. I'll try to have a look this week but it is a short week for me."}}
{"actor":{"display_login":"magiconair"}}