{"action":"created","issue":{"url":"https://api.github.com/repos/grobian/carbon-c-relay/issues/262","repository_url":"https://api.github.com/repos/grobian/carbon-c-relay","labels_url":"https://api.github.com/repos/grobian/carbon-c-relay/issues/262/labels{/name}","comments_url":"https://api.github.com/repos/grobian/carbon-c-relay/issues/262/comments","events_url":"https://api.github.com/repos/grobian/carbon-c-relay/issues/262/events","html_url":"https://github.com/grobian/carbon-c-relay/issues/262","id":223527796,"number":262,"title":"2.6+ Round-Robin DNS Entry forward any_of causes hung connections","user":{"login":"g2bg","id":5659834,"avatar_url":"https://avatars0.githubusercontent.com/u/5659834?v=3","gravatar_id":"","url":"https://api.github.com/users/g2bg","html_url":"https://github.com/g2bg","followers_url":"https://api.github.com/users/g2bg/followers","following_url":"https://api.github.com/users/g2bg/following{/other_user}","gists_url":"https://api.github.com/users/g2bg/gists{/gist_id}","starred_url":"https://api.github.com/users/g2bg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/g2bg/subscriptions","organizations_url":"https://api.github.com/users/g2bg/orgs","repos_url":"https://api.github.com/users/g2bg/repos","events_url":"https://api.github.com/users/g2bg/events{/privacy}","received_events_url":"https://api.github.com/users/g2bg/received_events","type":"User","site_admin":false},"labels":[{"id":71657780,"url":"https://api.github.com/repos/grobian/carbon-c-relay/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-04-22T00:33:36Z","updated_at":"2017-04-23T12:31:59Z","closed_at":null,"body":"On 2.5 the functionality was that it would pick 1 ip from the DNS entry to forward to and stick with it.\r\nIn 2.6 and 3.0 it will connect to 1 of the ip's from the DNS entry and send metrics then leaves the connection open.\r\nThis leads to ever increased connection counts until it runs out of file descriptors.\r\n\r\nSo 2 prong request would be \r\n 1. Fix connection close issue\r\n 2. in any_of load all ip's from a multi-ip dns resolution\r\n\r\n\r\n\r\n[2017-04-21 17:26:56] (MSG) starting carbon-c-relay v3.0 (98424a-dirty), pid=39656\r\nconfiguration:\r\n    relay hostname = server_name\r\n    listen port = 2003\r\n    listen interface = 127.0.0.1\r\n    workers = 4\r\n    send batch size = 2500\r\n    server queue size = 25000\r\n    server max stalls = 4\r\n    listen backlog = 32\r\n    server connection IO timeout = 600ms\r\n    debug = true\r\n    configuration = /etc/carbon-c-relay.conf\r\n\r\nparsed configuration follows:\r\nstatistics\r\n    submit every 60 seconds\r\n    prefix with carbon.relays.server_name\r\n    ;\r\n\r\ncluster local_carbon\r\n    any_of\r\n        multiipdnsentry:2003\r\n    ;\r\n\r\nmatch *\r\n    send to local_carbon\r\n    stop\r\n    ;\r\n\r\n[2017-04-21 17:26:56] (MSG) listening on tcp4 127.0.0.1 port 2003\r\n[2017-04-21 17:26:56] (MSG) listening on udp4 127.0.0.1 port 2003\r\n[2017-04-21 17:26:56] (MSG) listening on UNIX socket /tmp/.s.carbon-c-relay.2003\r\n[2017-04-21 17:26:56] (MSG) starting 4 workers\r\n[2017-04-21 17:26:56] (MSG) starting statistics collector\r\n[2017-04-21 17:26:56] (MSG) starting servers\r\n\r\n\r\n/usr/bin/carbon-c-relay -P /var/run/carbon-c-relay/carbon-c-relay.pid -D -p 2003 -i 127.0.0.1 -w 4 -b 2500 -q 25000 -l /var/log/carbon-c-relay/carbon-c-relay.log -s -f /etc/carbon-c-relay.conf\r\n"},"comment":{"url":"https://api.github.com/repos/grobian/carbon-c-relay/issues/comments/296440060","html_url":"https://github.com/grobian/carbon-c-relay/issues/262#issuecomment-296440060","issue_url":"https://api.github.com/repos/grobian/carbon-c-relay/issues/262","id":296440060,"user":{"login":"g2bg","id":5659834,"avatar_url":"https://avatars0.githubusercontent.com/u/5659834?v=3","gravatar_id":"","url":"https://api.github.com/users/g2bg","html_url":"https://github.com/g2bg","followers_url":"https://api.github.com/users/g2bg/followers","following_url":"https://api.github.com/users/g2bg/following{/other_user}","gists_url":"https://api.github.com/users/g2bg/gists{/gist_id}","starred_url":"https://api.github.com/users/g2bg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/g2bg/subscriptions","organizations_url":"https://api.github.com/users/g2bg/orgs","repos_url":"https://api.github.com/users/g2bg/repos","events_url":"https://api.github.com/users/g2bg/events{/privacy}","received_events_url":"https://api.github.com/users/g2bg/received_events","type":"User","site_admin":false},"created_at":"2017-04-23T12:31:59Z","updated_at":"2017-04-23T12:31:59Z","body":"Sure, this is a per host relay to relay setup. \r\nClient / Host is setup as above and has a collectd and other application metrics pumping in on a 10s interval. I believe last count was around 2k metrics or so.\r\nThe Destinations 4 of them have carbon-c-relay (I've tried this with 2.5,2.6 and 3.0 no diff) talk to the multitude of carbon-cache instances. Currently running around 1m/s metrics on 3.0.\r\nWatching the behavior of 2.5 it is opening and closing connections on interval. \r\nIn 2.6/3.0 it opens the connections on interval as well, just never closes the old.\r\nOnly with the RR DNS though if I put all 4 destination ip's in there's no issue.\r\n\r\nInteresting side note. If I set the DNS entry with useall it expands to the IP's as members in the log when it outputs the config. This has the connection issues though. If I copy / paste that config and use it there's no problems.\r\n\r\nIs there a way to adjust the disconnect timeout? "}}
{"actor":{"display_login":"g2bg"}}