{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/1873","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/1873/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1873/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/1873/events","html_url":"https://github.com/prometheus/prometheus/issues/1873","id":169774945,"number":1873,"title":"Too many open files (established connections to same nodes)","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars0.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[{"id":365563588,"url":"https://api.github.com/repos/prometheus/prometheus/labels/kind/bug","name":"kind/bug","color":"ff0000","default":false},{"id":365563606,"url":"https://api.github.com/repos/prometheus/prometheus/labels/priority/P1","name":"priority/P1","color":"eb6420","default":false}],"state":"open","locked":false,"assignee":{"login":"beorn7","id":5609886,"avatar_url":"https://avatars0.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"assignees":[{"login":"beorn7","id":5609886,"avatar_url":"https://avatars0.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false}],"milestone":null,"comments":30,"created_at":"2016-08-07T01:39:24Z","updated_at":"2017-03-28T11:18:12Z","closed_at":null,"body":"**What did you do?**\nLeft prometheus running against ~20 targets using DNS discovery.\n\n**What did you expect to see?**\nHappy prometheus.\n\n**What did you see instead? Under which circumstances?**\nUnhappy prometheus: after about 2 weeks of prometheus running just fine, prometheus is unresponsive on its web port and is logging lots of \"too many open files\" errors..\n\n**Environment**\nGCE debian instance with 4vCPUs, 15GiB ram.\n- System information:\n\n```\n$ uname -a\nLinux monitoring 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux\n```\n- Prometheus version: \n\n```\n$ ./prometheus/prometheus -version\nprometheus, version 1.0.1 (branch: master, revision: be40190)\n  build user:       root@e881b289ce76\n  build date:       20160722-19:54:46\n  go version:       go1.6.2\n```\n- Alertmanager version:\n  (probably not relevant, but it is running, so here it is anyway)\n\n```\n$ ./alertmanager/alertmanager -version\nalertmanager, version 0.3.0 (branch: master, revision: 97eca6d)\n  build user:       root@b5b16edd7db9\n  build date:       20160708-10:52:58\n  go version:       go1.6.2\n```\n- Prometheus configuration file:\n  My prometheus config is as follows. The salient details are: `dns_sd_config` using 'A' records, and `tls_config` with `insecure-skip-verify`. The DNS records have not changed since prometheus started.\n\n```\nglobal:\n  scrape_interval: 10s\n\nrule_files:\n- \"alerts.rules\"\n\nscrape_configs:\n  - job_name: 'prometheus'\n    static_configs:\n      - targets: ['localhost:9090']\n\n  - job_name: 'cockroach'\n    # metrics_path defaults to '/metrics'\n    metrics_path: '/_status/vars'\n    # scheme defaults to 'http'.\n    scheme: 'https'\n    # Insecure for now. TODO(mberhault): use self-signed CA.\n    tls_config:\n      insecure_skip_verify: true\n\n    # Resolve DNS to find hosts.\n    # *NOTE* this requires up-to-date DNS.\n    dns_sd_configs:\n    - names: ['beta.gce.cockroachdb.com',\n              'gamma.gce.cockroachdb.com',\n              'rho.gce.cockroachdb.com']\n      type: 'A'\n      port: 8080\n    # Create a \"cluster\" label using the beginning of the dns name (eg: 'beta') as the value.\n    # *NOTE* this requires accurate DNS names.\n    relabel_configs:\n    - action: replace\n      source_labels: [__meta_dns_name]\n      target_label: cluster\n      regex: ([^.]+)\\.gce\\.cockroachdb\\.com\n      replacement: $1\n```\n- Logs:\n  Please find the entire prometheus log attached:\n  [prometheus.stderr.gz](https://github.com/prometheus/prometheus/files/405253/prometheus.stderr.gz)\n\n**Additional details**\nA quick lsof shows the following many established connections along the lines of:\n\n```\nprometheu 2364 root 2037u  IPv4 2981837      0t0     TCP monitoring.c.cockroach-shared.internal:56735->72.157.196.104.bc.googleusercontent.com:http-alt (ESTABLISHED)\n```\n\nCounting by target address, I get the following counts:\n\n```\n$ sudo lsof -p 2364|awk '{print $9}'|cut -d'>' -f2|sort|uniq -c|sort -n|tail\n      3 pipe\n     14 82.54.196.104.bc.googleusercontent.com:http-alt\n     67 126.150.196.104.bc.googleusercontent.com:http-alt\n    161 63.18.196.104.bc.googleusercontent.com:http-alt\n    172 89.153.196.104.bc.googleusercontent.com:http-alt\n    185 72.157.196.104.bc.googleusercontent.com:http-alt\n    221 49.102.196.104.bc.googleusercontent.com:http-alt\n    245 29.141.196.104.bc.googleusercontent.com:http-alt\n    290 144.153.196.104.bc.googleusercontent.com:http-alt\n    659 29.164.196.104.bc.googleusercontent.com:http-alt\n```\n\nThose addresses match targets that have been flapping for a while, with sometimes hours of down-time between restarts.\n\nnetstat confirms the connections are established:\n\n```\n# Small snippet of :8080 targets using netstat -tpen\ntcp        0      0 10.142.0.32:51342       104.196.164.29:8080     ESTABLISHED 0          317662      -\ntcp        0      0 10.142.0.32:59371       104.196.164.29:8080     ESTABLISHED 0          2696638     -\ntcp        0      0 10.142.0.32:36329       104.196.153.144:8080    ESTABLISHED 0          2810646     -\ntcp        0      0 10.142.0.32:54494       104.196.153.89:8080     ESTABLISHED 0          2898911     -\n```\n\n**Further debugging**\nI am currently leaving prometheus running in case there is some extra debugging information I can provide. Please let me know.\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/289738676","html_url":"https://github.com/prometheus/prometheus/issues/1873#issuecomment-289738676","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1873","id":289738676,"user":{"login":"mjansing","id":1260530,"avatar_url":"https://avatars2.githubusercontent.com/u/1260530?v=3","gravatar_id":"","url":"https://api.github.com/users/mjansing","html_url":"https://github.com/mjansing","followers_url":"https://api.github.com/users/mjansing/followers","following_url":"https://api.github.com/users/mjansing/following{/other_user}","gists_url":"https://api.github.com/users/mjansing/gists{/gist_id}","starred_url":"https://api.github.com/users/mjansing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mjansing/subscriptions","organizations_url":"https://api.github.com/users/mjansing/orgs","repos_url":"https://api.github.com/users/mjansing/repos","events_url":"https://api.github.com/users/mjansing/events{/privacy}","received_events_url":"https://api.github.com/users/mjansing/received_events","type":"User","site_admin":false},"created_at":"2017-03-28T11:18:12Z","updated_at":"2017-03-28T11:18:12Z","body":"I tried to dig a bit more into this issue. I use Prometheus 1.5.2 to scrape five hosts. Each host serves node_exporter metrics behind an nginx reverse proxy using an self-signed certificate (protected via basic-auth). After 2 days Prometheus becomes unresponsive until reaching max open fd limit:\r\n\r\n```\r\n$ ulimit -n => 1024\r\n$ ls -l /proc/$PROMETHEUS_PID/fd | wc -l => 1025\r\n```\r\n\r\nIt looks like that Prometheus keeps connections to some of these hosts until fd limit is reached:\r\n\r\n```\r\n$ lsof -p $PROMETHEUS_PID | awk '{print $9}'| cut -d'>' -f2 | sort | uniq -c | sort -n | tail\r\n      1 /var/lib/prometheus/data/labelpair_to_fingerprints/000003.log\r\n      1 /var/lib/prometheus/data/labelpair_to_fingerprints/LOCK\r\n      1 /var/lib/prometheus/data/labelpair_to_fingerprints/LOG\r\n      1 /var/lib/prometheus/data/labelpair_to_fingerprints/MANIFEST-000004\r\n      1 NAME\r\n      1 [eventpoll]\r\n      2 type=STREAM\r\n  206 ip213-244-XXX-XX.XXXXXX.com:https\r\n  274 ip185-48-XXX-XXX.XXXXXX.com:https\r\n  513 ip213-244-XXX-XX.XXXXXX.com:https\r\n```\r\n\r\n```\r\n$ lsof $PROMETHEUS_PID\r\n...snip...\r\nprometheu 56128 root   71u     IPv4            6247120      0t0     TCP monitoring.XXXXX.XXXX.de:53984->ip185-48-XXX-XXX.XXXXXX.com:https (ESTABLISHED)\r\nprometheu 56128 root   72u     IPv4            6249575      0t0     TCP monitoring.XXXXX.XXXX.de:54124->ip185-48-XXX-XXX.XXXXXX.com:https (ESTABLISHED)\r\nprometheu 56128 root   73u     IPv4            6246813      0t0     TCP monitoring.XXXXX.XXXX.de:57626->ip213-244-XXX-XX.XXXXXX.com:https (ESTABLISHED)\r\nprometheu 56128 root   74u     IPv4            6249438      0t0     TCP monitoring.XXXXX.XXXX.de:54090->ip185-48-XXX-XXX.XXXXXX.com:https (ESTABLISHED)\r\nprometheu 56128 root   75u     IPv4            6247208      0t0     TCP monitoring.XXXXX.XXXX.de:54052->ip185-48-XXX-XXX.XXXXXX.com:https (ESTABLISHED)\r\nprometheu 56128 root   76u     IPv4            6247175      0t0     TCP monitoring.XXXXX.XXXX.de:54018->ip185-48-XXX-XXX.XXXXXX.com:https (ESTABLISHED)\r\nprometheu 56128 root   77u     IPv4            6249728      0t0     TCP monitoring.XXXXX.XXXX.de:54158->ip185-48-XXX-XXX.XXXXXX.com:https (ESTABLISHED)\r\nprometheu 56128 root   78u     IPv4            6249839      0t0     TCP monitoring.XXXXX.XXXX.de:54192->ip185-48-XXX-XXX.XXXXXX.com:https (ESTABLISHED)\r\n...snip...\r\n```\r\n\r\nI often get noisy alerts because Prometheus sometimes cannot scrape data from these hosts and then I see `context deadline exceeded`-errors in target status page. During these errors I cannot even scrape hosts via curl:\r\n\r\n```\r\n$ curl -u $USER:$PW --cacert /path/to/cert.crt https://myhost.XXXXX.XXXX.de/metrics -vvv\r\n*   Trying 213.244.XXX.XX...\r\n* Connected to XXXXX.XXXXX.XXX.de (213.244.XXX.XX) port 443 (#0)\r\n* found 1 certificates in /path/to/cert.crt\r\n* found 692 certificates in /etc/ssl/certs\r\n* ALPN, offering http/1.1\r\n* gnutls_handshake() failed: Error in the pull function.\r\n* Closing connection 0\r\ncurl: (35) gnutls_handshake() failed: Error in the pull function.\r\n```\r\n\r\nNginx error logs from scraping target (switched log level to debug) during curl-request:\r\n\r\n```\r\n2017/03/23 14:20:34 [debug] 826#0: accept on 0.0.0.0:443, ready: 0\r\n2017/03/23 14:20:34 [debug] 827#0: accept on 0.0.0.0:443, ready: 0\r\n2017/03/23 14:20:34 [debug] 826#0: posix_memalign: 00000000022FAC00:512 @16\r\n2017/03/23 14:20:34 [debug] 827#0: accept() not ready (11: Resource temporarily unavailable)\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 accept: 78.137.XXX.XXX:51722 fd:4\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 event timer add: 4: 60000:1490278894566\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 reusable connection: 1\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 epoll add event: fd:4 op:1 ev:80002001\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 http check ssl handshake\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 http recv(): 1\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 https ssl handshake: 0x16\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 SSL server name: \"XXXXX.XXXXX.XXXX.de\"\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 SSL ALPN supported by client: http/1.1\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 SSL ALPN selected: http/1.1\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 SSL_do_handshake: -1\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 SSL_get_error: 2\r\n2017/03/23 14:20:34 [debug] 826#0: *18874 reusable connection: 0\r\n2017/03/23 14:21:34 [debug] 826#0: *18874 event timer del: 4: 1490278894566\r\n2017/03/23 14:21:34 [debug] 826#0: *18874 SSL handshake handler: 0\r\n2017/03/23 14:21:34 [info] 826#0: *18874 client timed out (110: Connection timed out) while SSL handshaking, client: 78.137.XXX.XX, server: 0.0.0.0:443\r\n2017/03/23 14:21:34 [debug] 826#0: *18874 close http connection: 4\r\n2017/03/23 14:21:34 [debug] 826#0: *18874 reusable connection: 0\r\n2017/03/23 14:21:34 [debug] 826#0: *18874 free: 00000000022FAC00, unused: 131\r\n```\r\n\r\nSo there is a problem while SSL handshaking and it's dedicated to my Prometheus hosts (Ubuntu 16.04). I don't have any problems when I run a local Prometheus instance or use curl on my local dev machine (macOS). \r\n\r\nFor me it looks like that Prometheus keeps open connections if one of these strange ssl handshake errors occurs. The ssl handshake-problem during curl-request seems to be a GnuTLS-Problem if target works behind a reverse proxy: http://askubuntu.com/questions/186847/error-gnutls-handshake-failed-when-connecting-to-https-servers.\r\n\r\nSo maybe there might be two problems:\r\n\r\n1. Prometheus becomes unresponsive after using all addressable fds (this issue)\r\n2. Failing scrape-requests which results in many open connections to same nodes\r\n\r\nCould this be a problem in Prometheus underlying HTTP/TLS-stack like the GnuTLS one?\r\n\r\nHope my small investigation helps.\r\n\r\n"}}
{"actor":{"display_login":"mjansing"}}