{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin/issues/13337","repository_url":"https://api.github.com/repos/openshift/origin","labels_url":"https://api.github.com/repos/openshift/origin/issues/13337/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin/issues/13337/comments","events_url":"https://api.github.com/repos/openshift/origin/issues/13337/events","html_url":"https://github.com/openshift/origin/pull/13337","id":213258647,"number":13337,"title":"Instrument the HAProxy router with metrics that contain route info","user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars3.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"labels":[{"id":210156114,"url":"https://api.github.com/repos/openshift/origin/labels/component/networking","name":"component/networking","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/openshift/origin/milestones/18","html_url":"https://github.com/openshift/origin/milestone/18","labels_url":"https://api.github.com/repos/openshift/origin/milestones/18/labels","id":2151886,"number":18,"title":"1.6.0","description":"","creator":{"login":"ncdc","id":354,"avatar_url":"https://avatars1.githubusercontent.com/u/354?v=3","gravatar_id":"","url":"https://api.github.com/users/ncdc","html_url":"https://github.com/ncdc","followers_url":"https://api.github.com/users/ncdc/followers","following_url":"https://api.github.com/users/ncdc/following{/other_user}","gists_url":"https://api.github.com/users/ncdc/gists{/gist_id}","starred_url":"https://api.github.com/users/ncdc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ncdc/subscriptions","organizations_url":"https://api.github.com/users/ncdc/orgs","repos_url":"https://api.github.com/users/ncdc/repos","events_url":"https://api.github.com/users/ncdc/events{/privacy}","received_events_url":"https://api.github.com/users/ncdc/received_events","type":"User","site_admin":false},"open_issues":71,"closed_issues":9,"state":"open","created_at":"2016-11-22T16:19:57Z","updated_at":"2017-03-21T20:57:44Z","due_on":null,"closed_at":null},"comments":20,"created_at":"2017-03-10T06:44:04Z","updated_at":"2017-03-22T04:35:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/openshift/origin/pulls/13337","html_url":"https://github.com/openshift/origin/pull/13337","diff_url":"https://github.com/openshift/origin/pull/13337.diff","patch_url":"https://github.com/openshift/origin/pull/13337.patch"},"body":"Instrument the template router with a new optional flag `--metrics-type` and `--listen-addr`, which follow the conventions of the other infrastructure components and register a health check (not used currently, since it doesn't check HAProxy), the pprof endpoints, and a prometheus metrics endpoint.  Pprof and metrics are protected by the stats endpoint basic auth.  `oadm router` now enables a default listen address of `0.0.0.0:1935`.\r\n\r\nOn the template router, start with the prometheus haproxy_exporter to extract metrics from the running HAProxy process.  Since large amounts of metrics are returned and reported, slim the list of metrics down to a smaller subset, with a focus on those that can provide meaningful input at large scales.  Add a server threshold to stop reporting server metrics (and instead only backend metrics), when the number of servers in haproxy exceeds that, defaults to 1000 endpoints.  Add a rate limiter on the number of times that the HAProxy stats endpoint can be called that is inversely proportional to the number total stats entries (usually dominated by backends + servers) to reduce total work - defaults to 5s for every 1000 servers or backends.  Also, right before reload, capture the latest stats and report them on the next call (since we don't know how to merge metrics).\r\n\r\nAdd additional metrics on the router itself to report reload times and write config.\r\n\r\nTo properly expose metrics to the user, the following changes were made to the router template:\r\n\r\n1. Switch from `_` as a separator in backend/server names to `:`, which is impossible to get from a route or ingress (previously ingress could result in `_` being added, which becomes ambiguous)\r\n2. Switch server names from the `IdHash` to `ID` (endpoint IP + port number), which was not a required security step (cookie values are loaded separately).\r\n\r\nWith these in place, a caller can get a wide set of metrics from the router, which can be used for custom autoscaling, monitoring, and problem detecting:\r\n\r\n(limited example, most backends removed).\r\n\r\n```\r\n# HELP haproxy_backend_connections_total Total number of connections.\r\n# TYPE haproxy_backend_connections_total gauge\r\nhaproxy_backend_connections_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\"} 20\r\nhaproxy_backend_connections_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"zeroweightroute\"} 0\r\n# HELP haproxy_backend_current_queue Current number of queued requests not assigned to any server.\r\n# TYPE haproxy_backend_current_queue gauge\r\nhaproxy_backend_current_queue{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\"} 0\r\nhaproxy_backend_current_queue{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"zeroweightroute\"} 0\r\n# HELP haproxy_backend_up Current health status of the backend (1 = UP, 0 = DOWN).\r\n# TYPE haproxy_backend_up gauge\r\nhaproxy_backend_up{backend=\"http\",namespace=\"default\",route=\"backend4\"} 1\r\nhaproxy_backend_up{backend=\"http\",namespace=\"default\",route=\"docker-registry\"} 1\r\nhaproxy_backend_up{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"zeroweightroute\"} 1\r\n# HELP haproxy_exporter_csv_parse_failures Number of errors while parsing CSV.\r\n# TYPE haproxy_exporter_csv_parse_failures counter\r\nhaproxy_exporter_csv_parse_failures 0\r\n# HELP haproxy_exporter_scrape_interval The time in seconds before another scrape is allowed, proportional to size of data.\r\n# TYPE haproxy_exporter_scrape_interval gauge\r\nhaproxy_exporter_scrape_interval 5\r\n# HELP haproxy_exporter_server_threshold Number of servers tracked and the current threshold value.\r\n# TYPE haproxy_exporter_server_threshold counter\r\nhaproxy_exporter_server_threshold{type=\"current\"} 13\r\nhaproxy_exporter_server_threshold{type=\"limit\"} 500\r\n# HELP haproxy_exporter_total_scrapes Current total HAProxy scrapes.\r\n# TYPE haproxy_exporter_total_scrapes counter\r\nhaproxy_exporter_total_scrapes 27\r\n# HELP haproxy_frontend_bytes_in_total Current total of incoming bytes.\r\n# TYPE haproxy_frontend_bytes_in_total gauge\r\nhaproxy_frontend_bytes_in_total{frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_bytes_in_total{frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_bytes_in_total{frontend=\"public\"} 1680\r\nhaproxy_frontend_bytes_in_total{frontend=\"public_ssl\"} 0\r\nhaproxy_frontend_bytes_in_total{frontend=\"stats\"} 207\r\n# HELP haproxy_frontend_bytes_out_total Current total of outgoing bytes.\r\n# TYPE haproxy_frontend_bytes_out_total gauge\r\nhaproxy_frontend_bytes_out_total{frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_bytes_out_total{frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_bytes_out_total{frontend=\"public\"} 2680\r\nhaproxy_frontend_bytes_out_total{frontend=\"public_ssl\"} 0\r\nhaproxy_frontend_bytes_out_total{frontend=\"stats\"} 479\r\n# HELP haproxy_frontend_connections_total Total number of connections.\r\n# TYPE haproxy_frontend_connections_total gauge\r\nhaproxy_frontend_connections_total{frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_connections_total{frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_connections_total{frontend=\"public\"} 20\r\nhaproxy_frontend_connections_total{frontend=\"public_ssl\"} 0\r\nhaproxy_frontend_connections_total{frontend=\"stats\"} 3\r\n# HELP haproxy_frontend_current_session_rate Current number of sessions per second over last elapsed second.\r\n# TYPE haproxy_frontend_current_session_rate gauge\r\nhaproxy_frontend_current_session_rate{frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_current_session_rate{frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_current_session_rate{frontend=\"public\"} 0\r\nhaproxy_frontend_current_session_rate{frontend=\"public_ssl\"} 0\r\nhaproxy_frontend_current_session_rate{frontend=\"stats\"} 0\r\n# HELP haproxy_frontend_current_sessions Current number of active sessions.\r\n# TYPE haproxy_frontend_current_sessions gauge\r\nhaproxy_frontend_current_sessions{frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_current_sessions{frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_current_sessions{frontend=\"public\"} 0\r\nhaproxy_frontend_current_sessions{frontend=\"public_ssl\"} 0\r\nhaproxy_frontend_current_sessions{frontend=\"stats\"} 0\r\n# HELP haproxy_frontend_http_responses_total Total of HTTP responses.\r\n# TYPE haproxy_frontend_http_responses_total gauge\r\nhaproxy_frontend_http_responses_total{code=\"2xx\",frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_http_responses_total{code=\"2xx\",frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_http_responses_total{code=\"2xx\",frontend=\"public\"} 20\r\nhaproxy_frontend_http_responses_total{code=\"2xx\",frontend=\"stats\"} 2\r\nhaproxy_frontend_http_responses_total{code=\"5xx\",frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_http_responses_total{code=\"5xx\",frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_http_responses_total{code=\"5xx\",frontend=\"public\"} 0\r\nhaproxy_frontend_http_responses_total{code=\"5xx\",frontend=\"stats\"} 0\r\n# HELP haproxy_frontend_max_session_rate Maximum observed number of sessions per second.\r\n# TYPE haproxy_frontend_max_session_rate gauge\r\nhaproxy_frontend_max_session_rate{frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_max_session_rate{frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_max_session_rate{frontend=\"public\"} 10\r\nhaproxy_frontend_max_session_rate{frontend=\"public_ssl\"} 0\r\nhaproxy_frontend_max_session_rate{frontend=\"stats\"} 2\r\n# HELP haproxy_frontend_max_sessions Maximum observed number of active sessions.\r\n# TYPE haproxy_frontend_max_sessions gauge\r\nhaproxy_frontend_max_sessions{frontend=\"fe_no_sni\"} 0\r\nhaproxy_frontend_max_sessions{frontend=\"fe_sni\"} 0\r\nhaproxy_frontend_max_sessions{frontend=\"public\"} 1\r\nhaproxy_frontend_max_sessions{frontend=\"public_ssl\"} 0\r\nhaproxy_frontend_max_sessions{frontend=\"stats\"} 1\r\n# HELP haproxy_process_cpu_seconds_total Total user and system CPU time spent in seconds.\r\n# TYPE haproxy_process_cpu_seconds_total counter\r\nhaproxy_process_cpu_seconds_total 0.01\r\n# HELP haproxy_process_max_fds Maximum number of open file descriptors.\r\n# TYPE haproxy_process_max_fds gauge\r\nhaproxy_process_max_fds 40046\r\n# HELP haproxy_process_resident_memory_bytes Resident memory size in bytes.\r\n# TYPE haproxy_process_resident_memory_bytes gauge\r\nhaproxy_process_resident_memory_bytes 6.836224e+06\r\n# HELP haproxy_process_start_time_seconds Start time of the process since unix epoch in seconds.\r\n# TYPE haproxy_process_start_time_seconds gauge\r\nhaproxy_process_start_time_seconds 1.48929703048e+09\r\n# HELP haproxy_process_virtual_memory_bytes Virtual memory size in bytes.\r\n# TYPE haproxy_process_virtual_memory_bytes gauge\r\nhaproxy_process_virtual_memory_bytes 5.4120448e+07\r\n# HELP haproxy_server_bytes_in_total Current total of incoming bytes.\r\n# TYPE haproxy_server_bytes_in_total gauge\r\nhaproxy_server_bytes_in_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 840\r\n# HELP haproxy_server_bytes_out_total Current total of outgoing bytes.\r\n# TYPE haproxy_server_bytes_out_total gauge\r\nhaproxy_server_bytes_out_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 1340\r\n# HELP haproxy_server_check_failures_total Total number of failed health checks.\r\n# TYPE haproxy_server_check_failures_total gauge\r\nhaproxy_server_check_failures_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_connection_errors_total Total of connection errors.\r\n# TYPE haproxy_server_connection_errors_total gauge\r\nhaproxy_server_connection_errors_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_connections_total Total number of connections.\r\n# TYPE haproxy_server_connections_total gauge\r\nhaproxy_server_connections_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 10\r\n# HELP haproxy_server_current_queue Current number of queued requests assigned to this server.\r\n# TYPE haproxy_server_current_queue gauge\r\nhaproxy_server_current_queue{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_current_session_rate Current number of sessions per second over last elapsed second.\r\n# TYPE haproxy_server_current_session_rate gauge\r\nhaproxy_server_current_session_rate{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_current_sessions Current number of active sessions.\r\n# TYPE haproxy_server_current_sessions gauge\r\nhaproxy_server_current_sessions{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_downtime_seconds_total Total downtime in seconds.\r\n# TYPE haproxy_server_downtime_seconds_total gauge\r\nhaproxy_server_downtime_seconds_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_http_responses_total Total of HTTP responses.\r\n# TYPE haproxy_server_http_responses_total gauge\r\nhaproxy_server_http_responses_total{backend=\"http\",code=\"2xx\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 10\r\nhaproxy_server_http_responses_total{backend=\"http\",code=\"5xx\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_max_session_rate Maximum observed number of sessions per second.\r\n# TYPE haproxy_server_max_session_rate gauge\r\nhaproxy_server_max_session_rate{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 5\r\n# HELP haproxy_server_max_sessions Maximum observed number of active sessions.\r\n# TYPE haproxy_server_max_sessions gauge\r\nhaproxy_server_max_sessions{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 1\r\n# HELP haproxy_server_response_errors_total Total of response errors.\r\n# TYPE haproxy_server_response_errors_total gauge\r\nhaproxy_server_response_errors_total{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 0\r\n# HELP haproxy_server_up Current health status of the server (1 = UP, 0 = DOWN).\r\n# TYPE haproxy_server_up gauge\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend1\",server=\"172.17.0.3:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend1-1\",server=\"172.17.0.3:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend1-2\",server=\"172.17.0.3:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend1-2\",server=\"172.17.0.4:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend1-3\",server=\"172.17.0.3:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend2\",server=\"172.17.0.4:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend3\",server=\"172.17.0.4:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"backend4\",server=\"172.17.0.4:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"default\",route=\"docker-registry\",server=\"172.17.0.5:5000\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.10:8080\"} 1\r\nhaproxy_server_up{backend=\"http\",namespace=\"extended-test-router-metrics-2l1f7-jk7c0\",route=\"weightedroute\",server=\"172.17.0.11:8080\"} 1\r\nhaproxy_server_up{backend=\"other/be_no_sni\",namespace=\"\",route=\"\",server=\"fe_no_sni\"} 1\r\nhaproxy_server_up{backend=\"other/be_sni\",namespace=\"\",route=\"\",server=\"fe_sni\"} 1\r\n# HELP haproxy_up Was the last scrape of haproxy successful.\r\n# TYPE haproxy_up gauge\r\nhaproxy_up 1\r\n# HELP openshift_build_info A metric with a constant '1' value labeled by major, minor, git commit & git version from which OpenShift was built.\r\n# TYPE openshift_build_info gauge\r\nopenshift_build_info{gitCommit=\"39ecc8d\",gitVersion=\"v1.5.0-alpha.3+39ecc8d-355-dirty\",major=\"1\",minor=\"5+\"} 1\r\n# HELP template_router_reload_seconds Measures the time spent reloading the router in seconds.\r\n# TYPE template_router_reload_seconds summary\r\ntemplate_router_reload_seconds{quantile=\"0.5\"} 0.067299316\r\ntemplate_router_reload_seconds{quantile=\"0.9\"} 0.118891744\r\ntemplate_router_reload_seconds{quantile=\"0.99\"} 0.128374662\r\ntemplate_router_reload_seconds_sum 1.69002939\r\ntemplate_router_reload_seconds_count 21\r\n# HELP template_router_write_config_seconds Measures the time spent writing out the router configuration to disk in seconds.\r\n# TYPE template_router_write_config_seconds summary\r\ntemplate_router_write_config_seconds{quantile=\"0.5\"} 0.004694066\r\ntemplate_router_write_config_seconds{quantile=\"0.9\"} 0.012401996\r\ntemplate_router_write_config_seconds{quantile=\"0.99\"} 0.017210267\r\ntemplate_router_write_config_seconds_sum 0.15450527399999994\r\ntemplate_router_write_config_seconds_count 21\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/openshift/origin/issues/comments/288296419","html_url":"https://github.com/openshift/origin/pull/13337#issuecomment-288296419","issue_url":"https://api.github.com/repos/openshift/origin/issues/13337","id":288296419,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars3.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2017-03-22T04:35:37Z","updated_at":"2017-03-22T04:35:37Z","body":"@DirectXMan12 since you know some prometheus, some metrics, and some networking, can you help review this?"}}
{"actor":{"display_login":"smarterclayton"}}