{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/2375","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/2375/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/2375/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/2375/events","html_url":"https://github.com/prometheus/prometheus/issues/2375","id":203717075,"number":2375,"title":"K8s apiservers target detection broken after upgrade to v1.5.0","user":{"login":"dannyk81","id":24274156,"avatar_url":"https://avatars.githubusercontent.com/u/24274156?v=3","gravatar_id":"","url":"https://api.github.com/users/dannyk81","html_url":"https://github.com/dannyk81","followers_url":"https://api.github.com/users/dannyk81/followers","following_url":"https://api.github.com/users/dannyk81/following{/other_user}","gists_url":"https://api.github.com/users/dannyk81/gists{/gist_id}","starred_url":"https://api.github.com/users/dannyk81/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dannyk81/subscriptions","organizations_url":"https://api.github.com/users/dannyk81/orgs","repos_url":"https://api.github.com/users/dannyk81/repos","events_url":"https://api.github.com/users/dannyk81/events{/privacy}","received_events_url":"https://api.github.com/users/dannyk81/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-01-27T19:18:29Z","updated_at":"2017-01-31T13:22:33Z","closed_at":null,"body":"**What did you do?**\r\nUpgraded Prometheus from v1.4.1 to v1.5.0 (Configuration file unchanged)\r\n\r\n**What did you expect to see?**\r\n\r\nAll targets discovered prior to the upgrade to be available.\r\n\r\n**What did you see instead? Under which circumstances?**\r\n\r\n`kubernetes-apiservers` targets are missing, however all other targets were discovered (`kubernetes-[nodes|pods|service-endpoints]`)\r\n\r\nOnce reverted back to v1.4.1 the missing targets reappeared.\r\n\r\n**Environment**\r\nPrometheus v1.4.1 upgraded to v1.5.0\r\nRunning as a service in K8s cluster version 1.5.2\r\n\r\n* System information: Official docker image `prom/prometheus`\r\n\r\n* Prometheus version: v1.5.0\r\n\r\n* Prometheus configuration file:\r\n```\r\nglobal:\r\n  external_labels:\r\n    datacenter: <dc-name>\r\nrule_files:\r\n- \"/etc/prometheus-rules/*.rules\"\r\nscrape_configs:\r\n- job_name: 'kubernetes-apiservers'\r\n  kubernetes_sd_configs:\r\n  - role: endpoints\r\n  scheme: https\r\n  tls_config:\r\n    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r\n    insecure_skip_verify: true\r\n  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\r\n  relabel_configs:\r\n  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]\r\n    action: keep\r\n    regex: default;kubernetes;https\r\n- job_name: 'kubernetes-nodes'\r\n  scheme: https\r\n  tls_config:\r\n    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r\n    insecure_skip_verify: true\r\n  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\r\n  kubernetes_sd_configs:\r\n  - role: node\r\n  relabel_configs:\r\n  - action: labelmap\r\n    regex: __meta_kubernetes_node_label_(.+)\r\n  - source_labels: [__meta_kubernetes_node_name]\r\n    regex: (.+)\r\n    target_label: instance\r\n    replacement: ${1}\r\n- job_name: 'kubernetes-service-endpoints'\r\n  kubernetes_sd_configs:\r\n  - role: endpoints\r\n  relabel_configs:\r\n  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]\r\n    action: keep\r\n    regex: true\r\n  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]\r\n    action: replace\r\n    target_label: __scheme__\r\n    regex: (https?)\r\n  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]\r\n    action: replace\r\n    target_label: __metrics_path__\r\n    regex: (.+)\r\n  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]\r\n    action: replace\r\n    target_label: __address__\r\n    regex: (.+)(?::\\d+);(\\d+)\r\n    replacement: $1:$2\r\n  - action: labelmap\r\n    regex: __meta_kubernetes_service_label_(.+)\r\n  - source_labels: [__meta_kubernetes_namespace]\r\n    action: replace\r\n    target_label: kubernetes_namespace\r\n  - source_labels: [__meta_kubernetes_service_name]\r\n    action: replace\r\n    target_label: kubernetes_name\r\n- job_name: 'kubernetes-services'\r\n  metrics_path: /probe\r\n  params:\r\n    module: [http_2xx]\r\n  kubernetes_sd_configs:\r\n  - role: service\r\n  relabel_configs:\r\n  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]\r\n    action: keep\r\n    regex: true\r\n  - source_labels: [__address__]\r\n    target_label: __param_target\r\n  - target_label: __address__\r\n    replacement: blackbox\r\n  - source_labels: [__param_target]\r\n    target_label: instance\r\n  - action: labelmap\r\n    regex: __meta_kubernetes_service_label_(.+)\r\n  - source_labels: [__meta_kubernetes_namespace]\r\n    target_label: kubernetes_namespace\r\n  - source_labels: [__meta_kubernetes_service_name]\r\n    target_label: kubernetes_name\r\n- job_name: 'kubernetes-pods'\r\n  kubernetes_sd_configs:\r\n  - role: pod\r\n  relabel_configs:\r\n  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\r\n    action: keep\r\n    regex: true\r\n  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]\r\n    action: replace\r\n    target_label: __metrics_path__\r\n    regex: (.+)\r\n  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]\r\n    action: replace\r\n    regex: (.+):(?:\\d+);(\\d+)\r\n    replacement: ${1}:${2}\r\n    target_label: __address__\r\n  - action: labelmap\r\n    regex: __meta_kubernetes_pod_label_(.+)\r\n  - source_labels: [__meta_kubernetes_namespace]\r\n    action: replace\r\n    target_label: kubernetes_namespace\r\n  - source_labels: [__meta_kubernetes_pod_name]\r\n    action: replace\r\n    target_label: kubernetes_pod_name\r\n  - source_labels: [__meta_kubernetes_pod_node_name]\r\n    action: replace\r\n    target_label: kubernetes_pod_node_name\r\n  - source_labels: [kubernetes_pod_node_name]\r\n    regex: (.+)\r\n    target_label: instance\r\n    replacement: ${1}\r\n- job_name: 'kubernetes-master-nodes'\r\n  static_configs:\r\n  - targets: ['<master-1>:9100', '<master-2>:9100', '<master-3>:9100']\r\n    labels:\r\n      node_type: master\r\n      metrics_source: node_exporter\r\n  relabel_configs:\r\n  - source_labels: [__address__]\r\n    regex: (.+):(.+)\r\n    target_label: instance\r\n    replacement: ${1}\r\n```\r\n\r\n* Logs:\r\n```\r\n2017-01-27 19:01:07\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T19:01:07Z\" level=info msg=\"See you next time!\" source=\"main.go:237\"\r\n2017-01-27 19:01:07\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T19:01:07Z\" level=warning msg=\"Received SIGTERM, exiting gracefully...\" source=\"main.go:230\"\r\n2017-01-27 19:00:17\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T19:00:17Z\" level=info msg=\"Done checkpointing in-memory metrics and chunks in 2.059339237s.\" source=\"persistence.go:638\"\r\n2017-01-27 19:00:15\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T19:00:15Z\" level=info msg=\"Checkpointing in-memory metrics and chunks...\" source=\"persistence.go:611\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"Using pod service account via in-cluster config\" source=\"kubernetes.go:94\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"Using pod service account via in-cluster config\" source=\"kubernetes.go:94\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"Using pod service account via in-cluster config\" source=\"kubernetes.go:94\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"Using pod service account via in-cluster config\" source=\"kubernetes.go:94\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"Using pod service account via in-cluster config\" source=\"kubernetes.go:94\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"Starting target manager...\" source=\"targetmanager.go:61\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"Listening on :9090\" source=\"web.go:259\"\r\n2017-01-27 18:50:14\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:14Z\" level=info msg=\"50500 series loaded.\" source=\"storage.go:378\"\r\n2017-01-27 18:50:13\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:13Z\" level=info msg=\"Loading series map and head chunks...\" source=\"storage.go:373\"\r\n2017-01-27 18:50:12\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:12Z\" level=info msg=\"Loading configuration file /etc/prometheus/prometheus.yml\" source=\"main.go:248\"\r\n2017-01-27 18:50:12\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:12Z\" level=info msg=\"Build context (go=go1.7.4, user=root@a04ed5b536e3, date=20170123-13:56:24)\" source=\"main.go:76\"\r\n2017-01-27 18:50:12\tprometheus-3701298219-knglc\tprometheus\ttime=\"2017-01-27T18:50:12Z\" level=info msg=\"Starting prometheus (version=1.5.0, branch=master, revision=d840f2c400629a846b210cf58d65b9fbae0f1d5c)\" source=\"main.go:75\"\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/276361305","html_url":"https://github.com/prometheus/prometheus/issues/2375#issuecomment-276361305","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/2375","id":276361305,"user":{"login":"dannyk81","id":24274156,"avatar_url":"https://avatars.githubusercontent.com/u/24274156?v=3","gravatar_id":"","url":"https://api.github.com/users/dannyk81","html_url":"https://github.com/dannyk81","followers_url":"https://api.github.com/users/dannyk81/followers","following_url":"https://api.github.com/users/dannyk81/following{/other_user}","gists_url":"https://api.github.com/users/dannyk81/gists{/gist_id}","starred_url":"https://api.github.com/users/dannyk81/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dannyk81/subscriptions","organizations_url":"https://api.github.com/users/dannyk81/orgs","repos_url":"https://api.github.com/users/dannyk81/repos","events_url":"https://api.github.com/users/dannyk81/events{/privacy}","received_events_url":"https://api.github.com/users/dannyk81/received_events","type":"User","site_admin":false},"created_at":"2017-01-31T13:22:33Z","updated_at":"2017-01-31T13:22:33Z","body":"Interesting,,. I guess I didn't wait long enough for them to reappear. "}}
{"actor":{"display_login":"dannyk81"}}