{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/heapster/issues/1908","repository_url":"https://api.github.com/repos/kubernetes/heapster","labels_url":"https://api.github.com/repos/kubernetes/heapster/issues/1908/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/heapster/issues/1908/comments","events_url":"https://api.github.com/repos/kubernetes/heapster/issues/1908/events","html_url":"https://github.com/kubernetes/heapster/issues/1908","id":280872955,"number":1908,"title":"Hpa cannot get metrics from heapster","user":{"login":"charleslieferando","id":5978370,"avatar_url":"https://avatars2.githubusercontent.com/u/5978370?v=4","gravatar_id":"","url":"https://api.github.com/users/charleslieferando","html_url":"https://github.com/charleslieferando","followers_url":"https://api.github.com/users/charleslieferando/followers","following_url":"https://api.github.com/users/charleslieferando/following{/other_user}","gists_url":"https://api.github.com/users/charleslieferando/gists{/gist_id}","starred_url":"https://api.github.com/users/charleslieferando/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/charleslieferando/subscriptions","organizations_url":"https://api.github.com/users/charleslieferando/orgs","repos_url":"https://api.github.com/users/charleslieferando/repos","events_url":"https://api.github.com/users/charleslieferando/events{/privacy}","received_events_url":"https://api.github.com/users/charleslieferando/received_events","type":"User","site_admin":false},"labels":[{"id":764074732,"url":"https://api.github.com/repos/kubernetes/heapster/labels/sig/autoscaling","name":"sig/autoscaling","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-12-11T02:06:40Z","updated_at":"2017-12-29T17:08:19Z","closed_at":null,"author_association":"NONE","body":"I am having troubles getting horizontal pod autoscaling to work on my k8s cluster on aws as it seems it is unable to retrieve metrics from heapster\r\n\r\n```yaml\r\n$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.2\", GitCommit:\"bdaeafa71f6c7c04636251031f93464384d54963\", GitTreeState:\"clean\", BuildDate:\"2017-10-24T19:48:57Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.3\", GitCommit:\"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\", GitTreeState:\"clean\", BuildDate:\"2017-11-08T18:27:48Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\nI have heapster deployed and powering the kubernetes dashboard without issues. This is the heapster k8s manifest i have deployed\r\n\r\n```yaml\r\napiVersion: rbac.authorization.k8s.io/v1\r\nkind: ClusterRoleBinding\r\nmetadata:\r\n  name: heapster-binding\r\n  labels:\r\n    kubernetes.io/cluster-service: \"true\"\r\n    addonmanager.kubernetes.io/mode: Reconcile\r\nroleRef:\r\n  apiGroup: rbac.authorization.k8s.io\r\n  kind: ClusterRole\r\n  name: system:heapster\r\nsubjects:\r\n- kind: ServiceAccount\r\n  name: heapster\r\n  namespace: kube-system\r\n---\r\n# Heapster's pod_nanny monitors the heapster deployment & its pod(s), and scales\r\n# the resources of the deployment if necessary.\r\napiVersion: rbac.authorization.k8s.io/v1\r\nkind: Role\r\nmetadata:\r\n  name: system:pod-nanny\r\n  namespace: kube-system\r\n  labels:\r\n    kubernetes.io/cluster-service: \"true\"\r\n    addonmanager.kubernetes.io/mode: Reconcile\r\nrules:\r\n- apiGroups:\r\n  - \"\"\r\n  resources:\r\n  - pods\r\n  verbs:\r\n  - get\r\n- apiGroups:\r\n  - \"extensions\"\r\n  resources:\r\n  - deployments\r\n  verbs:\r\n  - get\r\n  - update\r\n---\r\napiVersion: rbac.authorization.k8s.io/v1\r\nkind: RoleBinding\r\nmetadata:\r\n  name: heapster-binding\r\n  namespace: kube-system\r\n  labels:\r\n    kubernetes.io/cluster-service: \"true\"\r\n    addonmanager.kubernetes.io/mode: Reconcile\r\nroleRef:\r\n  apiGroup: rbac.authorization.k8s.io\r\n  kind: Role\r\n  name: system:pod-nanny\r\nsubjects:\r\n- kind: ServiceAccount\r\n  name: heapster\r\n  namespace: kube-system\r\n---\r\n---\r\napiVersion: v1\r\nkind: ServiceAccount\r\nmetadata:\r\n  name: heapster\r\n  namespace: kube-system\r\n---\r\napiVersion: extensions/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n  name: heapster\r\n  namespace: kube-system\r\nspec:\r\n  replicas: 1\r\n  template:\r\n    metadata:\r\n      labels:\r\n        task: monitoring\r\n        k8s-app: heapster\r\n    spec:\r\n      serviceAccountName: heapster\r\n      containers:\r\n      - name: heapster\r\n        image: gcr.io/google_containers/heapster-amd64:v1.4.0\r\n        imagePullPolicy: IfNotPresent\r\n        command:\r\n        - /heapster\r\n        - --source=kubernetes.summary_api:''\r\n        - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086\r\n---\r\napiVersion: v1\r\nkind: Service\r\nmetadata:\r\n  labels:\r\n    task: monitoring\r\n    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)\r\n    # If you are NOT using this as an addon, you should comment out this line.\r\n    kubernetes.io/cluster-service: 'true'\r\n    kubernetes.io/name: Heapster\r\n  name: heapster\r\n  namespace: kube-system\r\nspec:\r\n  ports:\r\n  - port: 80\r\n    targetPort: 8082\r\n  selector:\r\n    k8s-app: heapster\r\n```\r\n\r\n\r\nI have deployed an application with horizontal pod autoscaling. However I can see from the `describe hpa` command for that deployment that there are issues  with the HPA unable to get resource metrics as shown below:\r\n\r\n```yaml\r\n$ kubectl describe hpa --namespace=staging avro-schema-registry-hpa \r\nName:                                                     avro-schema-registry-hpa\r\nNamespace:                                                staging\r\nLabels:                                                   app=avro-schema-registry\r\n                                                          confluent-version=3.3.1\r\n                                                          env=staging\r\n                                                          version=0.11.0.1\r\nAnnotations:                                              kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"autoscaling/v2beta1\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"avro-schema-registry\",\"confluent-ver...\r\nCreationTimestamp:                                        Thu, 30 Nov 2017 10:18:12 +0100\r\nReference:                                                Deployment/avro-schema-registry\r\nMetrics:                                                  ( current / target )\r\n  resource memory on pods  (as a percentage of request):  <unknown> / 90%\r\n  resource cpu on pods  (as a percentage of request):     <unknown> / 90%\r\nMin replicas:                                             3\r\nMax replicas:                                             7\r\nConditions:\r\n  Type           Status  Reason                   Message\r\n  ----           ------  ------                   -------\r\n  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale\r\n  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource memory: failed to get pod resource metrics: an error on the server (\"Error: 'dial tcp 10.40.0.8:8082: getsockopt: connection timed out'\\nTrying to reach: 'http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelSelector=app%3Davro-schema-registry%2Cconfluent-version%3D3.3.1%2Cenv%3Dstaging%2Cversion%3D0.11.0.1'\") has prevented the request from succeeding (get services http:heapster:)\r\nEvents:\r\n  Type     Reason                        Age               From                       Message\r\n  ----     ------                        ----              ----                       -------\r\n  Warning  FailedGetResourceMetric       30s (x4 over 1h)  horizontal-pod-autoscaler  unable to get metrics for resource memory: failed to get pod resource metrics: an error on the server (\"Error: 'dial tcp 10.40.0.8:8082: getsockopt: connection timed out'\\nTrying to reach: 'http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelSelector=app%3Davro-schema-registry%2Cconfluent-version%3D3.3.1%2Cenv%3Dstaging%2Cversion%3D0.11.0.1'\") has prevented the request from succeeding (get services http:heapster:)\r\n  Warning  FailedComputeMetricsReplicas  30s (x4 over 1h)  horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: failed to get pod resource metrics: an error on the server (\"Error: 'dial tcp 10.40.0.8:8082: getsockopt: connection timed out'\\nTrying to reach: 'http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelSelector=app%3Davro-schema-registry%2Cconfluent-version%3D3.3.1%2Cenv%3Dstaging%2Cversion%3D0.11.0.1'\") has prevented the request from succeeding (get services http:heapster:)\r\n```\r\n\r\nI can curl some those URLs from the k8s worker machines e.g. \r\n\r\n\r\n```yaml\r\n$ curl -vvv http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelSelector=app%3Davro-schema-registry%2Cconfluent-version%3D3.3.1%2Cenv%3Dstaging%2Cversion%3D0.11.0.1\r\n*   Trying 10.40.0.8...\r\n* Connected to 10.40.0.8 (10.40.0.8) port 8082 (#0)\r\n> GET /apis/metrics/v1alpha1/namespaces/staging/pods?labelSelector=app%3Davro-schema-registry%2Cconfluent-version%3D3.3.1%2Cenv%3Dstaging%2Cversion%3D0.11.0.1 HTTP/1.1\r\n> Host: 10.40.0.8:8082\r\n> User-Agent: curl/7.47.0\r\n> Accept: */*\r\n> \r\n< HTTP/1.1 200 OK\r\n< Content-Type: application/json\r\n< Date: Mon, 11 Dec 2017 02:00:33 GMT\r\n< Content-Length: 1535\r\n< \r\n{\r\n  \"metadata\": {},\r\n  \"items\": [\r\n   {\r\n    \"metadata\": {\r\n     \"name\": \"avro-schema-registry-6b5475fdcd-9x8gb\",\r\n     \"namespace\": \"staging\",\r\n     \"creationTimestamp\": \"2017-12-11T02:00:33Z\"\r\n    },\r\n    \"timestamp\": \"2017-12-11T02:00:00Z\",\r\n    \"window\": \"1m0s\",\r\n    \"containers\": [\r\n     {\r\n      \"name\": \"avro-schema-registry\",\r\n      \"usage\": {\r\n       \"cpu\": \"1m\",\r\n       \"memory\": \"561496Ki\"\r\n      }\r\n     },\r\n     {\r\n      \"name\": \"istio-proxy\",\r\n      \"usage\": {\r\n       \"cpu\": \"30m\",\r\n       \"memory\": \"40064Ki\"\r\n      }\r\n     }\r\n    ]\r\n   },\r\n   {\r\n    \"metadata\": {\r\n     \"name\": \"avro-schema-registry-6b5475fdcd-cwgnt\",\r\n     \"namespace\": \"staging\",\r\n     \"creationTimestamp\": \"2017-12-11T02:00:33Z\"\r\n    },\r\n    \"timestamp\": \"2017-12-11T02:00:00Z\",\r\n    \"window\": \"1m0s\",\r\n    \"containers\": [\r\n     {\r\n      \"name\": \"avro-schema-registry\",\r\n      \"usage\": {\r\n       \"cpu\": \"1m\",\r\n       \"memory\": \"551252Ki\"\r\n      }\r\n     },\r\n     {\r\n      \"name\": \"istio-proxy\",\r\n      \"usage\": {\r\n       \"cpu\": \"31m\",\r\n       \"memory\": \"40236Ki\"\r\n      }\r\n     }\r\n    ]\r\n   },\r\n   {\r\n    \"metadata\": {\r\n     \"name\": \"avro-schema-registry-6b5475fdcd-d78gx\",\r\n     \"namespace\": \"staging\",\r\n     \"creationTimestamp\": \"2017-12-11T02:00:33Z\"\r\n    },\r\n    \"timestamp\": \"2017-12-11T02:00:00Z\",\r\n    \"window\": \"1m0s\",\r\n    \"containers\": [\r\n     {\r\n      \"name\": \"avro-schema-registry\",\r\n      \"usage\": {\r\n       \"cpu\": \"1m\",\r\n       \"memory\": \"542496Ki\"\r\n      }\r\n     },\r\n     {\r\n      \"name\": \"istio-proxy\",\r\n      \"usage\": {\r\n       \"cpu\": \"32m\",\r\n       \"memory\": \"41296Ki\"\r\n      }\r\n     }\r\n    ]\r\n   }\r\n  ]\r\n* Connection #0 to host 10.40.0.8 left intact\r\n }\r\n\r\n```\r\n\r\nHowever it doesnt work when I try from within another pod in the cluster\r\n\r\n```yaml\r\nroot@kafka-connect-564599bcc9-4pqjg:/# curl -vvv http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelSelector=app%3Davro-schema-registry%2Cconfluent-version%3D3.3.1%2Cenv%3Dstaging%2Cversion%3D0.11.0.1\r\n* Hostname was NOT found in DNS cache\r\n*   Trying 10.40.0.8...\r\n* Connected to 10.40.0.8 (10.40.0.8) port 8082 (#0)\r\n> GET /apis/metrics/v1alpha1/namespaces/staging/pods?labelSelector=app%3Davro-schema-registry%2Cconfluent-version%3D3.3.1%2Cenv%3Dstaging%2Cversion%3D0.11.0.1 HTTP/1.1\r\n> User-Agent: curl/7.38.0\r\n> Host: 10.40.0.8:8082\r\n> Accept: */*\r\n> \r\n< HTTP/1.1 404 Not Found\r\n< date: Mon, 11 Dec 2017 02:00:45 GMT\r\n* Server envoy is not blacklisted\r\n< server: envoy\r\n< content-length: 0\r\n< \r\n* Connection #0 to host 10.40.0.8 left intact\r\n```\r\n\r\n\r\nYou can find the logs for my heapster deployment here: https://gist.github.com/charleslieferando/2a4768ce832a902a958e8dff6aba7113\r\n\r\n\r\nDoes someone know what might be the issue?  "},"comment":{"url":"https://api.github.com/repos/kubernetes/heapster/issues/comments/354472797","html_url":"https://github.com/kubernetes/heapster/issues/1908#issuecomment-354472797","issue_url":"https://api.github.com/repos/kubernetes/heapster/issues/1908","id":354472797,"user":{"login":"shadycuz","id":17090999,"avatar_url":"https://avatars1.githubusercontent.com/u/17090999?v=4","gravatar_id":"","url":"https://api.github.com/users/shadycuz","html_url":"https://github.com/shadycuz","followers_url":"https://api.github.com/users/shadycuz/followers","following_url":"https://api.github.com/users/shadycuz/following{/other_user}","gists_url":"https://api.github.com/users/shadycuz/gists{/gist_id}","starred_url":"https://api.github.com/users/shadycuz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shadycuz/subscriptions","organizations_url":"https://api.github.com/users/shadycuz/orgs","repos_url":"https://api.github.com/users/shadycuz/repos","events_url":"https://api.github.com/users/shadycuz/events{/privacy}","received_events_url":"https://api.github.com/users/shadycuz/received_events","type":"User","site_admin":false},"created_at":"2017-12-29T17:08:19Z","updated_at":"2017-12-29T17:08:19Z","author_association":"NONE","body":"@MaciekPytel Different errors in the event log?\r\n`unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)`\r\nbut  I will check into this."}}
{"actor":{"display_login":"shadycuz"}}