title: Setting up Kubernetes Autoscaling with Custom Metrics
body: <p>I am trying to set up autoscaling on a Kubernetes 1.2.3 (beta) cluster based on custom metrics. (I already tried CPU-based autoscaling on the cluster, and it worked fine.)</p>
  
  <p>I tried to follow their <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/custom-metrics.md" rel="nofollow">custom metrics proposal</a>, but I'm having problems in creating the necessary set-up.</p>
  
  <p>This is what I have done so far:</p>
  
  <ol>
  <li><p>Added a custom metrics annotation to the pod spec being deployed
  (similar to the configuration provided in their proposal):</p>
  
  <pre><code>apiVersion: v1
  kind: ReplicationController
  metadata:
    name: metrix
    namespace: "default"
  spec:
    replicas: 1
    template:
      metadata:
        labels:
          app: metrix
        annotations:
          metrics.alpha.kubernetes.io/custom-endpoints: &gt;
            [
              {
                "api": "prometheus",
                "path": "/status",
                "port": "9090",
                "names": ["test1"]
              },
              {
                "api": "prometheus",
                "path": "/metrics",
                "port": "9090"
                "names": ["test2"]
              }
            ]
      spec:
        containers:
        - name: metrix
          image: janaka/prometheus-ep:v1
          resources:
            requests:
              cpu: 400m
  </code></pre></li>
  <li><p>Created a Docker container tagged <code>janaka/prometheus-ep:v1</code> (local) running a Prometheus-compatible server on port 9090, with <code>/status</code> and <code>/metrics</code> endpoints</p></li>
  <li><p>Enabled custom metrics on the kubelet by appending <code>--enable-custom-metrics=true</code> to <code>KUBELET_OPTS</code> at <code>/etc/default/kubelet</code> (based on <a href="http://kubernetes.io/docs/admin/kubelet/" rel="nofollow">the kubelet CLI reference</a>) and restarted the kubelet</p></li>
  </ol>
  
  <p>All pods (in <code>default</code> and <code>kube-system</code> namespaces) are running, and the heapster pod log doesn't contain any 'anomalous' outputs either (except for a small glitch at startup, due to temporary unavailability of InfluxDB):</p>
  
  <pre><code>$ kubesys logs -f heapster-daftr
  
  I0427 05:07:45.807277       1 heapster.go:60] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
  I0427 05:07:45.807359       1 heapster.go:61] Heapster version 1.1.0-beta1
  I0427 05:07:45.807638       1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version "v1"
  I0427 05:07:45.807661       1 configs.go:61] Using kubelet port 10255
  E0427 05:08:15.847319       1 influxdb.go:185] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp xxx.xxx.xxx.xxx:8086: i/o timeout, will retry on use
  I0427 05:08:15.847376       1 influxdb.go:199] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
  I0427 05:08:15.847412       1 heapster.go:87] Starting with InfluxDB Sink
  I0427 05:08:15.847427       1 heapster.go:87] Starting with Metric Sink
  I0427 05:08:15.877349       1 heapster.go:166] Starting heapster on port 8082
  I0427 05:08:35.000342       1 manager.go:79] Scraping metrics start: 2016-04-27 05:08:00 +0000 UTC, end: 2016-04-27 05:08:30 +0000 UTC
  I0427 05:08:35.035800       1 manager.go:152] ScrapeMetrics: time: 35.209696ms size: 24
  I0427 05:08:35.044674       1 influxdb.go:177] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
  I0427 05:09:05.000441       1 manager.go:79] Scraping metrics start: 2016-04-27 05:08:30 +0000 UTC, end: 2016-04-27 05:09:00 +0000 UTC
  I0427 05:09:06.682941       1 manager.go:152] ScrapeMetrics: time: 1.682157776s size: 24
  I0427 06:43:38.767146       1 manager.go:79] Scraping metrics start: 2016-04-27 05:09:00 +0000 UTC, end: 2016-04-27 05:09:30 +0000 UTC
  I0427 06:43:38.810243       1 manager.go:152] ScrapeMetrics: time: 42.940682ms size: 1
  I0427 06:44:05.012989       1 manager.go:79] Scraping metrics start: 2016-04-27 06:43:30 +0000 UTC, end: 2016-04-27 06:44:00 +0000 UTC
  I0427 06:44:05.063583       1 manager.go:152] ScrapeMetrics: time: 50.368106ms size: 24
  I0427 06:44:35.002038       1 manager.go:79] Scraping metrics start: 2016-04-27 06:44:00 +0000 UTC, end: 2016-04-27 06:44:30 +0000 UTC
  </code></pre>
  
  <p>However, the custom endpoints are not being scraped. (I verified it by adding stderr logs for startup and endpoint handlers of my server; only the server initialization logs are displayed on kubectl logs of the pod.)</p>
  
  <p>As I'm a newbie to Kubernetes, any help is appreciated.</p>
  
  <p>(From what I understood from the proposal as well as <a href="https://github.com/kubernetes/kubernetes/issues/18352" rel="nofollow">this issue</a>, we don't have to run a separate Prometheus collector in the cluster as <strong>cAdvisor</strong> should already pull data from the endpoints defined in the pod spec. Is this true, or do I need a separate Prometheus collector as well?)</p>