title: Unable to scrape metrics from pods
body: <p>I am able to scrape Prometheus metrics from a Kubernetes service using this Prometheus job configuration:</p>
  
  <pre><code>- job_name: 'prometheus-potapi'
    static_configs:
    - targets: ['potapi-service.potapi:1234']
  </code></pre>
  
  <p>It uses Kubernetes DNS and it gives me the metrics from any of my three pods I use for my service.</p>
  
  <p>I would like to see the result for each pod. </p>
  
  <p>I am able to see the data I want using this configuration:</p>
  
  <pre><code>- job_name: 'prometheus-potapi-pod'
    static_configs:
    - targets: ['10.1.0.126:1234']
  </code></pre>
  
  <p>I have searched and experimented using the service discovery mechanism available in Prometheus. Unfortunately, I don't understand how it should be setup. The <a href="https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml" rel="nofollow noreferrer">service discovery reference</a> isn't really helpful if you don't know how it works.</p>
  
  <p>I am looking for an example where the job using the IP number is replaced with some service discovery mechanism. Specifying the IP was enough for me to see that the data I'm looking for is exposed.</p>
  
  <p>The pods I want to scrape metrics from all live in the same namespace, <code>potapi</code>. </p>
  
  <p>The metrics are always exposed through the same port, <code>1234</code>.</p>
  
  <p>Finally, the are all named like this:</p>
  
  <pre><code>potapi-deployment-754d96f855-lkh4x
  potapi-deployment-754d96f855-pslgg
  potapi-deployment-754d96f855-z2zj2 
  </code></pre>
  
  <p>When I do </p>
  
  <pre><code>kubectl describe pod potapi-deployment-754d96f855-pslgg -n potapi
  </code></pre>
  
  <p>I get this description:</p>
  
  <pre><code>Name:           potapi-deployment-754d96f855-pslgg
  Namespace:      potapi
  Node:           docker-for-desktop/192.168.65.3
  Start Time:     Tue, 07 Aug 2018 14:18:55 +0200
  Labels:         app=potapi
    pod-template-hash=3108529411
  Annotations:    &lt;none&gt;
  Status:         Running
  IP:             10.1.0.127
  Controlled By:  ReplicaSet/potapi-deployment-754d96f855
  Containers:
    potapi:
      Container ID:   docker://72a0bafbda9b82ddfc580d79488a8e3c480d76a6d17c43d7f7d7ab18458c56ee
      Image:          potapi-service
      Image ID:       docker://sha256:d64e94c2dda43c40f641008c122e6664845d73cab109768efa0c3619cb0836bb
      Ports:          4567/TCP, 4568/TCP, 1234/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Tue, 07 Aug 2018 14:18:57 +0200
      Ready:          True
      Restart Count:  0
      Environment:    &lt;none&gt;
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-4fttn (ro)
  Conditions:
    Type           Status
    Initialized    True
    Ready          True
    PodScheduled   True
  Volumes:
    default-token-4fttn:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  default-token-4fttn
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  &lt;none&gt;
  Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
    node.kubernetes.io/unreachable:NoExecute for 300s
  Events:          &lt;none&gt;
  </code></pre>
  
  <p>How would you rewrite the job definition given these prerequisites?</p>