title: Kubernetes metrics-server FailedDiscoveryCheck
body: <p>was hoping to get a little help, my Google-Fu didnt get me much closer. I'm trying to install the metrics server for my fedora-coreos kubernetes 4 node cluster like so:</p>
  
  <pre><code>kubectl apply -f deploy/kubernetes/
  clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
  clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
  rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
  apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
  serviceaccount/metrics-server created
  deployment.apps/metrics-server created
  service/metrics-server created
  clusterrole.rbac.authorization.k8s.io/system:metrics-server created
  clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
  </code></pre>
  
  <p>the service seems to never start</p>
  
  <pre><code>kubectl describe apiservice v1beta1.metrics.k8s.io
  Name:         v1beta1.metrics.k8s.io
  Namespace:
  Labels:       &lt;none&gt;
  Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                  {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.metrics.k8s.io"},"spec":{"...
  API Version:  apiregistration.k8s.io/v1
  Kind:         APIService
  Metadata:
    Creation Timestamp:  2020-03-04T16:53:33Z
    Resource Version:    1611816
    Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
    UID:                 65d9a56a-c548-4d7e-a647-8ce7a865a266
  Spec:
    Group:                     metrics.k8s.io
    Group Priority Minimum:    100
    Insecure Skip TLS Verify:  true
    Service:
      Name:            metrics-server
      Namespace:       kube-system
      Port:            443
    Version:           v1beta1
    Version Priority:  100
  Status:
    Conditions:
      Last Transition Time:  2020-03-04T16:53:33Z
      Message:               failing or missing response from https://10.3.230.59:443/apis/metrics.k8s.io/v1beta1: bad status from https://10.3.230.59:443/apis/metrics.k8s.io/v1beta1: 403
      Reason:                FailedDiscoveryCheck
      Status:                False
      Type:                  Available
  Events:                    &lt;none&gt;
  </code></pre>
  
  <p>Diagnosing I have found googling around:</p>
  
  <pre><code>kubectl get deploy,svc -n kube-system |egrep metrics-server
  deployment.apps/metrics-server   1/1     1            1           8m7s
  service/metrics-server   ClusterIP   10.3.230.59   &lt;none&gt;        443/TCP         8m7s
  
  kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
  Error from server (ServiceUnavailable): the server is currently unable to handle the request
  
  kubectl get all --all-namespaces | grep -i metrics-server
  kube-system      pod/metrics-server-75b5d446cd-zj4jm                              1/1     Running   0          9m11s
  kube-system   service/metrics-server   ClusterIP      10.3.230.59    &lt;none&gt;        443/TCP                                     9m11s
  kube-system      deployment.apps/metrics-server   1/1     1            1           9m11s
  kube-system      replicaset.apps/metrics-server-75b5d446cd   1         1         1       9m11s
  
  kubectl logs -f metrics-server-75b5d446cd-zj4jm -n kube-system
  I0304 16:53:36.475657       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
  W0304 16:53:38.229267       1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
  I0304 16:53:38.267760       1 secure_serving.go:116] Serving securely on [::]:4443
  
  kubectl get -n kube-system deployment metrics-server -o yaml | grep -i args -A 10
        {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"metrics-server","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"metrics-server"}},"template":{"metadata":{"labels":{"k8s-app":"metrics-server"},"name":"metrics-server"},"spec":{"containers":[{"args":["--cert-dir=/tmp","--secure-port=4443","--kubelet-insecure-tls","--kubelet-preferred-address-types=InternalIP"],"image":"k8s.gcr.io/metrics-server-amd64:v0.3.6","imagePullPolicy":"IfNotPresent","name":"metrics-server","ports":[{"containerPort":4443,"name":"main-port","protocol":"TCP"}],"securityContext":{"readOnlyRootFilesystem":true,"runAsNonRoot":true,"runAsUser":1000},"volumeMounts":[{"mountPath":"/tmp","name":"tmp-dir"}]}],"nodeSelector":{"beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64"},"serviceAccountName":"metrics-server","volumes":[{"emptyDir":{},"name":"tmp-dir"}]}}}}
    creationTimestamp: "2020-03-04T16:53:33Z"
    generation: 1
    labels:
      k8s-app: metrics-server
    name: metrics-server
    namespace: kube-system
    resourceVersion: "1611810"
    selfLink: /apis/apps/v1/namespaces/kube-system/deployments/metrics-server
    uid: 006e758e-bd33-47d7-8378-d3a8081ee8a8
  spec:
  --
        - args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP
          image: k8s.gcr.io/metrics-server-amd64:v0.3.6
          imagePullPolicy: IfNotPresent
          name: metrics-server
          ports:
          - containerPort: 4443
            name: main-port
  </code></pre>
  
  <p>finally my deployment config:</p>
  
  <pre><code> spec:
    selector:
      matchLabels:
        k8s-app: metrics-server
    template:
      metadata:
        name: metrics-server
        labels:
          k8s-app: metrics-server
      spec:
        serviceAccountName: metrics-server
        volumes:
        # mount in tmp so we can safely use from-scratch images and/or read-only containers
        - name: tmp-dir
          emptyDir: {}
        containers:
        - name: metrics-server
          image: k8s.gcr.io/metrics-server-amd64:v0.3.6
          command:
            - /metrics-server
            - --kubelet-insecure-tls
            - --kubelet-preferred-address-types=InternalIP
          args:
            - --cert-dir=/tmp
            - --secure-port=4443
            - --kubelet-insecure-tls
            - --kubelet-preferred-address-types=InternalIP
          ports:
          - name: main-port
            containerPort: 4443
            protocol: TCP
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          imagePullPolicy: IfNotPresent
          volumeMounts:
          - name: tmp-dir
            mountPath: /tmp
        hostNetwork: true
        nodeSelector:
          beta.kubernetes.io/os: linux
          kubernetes.io/arch: "amd64"
  </code></pre>
  
  <p>I'm at a loss of what it could be getting the metrics service to start and just get the basic kubectl top node to display any info all I get is</p>
  
  <pre><code>Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
  </code></pre>
  
  <p>I have searched the internet and tried adding the args: and command: lines but no luck</p>
  
  <pre><code>command:
             - /metrics-server
             - --kubelet-insecure-tls
             - --kubelet-preferred-address-types=InternalIP
  args:
            - --cert-dir=/tmp
            - --secure-port=4443
            - --kubelet-insecure-tls
            - --kubelet-preferred-address-types=InternalIP
  </code></pre>
  
  <p>Can anyone shed light on how to fix this? Thanks</p>
  
  <p>Pastebin log file
  <a href="https://pastebin.com/embed_js/FM6TnDhZ" rel="noreferrer">Log File</a></p>