title: kubectl port-forward: "pod does not exist" at the first time running?
body: <p>Follow <a href="https://blog.hasura.io/sharing-a-local-registry-for-minikube-37c7240d0615" rel="nofollow noreferrer">this</a> guide, I'm trying to start <code>minikube</code> and forward port at the boot time.</p>
  
  <p>My script:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  
  set -eux
  
  export PATH=/usr/local/bin:$PATH
  
  minikube status || minikube start
  minikube ssh 'grep docker.for.mac.localhost /etc/hosts || echo -e "127.0.0.1\tdocker.for.mac.localhost" | sudo tee -a /etc/hosts'
  minikube ssh 'test -f wait-for-it.sh || curl -O https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh'
  minikube ssh 'chmod +x wait-for-it.sh &amp;&amp; ./wait-for-it.sh 127.0.1.1:10250'
  POD=$(kubectl get po --namespace kube-system | awk '/kube-registry-v0/ { print $1 }')
  kubectl port-forward --namespace kube-system $POD 5000:5000
  </code></pre>
  
  <p>Everything works fine except that <code>kubectl port-forward</code> said that pod does not exist at the first time running:</p>
  
  <pre class="lang-sh prettyprint-override"><code>++ kubectl get po --namespace kube-system
  ++ awk '/kube-registry-v0/ { print $1 }'
  + POD=kube-registry-v0-qr2ml
  + kubectl port-forward --namespace kube-system kube-registry-v0-qr2ml 5000:5000
  error: error upgrading connection: unable to upgrade connection: pod does not exist
  </code></pre>
  
  <p>If I re-run:</p>
  
  <pre class="lang-sh prettyprint-override"><code>+ minikube status
  minikube: Running
  cluster: Running
  kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
  + minikube ssh 'grep docker.for.mac.localhost /etc/hosts || echo -e "127.0.0.1\tdocker.for.mac.localhost" | sudo tee -a /etc/hosts'
  127.0.0.1   docker.for.mac.localhost
  + minikube ssh 'test -f wait-for-it.sh || curl -O https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh'
  + minikube ssh 'chmod +x wait-for-it.sh &amp;&amp; ./wait-for-it.sh 127.0.1.1:10250'
  wait-for-it.sh: waiting 15 seconds for 127.0.1.1:10250
  wait-for-it.sh: 127.0.1.1:10250 is available after 0 seconds
  ++ kubectl get po --namespace kube-system
  ++ awk '/kube-registry-v0/ { print $1 }'
  + POD=kube-registry-v0-qr2ml
  + kubectl port-forward --namespace kube-system kube-registry-v0-qr2ml 5000:5000
  Forwarding from 127.0.0.1:5000 -&gt; 5000
  Forwarding from [::1]:5000 -&gt; 5000
  </code></pre>
  
  <p>I added a debug line before forwarding:</p>
  
  <pre class="lang-sh prettyprint-override"><code>kubectl describe pod --namespace kube-system $POD
  </code></pre>
  
  <p>and saw this:</p>
  
  <pre class="lang-sh prettyprint-override"><code>+ POD=kube-registry-v0-qr2ml
  + kubectl describe pod --namespace kube-system kube-registry-v0-qr2ml
  Name:       kube-registry-v0-qr2ml
  Namespace:  kube-system
  Node:       minikube/192.168.99.100
  Start Time: Thu, 28 Dec 2017 10:00:00 +0700
  Labels:     k8s-app=kube-registry
          version=v0
  Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"kube-system","name":"kube-registry-v0","uid":"317ecc42-eb7b-11e7-a8ce-...
  Status:     Running
  IP:     172.17.0.6
  Controllers:    ReplicationController/kube-registry-v0
  Containers:
    registry:
      Container ID:   docker://6e8f3f33399605758354f3f546996067d834459781235d51eef3ffa9c6589947
      Image:      registry:2.5.1
      Image ID:       docker-pullable://registry@sha256:946480a23b33480b8e7cdb89b82c1bd6accae91a8e66d017e21e8b56551f6209
      Port:       5000/TCP
      State:      Running
        Started:      Thu, 28 Dec 2017 13:22:44 +0700
  </code></pre>
  
  <p>Why <code>kubectl</code> said that it does not exist?</p>
  
  <hr>
  
  <p><strong>Fri Dec 29 04:58:06 +07 2017</strong></p>
  
  <p>Looking carefully at the events, I found something:</p>
  
  <pre><code>Events:
    FirstSeen     LastSeen        Count   From                    SubObjectPath                   Type            Reason                  Message
    ---------     --------        -----   ----                    -------------                   --------        ------                  -------
    20m           20m             1       kubelet, minikube                                       Normal          SuccessfulMountVolume   MountVolume.SetUp succ
  eeded for volume "image-store"
    20m           20m             1       kubelet, minikube                                       Normal          SuccessfulMountVolume   MountVolume.SetUp succ
  eeded for volume "default-token-fs7kr"
    20m           20m             1       kubelet, minikube                                       Normal          SandboxChanged          Pod sandbox changed, it will be killed and re-created.
    20m           20m             1       kubelet, minikube       spec.containers{registry}       Normal          Pulled                  Container image "registry:2.5.1" already present on machine
    20m           20m             1       kubelet, minikube       spec.containers{registry}       Normal          Created                 Created container
    20m           20m             1       kubelet, minikube       spec.containers{registry}       Normal          Started                 Started container
  </code></pre>
  
  <blockquote>
    <p>Pod sandbox changed, it will be killed and re-created.</p>
  </blockquote>
  
  <p>Before:</p>
  
  <pre><code>Containers:
    registry:
      Container ID:       docker://47c510dce00c6c2c29c9fe69665e1241c457d0666174a7723062c534e7229c58
      Image:              registry:2.5.1
      Image ID:           docker-pullable://registry@sha256:946480a23b33480b8e7cdb89b82c1bd6accae91a8e66d017e21e8b56551f6209
      Port:               5000/TCP
      State:              Running
        Started:          Thu, 28 Dec 2017 13:47:02 +0700
      Last State:         Terminated
        Reason:           Error
        Exit Code:        2
        Started:          Thu, 28 Dec 2017 13:22:44 +0700
        Finished:         Thu, 28 Dec 2017 13:45:18 +0700
      Ready:              True
      Restart Count:      14
  </code></pre>
  
  <p>After:</p>
  
  <pre><code>Containers:
    registry:
      Container ID:       docker://3a7da784d3d596796111348757725f5af22b47c5edd0fc29a4ffbb84f3f08956
      Image:              registry:2.5.1
      Image ID:           docker-pullable://registry@sha256:946480a23b33480b8e7cdb89b82c1bd6accae91a8e66d017e21e8b56551f6209
      Port:               5000/TCP
      State:              Running
        Started:          Thu, 28 Dec 2017 19:03:04 +0700
      Last State:         Terminated
        Reason:           Error
        Exit Code:        2
        Started:          Thu, 28 Dec 2017 13:47:02 +0700
        Finished:         Thu, 28 Dec 2017 19:00:48 +0700
      Ready:              True
      Restart Count:      15
  </code></pre>
  
  <p>minikube logs:</p>
  
  <blockquote>
    <p>Dec 28 22:15:41 minikube localkube[3250]: W1228 22:15:41.102038<br>
    3250 docker_sandbox.go:343] failed to read pod IP from plugin/docker:
    Couldn't find network status for kube-system/kube-registry-v0-qr2ml
    through plugin: invalid network status for</p>
  </blockquote>