title: Trouble initializing Vault
body: <p>I am trying to initialize Vault on Kubernetes deployed through Helm in HA mode with TLS and Consul as the backend on EKS.</p>
  <pre><code>Helm 3.0.2
  Kubernetes 1.17
  Vault 1.5.4
  Vault Helm Chart Version 0.8.0
  Consul 1.8.4
  Consul Helm Chart Version 0.25.0
  </code></pre>
  <p>The cluster is deployed through <code>eksctl</code> on AWS EKS.</p>
  <p>The error I am seeing</p>
  <pre><code>helm install -f values.yaml hashicorp/vault
  kubectl exec -it vault-o -- vault operator init
  Error initializing: Put &quot;https://127.0.0.1:8200/v1/sys/init&quot;: dial tcp 127.0.0.1:8200: connect: connection refused
  </code></pre>
  <p>My values.yaml</p>
  <pre><code>global:
    tlsDisable: false
  injector:
    metrics:
      enabled: true
  server:
    extraSecretEnvironmentVars:
      - envName: AWS_ACCESS_KEY_ID
        secretName: eks-creds
        secretKey: AWS_ACCESS_KEY_ID
      - envName: AWS_SECRET_ACCESS_KEY
        secretName: eks-creds
        secretKey: AWS_SECRET_ACCESS_KEY
    extraVolumes:
      - type: secret
        name: vault-tls
      - type: secret
        name: eks-creds
    standalone:
      enable: false
    ha:
      enable: true
      config: |
        ui = true
        api_addr = &quot;[::]:8200&quot;
        cluster_addr = &quot;[::]:8201&quot;
        listener &quot;tcp&quot; {
          tls_disable = 0
          tls_cert_file = &quot;/vault/userconfig/vault-tls/vault.crt&quot;
          tls_key_file = &quot;/vault/userconfig/vault-tls/vault.key&quot;
          tls_client_ca_file = &quot;/vault/userconfig/vault-tls/vault.ca&quot;
          address = &quot;[::]:8200&quot;
          cluster_address = &quot;[::]:8201&quot;
        }
        storage &quot;consul&quot; {
          path = &quot;vault&quot;
          address = &quot;HOST_IP:8501&quot;
        }
        disable_mlock = true
        service_registration &quot;kubernetes&quot; {}
  ui:
    enabled: true
  </code></pre>
  <p>I have tried changing <code>api_addr</code>, <code>cluster_addr</code>, <code>address</code>, and <code>cluster_address</code> over to <code>PID_IP</code>, <code>https://127.0.0.1:8200</code>, <code>vault.default.svc.cluster.local</code>, <code>https://0.0.0.0:8200</code> and even a few other variants. So many I really haven't kept track. All with the same issue. I exec into the container and check the <code>VAULT_ADDR</code> env var. It's always reflected as I set it in the config. Even updated the <code>VAULT_ADDR</code> in the container to the vault kubernetes service ClusterIP. So I doubt it is this.</p>
  <p>Some outputs in the vault container</p>
  <pre><code>/ $ env | grep VAULT_ADDR
  / $ VAULT_ADDR=https://127.0.0.1:8200
  / $ env | grep VAULT_CACERT
  / $ export VAULT_CACERT=&quot;/vault/userconfig/vault-tls/vault.ca&quot;
  / $ ls -lAh /vault/userconfig/vault-tls/
  total 0
  drwxr-sr-x    2 root     vault        100 Oct 27 18:21 ..2020_10_27_18_21_40.431431606
  lrwxrwxrwx    1 root     root          31 Oct 27 18:21 ..data -&gt; ..2020_10_27_18_21_40.431431606
  lrwxrwxrwx    1 root     root          15 Oct 27 18:21 vault.ca -&gt; ..data/vault.ca
  lrwxrwxrwx    1 root     root          16 Oct 27 18:21 vault.crt -&gt; ..data/vault.crt
  lrwxrwxrwx    1 root     root          16 Oct 27 18:21 vault.key -&gt; ..data/vault.key
  </code></pre>
  <p>I've tried checking the open/listening ports inside the container</p>
  <pre><code>/ $ netstat -tnlp
  Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  / $ 
  </code></pre>
  <p>Which seems strange. But how would any of my value overrides cause this? So I feel like this is somehow intentional with the way it's setup. But maybe not.</p>
  <pre><code>/ $ wget https://127.0.0.1:8200
  Connecting to 127.0.0.1:8200 (127.0.0.1:8200)
  wget: can't connect to remote host (127.0.0.1): Connection refused
  </code></pre>
  <p>And while <code>kubectl port-forward vault-0 8200</code></p>
  <pre><code>curl -v https://localhost:8200
  *   Trying ::1:8200...
  * Connected to localhost (::1) port 8200 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *   CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8200
  * Closing connection 0
  curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8200
  </code></pre>
  <p>Here is some kubernetes output</p>
  <pre><code>kubectl get pods
  vault-0                                          0/1     Running   0          4s
  vault-1                                          0/1     Running   0          4s
  vault-2                                          0/1     Running   0          4s
  vault-agent-injector-7bd76b66dd-8gjf4            1/1     Running   0          4s
  
  kubectl get service
  vault                                 ClusterIP   10.100.165.241   &lt;none&gt;        8200/TCP,8201/TCP                                                         2m1s
  vault-active                          ClusterIP   10.100.22.197    &lt;none&gt;        8200/TCP,8201/TCP                                                         2m1s
  vault-agent-injector-svc              ClusterIP   10.100.93.216    &lt;none&gt;        443/TCP                                                                   2m1s
  vault-internal                        ClusterIP   None             &lt;none&gt;        8200/TCP,8201/TCP                                                         2m1s
  vault-standby                         ClusterIP   10.100.96.141    &lt;none&gt;        8200/TCP,8201/TCP                                                         2m1s
  vault-ui                              ClusterIP   10.100.219.196   &lt;none&gt;        8200/TCP                                                                  2m1s
  
  kubectl logs vault-0
  
  WARNING! Unable to read storage migration status.
  2020-10-28T03:31:47.639Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
  2020-10-28T03:31:47.640Z [WARN]  storage.consul: appending trailing forward slash to path
  2020-10-28T03:31:47.640Z [WARN]  storage migration check error: error=&quot;Unexpected response code: 400&quot;
  </code></pre>
  <pre><code>kubectl describe pod vault-0
  Name:         vault-0
  Namespace:    default
  Priority:     0
  Node:         ip-172-16-3-151.ec2.internal/172.16.3.151
  Start Time:   Tue, 27 Oct 2020 20:31:46 -0700
  Labels:       app.kubernetes.io/instance=vault
                app.kubernetes.io/name=vault
                component=server
                controller-revision-hash=vault-5dd945bc6c
                helm.sh/chart=vault-0.8.0
                statefulset.kubernetes.io/pod-name=vault-0
  Annotations:  kubernetes.io/psp: eks.privileged
  Status:       Running
  IP:           172.16.3.19
  IPs:
    IP:           172.16.3.19
  Controlled By:  StatefulSet/vault
  Containers:
    vault:
      Container ID:  docker://1c70a35dbb2c870d21616e825200bb766fc10ab8076cd70931305f56386891fb
      Image:         vault:1.5.4
      Image ID:      docker-pullable://vault@sha256:121c1eb16a474f5a4c1d92256184dae333ab7284f8c744d4e2754300f84f68f0
      Ports:         8200/TCP, 8201/TCP, 8202/TCP
      Host Ports:    0/TCP, 0/TCP, 0/TCP
      Command:
        /bin/sh
        -ec
      Args:
        cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
        [ -n &quot;${HOST_IP}&quot; ] &amp;&amp; sed -Ei &quot;s|HOST_IP|${HOST_IP?}|g&quot; /tmp/storageconfig.hcl;
        [ -n &quot;${POD_IP}&quot; ] &amp;&amp; sed -Ei &quot;s|POD_IP|${POD_IP?}|g&quot; /tmp/storageconfig.hcl;
        [ -n &quot;${HOSTNAME}&quot; ] &amp;&amp; sed -Ei &quot;s|HOSTNAME|${HOSTNAME?}|g&quot; /tmp/storageconfig.hcl;
        [ -n &quot;${API_ADDR}&quot; ] &amp;&amp; sed -Ei &quot;s|API_ADDR|${API_ADDR?}|g&quot; /tmp/storageconfig.hcl;
        [ -n &quot;${TRANSIT_ADDR}&quot; ] &amp;&amp; sed -Ei &quot;s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g&quot; /tmp/storageconfig.hcl;
        [ -n &quot;${RAFT_ADDR}&quot; ] &amp;&amp; sed -Ei &quot;s|RAFT_ADDR|${RAFT_ADDR?}|g&quot; /tmp/storageconfig.hcl;
        /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl
  
      State:          Running
        Started:      Tue, 27 Oct 2020 20:31:47 -0700
      Ready:          False
      Restart Count:  0
      Readiness:      exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
      Environment:
        HOST_IP:                 (v1:status.hostIP)
        POD_IP:                  (v1:status.podIP)
        VAULT_K8S_POD_NAME:     vault-0 (v1:metadata.name)
        VAULT_K8S_NAMESPACE:    default (v1:metadata.namespace)
        VAULT_ADDR:             https://127.0.0.1:8200
        VAULT_API_ADDR:         https://$(POD_IP):8200
        SKIP_CHOWN:             true
        SKIP_SETCAP:            true
        HOSTNAME:               vault-0 (v1:metadata.name)
        VAULT_CLUSTER_ADDR:     https://$(HOSTNAME).vault-internal:8201
        HOME:                   /home/vault
        AWS_ACCESS_KEY_ID:      &lt;set to the key 'AWS_ACCESS_KEY_ID' in secret 'eks-creds'&gt;      Optional: false
        AWS_SECRET_ACCESS_KEY:  &lt;set to the key 'AWS_SECRET_ACCESS_KEY' in secret 'eks-creds'&gt;  Optional: false
      Mounts:
        /home/vault from home (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from vault-token-kfmn5 (ro)
        /vault/config from config (rw)
        /vault/userconfig/eks-creds from userconfig-eks-creds (ro)
        /vault/userconfig/vault-tls from userconfig-vault-tls (ro)
  Conditions:
    Type              Status
    Initialized       True
    Ready             False
    ContainersReady   False
    PodScheduled      True
  Volumes:
    config:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      vault-config
      Optional:  false
    userconfig-vault-tls:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  vault-tls
      Optional:    false
    userconfig-eks-creds:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  eks-creds
      Optional:    false
    home:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:
      SizeLimit:  &lt;unset&gt;
    vault-token-kfmn5:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  vault-token-kfmn5
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  &lt;none&gt;
  Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
  Events:
    Type     Reason     Age                  From                                   Message
    ----     ------     ----                 ----                                   -------
    Normal   Scheduled  2m16s                default-scheduler                      Successfully assigned default/vault-0 to ip-172-16-3-151.ec2.internal
    Normal   Pulled     2m15s                kubelet, ip-172-16-3-151.ec2.internal  Container image &quot;vault:1.5.4&quot; already present on machine
    Normal   Created    2m15s                kubelet, ip-172-16-3-151.ec2.internal  Created container vault
    Normal   Started    2m15s                kubelet, ip-172-16-3-151.ec2.internal  Started container vault
    Warning  Unhealthy  21s (x22 over 2m5s)  kubelet, ip-172-16-3-151.ec2.internal  Readiness probe failed: Error checking seal status: Get &quot;https://127.0.0.1:8200/v1/sys/seal-status&quot;: dial tcp 127.0.0.1:8200: connect: connection refused
  </code></pre>
  <p>I port forward to localhost and get a <code>PR_END_OF_FILE_ERROR</code> error in my browser.</p>
  <p>And I always have the same error. I understand why <code>vault operator init</code> is complaining. But I don't understand why that port is not allowing connection to it in regards to how vault has been configured. I'm running out of documentation to read and scowering the net for others with a similiar setup haveing the same error. Most of the time when I find others with this issue. They just have to export the <code>VAULT_ADDR</code> and it resolves the issue. Any ideas?</p>