title: Issues with Kubernetes multi-master using kubeadm on premises
body: <p>Following Kubernetes v1.11 <a href="https://kubernetes.io/docs/setup/independent/high-availability/" rel="nofollow noreferrer">documentation</a>, I have managed to setup Kubernetes high availability using kubeadm, stacked control plane nodes, with 3 masters running on-premises on CentOS7 VMs. But with no load-balancer available, I used Keepalived to set a failover virtual IP (10.171.4.12) for apiserver as described in Kubernetes v1.10 <a href="https://v1-10.docs.kubernetes.io/docs/setup/independent/high-availability/#set-up-master-load-balancer" rel="nofollow noreferrer">documentation</a>. As a result, my "kubeadm-config.yaml" used to boostrap the control planes had the following header:</p>
  
  <pre><code>apiVersion: kubeadm.k8s.io/v1alpha2  
  kind: MasterConfiguration  
  kubernetesVersion: v1.11.0  
  apiServerCertSANs:  
  - "10.171.4.12"  
  api:  
      controlPlaneEndpoint: "10.171.4.12:6443"  
  etcd:  
    ...
  </code></pre>
  
  <p>The configuration went fine with the following Warning that appeared when boostrapping all 3 Masters:  </p>
  
  <blockquote>
    <p><em>[endpoint] WARNING: port specified in api.controlPlaneEndpoint
    overrides api.bindPort in the controlplane address</em></p>
  </blockquote>
  
  <p>And this Warning when joining Workers:  </p>
  
  <blockquote>
    <p>[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will
    not be used, because the following required kernel modules are not
    loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs
    support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{}
    nf_conntrack_ipv4:{}] you can solve this problem with following
    methods:<br>
     1. Run 'modprobe -- ' to load missing kernel modules;<br>
     2. Provide the missing builtin kernel ipvs support</p>
  </blockquote>
  
  <h1>Afterwards, basic tests succeed:</h1>
  
  <ul>
  <li>When stopped, Keepalived is failing over to another Master and apiserver is always accessible (all kubectl commands succeed).  </li>
  <li>When stopping the main Master (with highest Keepalived preference), the deployment of apps is successful (tested with Kubernetes bootcamp) and everything syncs properly with the main Master when it is back online.  </li>
  <li>Kubernetes bootcamp application runs successfully, and all master &amp; worker nodes respond properly when the service exposing bootcamp with NodePort is curled.  </li>
  <li>Successfully deployed docker-registry as per <a href="https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/docker-registry" rel="nofollow noreferrer">https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/docker-registry</a></li>
  </ul>
  
  <h1>But then comes these issues:</h1>
  
  <ul>
  <li>Nginx Ingress Controller pod fails to run and enters state CrashLoopBackOff (refer to events below)</li>
  <li>After installing helm and tiller on any Master, all commands using "helm install" or "helm list" failed to execute (refer to command ouputs below)</li>
  </ul>
  
  <p>I am running Kubernetes v1.11.1 but kubeadm-config.yaml mentions 1.11.0, is this something I should worry about?</p>
  
  <p>Shall I not follow the official documentation and go for other alternatives such as described at: <a href="https://medium.com/@bambash/ha-kubernetes-cluster-via-kubeadm-b2133360b198" rel="nofollow noreferrer">https://medium.com/@bambash/ha-kubernetes-cluster-via-kubeadm-b2133360b198</a></p>
  
  <p>Note: same issue with new Kubernetes HA installation using the latest version 1.11.2 (three masters + one worker) and deployed nginx latest ingress controller release 0.18.0.</p>
  
  <h1>-- Nginx controller pod events &amp; logs:</h1>
  
  <pre><code>  Normal   Pulled     28m (x38 over 2h)  kubelet, node3.local  Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1" already present on machine  
    Warning  Unhealthy  7m (x137 over 2h)  kubelet, node3.local  Liveness probe failed: Get http://10.240.3.14:10254/healthz: dial tcp 10.240.3.14:10254: connect: connection refused  
    Warning  BackOff    2m (x502 over 2h)  kubelet, node3.local  Back-off restarting failed container  
  
  nginx version: nginx/1.13.12  
  W0809 14:05:46.171066       5 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.  
  I0809 14:05:46.171748       5 main.go:191] Creating API client for https://10.250.0.1:443
  </code></pre>
  
  <h1>-- helm command outputs:</h1>
  
  <pre><code>'# helm install ...  
  Error: no available release name found
  
  '# helm list  
  Error: Get https://10.250.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 10.250.0.1:443: i/o timeout
  </code></pre>
  
  <h1>-- kubernetes service &amp; endpoints:</h1>
  
  <pre><code># kubectl describe svc kubernetes
  Name:              kubernetes
  Namespace:         default
  Labels:            component=apiserver
                     provider=kubernetes
  Annotations:       &lt;none&gt;
  Selector:          &lt;none&gt;
  Type:              ClusterIP
  IP:                10.250.0.1
  Port:              https  443/TCP
  TargetPort:        6443/TCP
  Endpoints:         10.171.4.10:6443,10.171.4.8:6443,10.171.4.9:6443
  Session Affinity:  None
  Events:            &lt;none&gt;
  
  
  # kubectl get endpoints --all-namespaces
  NAMESPACE       NAME                      ENDPOINTS                                               AGE
  default         bc-svc                    10.240.3.27:8080                                        6d
  default         kubernetes                10.171.4.10:6443,10.171.4.8:6443,10.171.4.9:6443        7d
  ingress-nginx   default-http-backend      10.240.3.24:8080                                        4d
  kube-system     kube-controller-manager   &lt;none&gt;                                                  7d
  kube-system     kube-dns                  10.240.2.4:53,10.240.2.5:53,10.240.2.4:53 + 1 more...   7d
  kube-system     kube-scheduler            &lt;none&gt;                                                  7d
  kube-system     tiller-deploy             10.240.3.25:44134                                       5d 
  </code></pre>