title: Received plaintext http traffic on an https channel, closing connection
body: <p>I have deployed ECK (using helm) on my k8s cluster and i am attempting to install elasticsearch following the docs. <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html" rel="nofollow noreferrer">https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html</a></p>
  <p>I have externally exposed service/elasticsearch-prod-es-http so that i can connect to it from outside of my k8s cluster. However as you can see when i try to connect to it either from curl or the browser i receive an error &quot;502 Bad Gateway&quot; error.</p>
  <pre><code>curl elasticsearch.dev.acme.com
  &lt;html&gt;
  &lt;head&gt;&lt;title&gt;502 Bad Gateway&lt;/title&gt;&lt;/head&gt;
  &lt;body&gt;
  &lt;center&gt;&lt;h1&gt;502 Bad Gateway&lt;/h1&gt;&lt;/center&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  <p>Upon checking the pod (elasticsearch-prod-es-default-0) i can see the following message repeated.</p>
  <blockquote>
  <p>{&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-04-27T13:12:20,048Z&quot;, &quot;level&quot;: &quot;WARN&quot;, &quot;component&quot;: &quot;o.e.x.s.t.n.SecurityNetty4HttpServerTransport&quot;, &quot;cluster.name&quot;: &quot;elasticsearch-prod&quot;, &quot;node.name&quot;: &quot;elasticsearch-prod-es-default-0&quot;, &quot;message&quot;: &quot;received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.0.5.81:9200, remoteAddress=/10.0.3.50:46380}&quot;, &quot;cluster.uuid&quot;: &quot;t0mRfv7kREGQhXW9DVM3Vw&quot;, &quot;node.id&quot;: &quot;nCyAItDmSqGZRa3lApsC6g&quot;  }</p>
  </blockquote>
  <p><strong>Can you help me understand why this is occuring and how to fix it?</strong></p>
  <p>I suspect it has something to do with my TLS configuration because when i disable TLS, im able to connect to it externally without issues. However in a production environment i think keeping TLS enabled is important?</p>
  <p>FYI i am able to port-forward the service and connect to it with curl using the -k flag.</p>
  <p><strong>What i have tried</strong></p>
  <ol>
  <li>I have tried adding my domain to the section as described here <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-http-settings-tls-sans.html#k8s-elasticsearch-http-service-san" rel="nofollow noreferrer">https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-http-settings-tls-sans.html#k8s-elasticsearch-http-service-san</a></li>
  <li>I have tried using openssl to generate a self signed certificate but that did not work. Trying to connect locally returns the following error message.</li>
  </ol>
  <blockquote>
  <p>curl -u &quot;elastic:$PASSWORD&quot; &quot;https://localhost:9200&quot;
  curl: (60) SSL certificate problem: unable to get local issuer certificate
  More details here: <a href="https://curl.haxx.se/docs/sslcerts.html" rel="nofollow noreferrer">https://curl.haxx.se/docs/sslcerts.html</a>
  curl failed to verify the legitimacy of the server and therefore could not
  establish a secure connection to it. To learn more about this situation and
  how to fix it, please visit the web page mentioned above.</p>
  </blockquote>
  <ol start="3">
  <li>I have tried generating a certificate using the tool <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/configuring-tls.html#tls-transport" rel="nofollow noreferrer">https://www.elastic.co/guide/en/elasticsearch/reference/7.9/configuring-tls.html#tls-transport</a></li>
  </ol>
  <blockquote>
  <p>bin/elasticsearch-certutil ca
  bin/elasticsearch-certutil cert --ca elastic-stack-ca.12 --pem</p>
  </blockquote>
  <p>Then using the .crt and .key generated i created a kubectl secret <code>elastic-tls-cert</code>. But again curling localhost without -k gave the following error:</p>
  <blockquote>
  <p>curl --cacert cacert.pem -u &quot;elastic:$PASSWORD&quot; -XGET &quot;https://localhost:9200&quot;
  curl: (60) SSL certificate problem: unable to get local issuer certificate
  More details here: <a href="https://curl.haxx.se/docs/sslcerts.html" rel="nofollow noreferrer">https://curl.haxx.se/docs/sslcerts.html</a>
  curl failed to verify the legitimacy of the server and therefore could not
  establish a secure connection to it. To learn more about this situation and
  how to fix it, please visit the web page mentioned above.</p>
  </blockquote>
  <p><strong>elasticsearch.yml</strong></p>
  <pre><code># This sample sets up an Elasticsearch cluster with 3 nodes.
  apiVersion: elasticsearch.k8s.elastic.co/v1
  kind: Elasticsearch
  metadata:
    name: elasticsearch-prod
    namespace: elastic-system
  spec:
    version: 7.12.0
    nodeSets:
    - name: default
      config:
        # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
        node.roles: [&quot;master&quot;, &quot;data&quot;, &quot;ingest&quot;, &quot;ml&quot;]
        # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
        node.store.allow_mmap: false
        xpack.security.enabled: true
      podTemplate:
        metadata:
          labels:
            # additional labels for pods
            foo: bar
        spec:
          nodeSelector: 
            acme/node-type: ops
  
          # this changes the kernel setting on the node to allow ES to use mmap
          # if you uncomment this init container you will likely also want to remove the
          # &quot;node.store.allow_mmap: false&quot; setting above
          # initContainers:
          # - name: sysctl
          #   securityContext:
          #     privileged: true
          #   command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          ###
          # uncomment the line below if you are using a service mesh such as linkerd2 that uses service account tokens for pod identification.
          # automountServiceAccountToken: true
          containers:
          - name: elasticsearch
            # specify resource limits and requests
            resources:
              limits:
                memory: 4Gi
                cpu: 1
            env:
            - name: ES_JAVA_OPTS
              value: &quot;-Xms2g -Xmx2g&quot;
      count: 3
    #   # request 2Gi of persistent data storage for pods in this topology element
      volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 250Gi
          storageClassName: elasticsearch
    # # inject secure settings into Elasticsearch nodes from k8s secrets references
    # secureSettings:
    # - secretName: ref-to-secret
    # - secretName: another-ref-to-secret
    #   # expose only a subset of the secret keys (optional)
    #   entries:
    #   - key: value1
    #     path: newkey # project a key to a specific path (optional)
    http:
      service:
        spec:
          # expose this cluster Service with a LoadBalancer
          type: NodePort
      # tls:
        # selfSignedCertificate:
          # add a list of SANs into the self-signed HTTP certificate
          subjectAltNames:
          # - ip: 192.168.1.2
          # - ip: 192.168.1.3
          # - dns: elasticsearch.dev.acme.com
          # - dns: localhost
        # certificate:
        #   # provide your own certificate
        #   secretName: elastic-tls-cert
  </code></pre>
  <p><strong>kubectl version</strong></p>
  <pre><code>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.4&quot;, GitCommit:&quot;e87da0bd6e03ec3fea7933c4b5263d151aafd07c&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-02-18T16:12:00Z&quot;, GoVersion:&quot;go1.15.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
  Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;19+&quot;, GitVersion:&quot;v1.19.6-eks-49a6c0&quot;, GitCommit:&quot;49a6c0bf091506e7bafcdb1b142351b69363355a&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2020-12-23T22:10:21Z&quot;, GoVersion:&quot;go1.15.5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
  </code></pre>
  <p><strong>helm list</strong></p>
  <pre><code>    NAME                NAMESPACE       REVISION    UPDATED                                 STATUS      CHART               APP VERSION
  elastic-operator    elastic-system  1           2021-04-26 11:18:02.286692269 +0100 BST deployed    eck-operator-1.5.0  1.5.0      
  </code></pre>
  <p><strong>resources</strong></p>
  <pre><code>pod/elastic-operator-0                1/1     Running   0          4h58m   10.0.5.142   ip-10-0-5-71.us-east-2.compute.internal    &lt;none&gt;           &lt;none&gt;
  pod/elasticsearch-prod-es-default-0   1/1     Running   0          9m5s    10.0.5.81    ip-10-0-5-71.us-east-2.compute.internal    &lt;none&gt;           &lt;none&gt;
  pod/elasticsearch-prod-es-default-1   1/1     Running   0          9m5s    10.0.1.128   ip-10-0-1-207.us-east-2.compute.internal   &lt;none&gt;           &lt;none&gt;
  pod/elasticsearch-prod-es-default-2   1/1     Running   0          9m5s    10.0.5.60    ip-10-0-5-71.us-east-2.compute.internal    &lt;none&gt;           &lt;none&gt;
  
  NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE    SELECTOR
  service/elastic-operator-webhook          ClusterIP   172.20.218.208   &lt;none&gt;        443/TCP          26h    app.kubernetes.io/instance=elastic-operator,app.kubernetes.io/name=elastic-operator
  service/elasticsearch-prod-es-default     ClusterIP   None             &lt;none&gt;        9200/TCP         9m5s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod,elasticsearch.k8s.elastic.co/statefulset-name=elasticsearch-prod-es-default
  service/elasticsearch-prod-es-http        NodePort    172.20.229.173   &lt;none&gt;        9200:30604/TCP   9m6s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod
  service/elasticsearch-prod-es-transport   ClusterIP   None             &lt;none&gt;        9300/TCP         9m6s   common.k8s.elastic.co/type=elasticsearch,elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-prod
  </code></pre>
  <p><strong>aws alb ingress controller</strong></p>
  <pre><code>apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: elastic-ingress
    namespace: elastic-system
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/group.name: &quot;&lt;redacted&gt;&quot;
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/listen-ports: '[{&quot;HTTP&quot;:80,&quot;HTTPS&quot;: 443}]'
      alb.ingress.kubernetes.io/certificate-arn: &lt;redacted&gt;
      alb.ingress.kubernetes.io/tags: Environment=prod,Team=dev
      alb.ingress.kubernetes.io/healthcheck-path: /health
      alb.ingress.kubernetes.io/healthcheck-interval-seconds: '300'
      alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=acme-aws-ingress-logs,access_logs.s3.prefix=dev-ingress
  spec:
    rules:
      - host: elasticsearch.dev.acme.com
        http:
          paths:
            - path: /*
              pathType: Prefix
              backend:
                service:
                  name: elasticsearch-prod-es-http
                  port:
                    number: 9200
      # - host: kibana.dev.acme.com
      #   http:
      #     paths:
      #       - path: /*
      #         pathType: Prefix
      #         backend:
      #           service:
      #             name: kibana-prod-kb-http
      #             port:
      #               number: 5601
  </code></pre>