title: How do I get hasura websocket to work on my local Kubernetes cluster?
body: <p>I have set up a local K8s cluster under Windows like this:</p>
  <ol>
  <li>install docker for desktop</li>
  <li>in docker for desktop, enable kubernetes</li>
  <li>install the nginx ingress controller</li>
  </ol>
  <pre class="lang-sh prettyprint-override"><code>kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.29.0/deploy/static/mandatory.yaml
  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.29.0/deploy/static/provider/cloud-generic.yaml
  </code></pre>
  <ol start="4">
  <li>add the following domain to the hosts (<code>C:\Windows\System32\drivers\etc\hosts</code>)</li>
  </ol>
  <pre><code>127.0.0.1  localhost api.shopozor
  </code></pre>
  <p>I'm doing nothing special here, I keep everything to the default setup.</p>
  <p>Then, I deployed hasura to my cluster, with the following yamls (I am not showing the postgres deployments for the sake of conciseness):</p>
  <pre class="lang-yaml prettyprint-override"><code>---
  # Source: api/templates/secrets.yaml
  apiVersion: v1
  kind: Secret
  metadata:
    name: api
    labels:
      app.kubernetes.io/name: api
      helm.sh/chart: api-0.0.0
      app.kubernetes.io/instance: api
      app.kubernetes.io/version: &quot;0.0&quot;
      app.kubernetes.io/managed-by: Helm
  type: Opaque
  data:
    admin-secret: &quot;c2VjcmV0&quot;
  ---
  # Source: api/templates/service.yaml
  apiVersion: v1
  kind: Service
  metadata:
    name: api
    labels:
      app.kubernetes.io/name: api
      helm.sh/chart: api-0.0.0
      app.kubernetes.io/instance: api
      app.kubernetes.io/version: &quot;0.0&quot;
      app.kubernetes.io/managed-by: Helm
  spec:
    type: ClusterIP
    ports:
      - port: 8080
        targetPort: 8080
        # TODO: we cannot use string port because devspace doesn't support it in its UI
        # targetPort: http
        protocol: TCP
        name: http
    selector:
      app.kubernetes.io/name: api
      app.kubernetes.io/instance: api
  ---
  # Source: api/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: api
    labels:
      app.kubernetes.io/name: api
      helm.sh/chart: api-0.0.0
      app.kubernetes.io/instance: api
      app.kubernetes.io/version: &quot;0.0&quot;
      app.kubernetes.io/managed-by: Helm
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/name: api
        app.kubernetes.io/instance: api
    template:
      metadata:
        labels:
          app.kubernetes.io/name: api
          app.kubernetes.io/instance: api
      spec:
        serviceAccountName: api
        securityContext:
          {}
        initContainers:
        # App has to wait for the database to be online &quot;depends_on&quot; workaround
        - name: wait-for-db
          image: darthcabs/tiny-tools:1
          args:
          - /bin/bash
          - -c
          - &gt;
            set -x;
            while [[ &quot;$(nc -zv 'postgres' 5432 &amp;&gt; /dev/null; echo $?)&quot; != 0 ]]; do
              echo '.'
              sleep 15;
            done
        containers:
        - name: api
          securityContext:
              {}
          image: shopozor/graphql-engine:EM5Aya
          imagePullPolicy: 
          env:
          - name: POSTGRES_USER
            valueFrom:
              secretKeyRef:
                name: shared-postgresql
                key: postgresql-username
          - name: POSTGRES_DATABASE
            value: shopozor
          - name: POSTGRES_PASSWORD
            valueFrom:
              secretKeyRef:
                name: shared-postgresql
                key: postgresql-password
          - name: POSTGRES_HOST
            value: postgres
          - name: POSTGRES_PORT
            value: &quot;5432&quot;
          - name: HASURA_GRAPHQL_SERVER_PORT
            value: &quot;8080&quot;
          - name: HASURA_GRAPHQL_ENABLE_CONSOLE
            value: &quot;true&quot;
          - name: HASURA_GRAPHQL_ENABLED_LOG_TYPES
            value: startup, http-log, webhook-log, websocket-log, query-log
          - name: HASURA_GRAPHQL_ENABLE_TELEMETRY
            value: &quot;false&quot;
          - name: HASURA_GRAPHQL_CORS_DOMAIN
            value: &quot;*&quot;
          - name: HASURA_GRAPHQL_DISABLE_CORS
            value: &quot;false&quot;
          - name: HASURA_GRAPHQL_UNAUTHORIZED_ROLE
            value: incognito
          - name: HASURA_GRAPHQL_ADMIN_SECRET
            valueFrom:
              secretKeyRef:
                name: api
                key: admin-secret
          - name: HASURA_GRAPHQL_JWT_SECRET
            value: &quot;{\&quot;type\&quot;: \&quot;HS256\&quot;, \&quot;key\&quot;: \&quot;my-access-token-signing-key-secret\&quot;, \&quot;audience\&quot;: [\&quot;58640fbe-9a6c-11ea-bb37-0242ac130002\&quot;, \&quot;6e707590-9a6c-11ea-bb37-0242ac130002\&quot;], \&quot;claims_namespace\&quot;: \&quot;https://hasura.io/jwt/claims\&quot;, \&quot;claims_format\&quot;: \&quot;json\&quot;, \&quot;issuer\&quot;: \&quot;shopozor.com\&quot; }&quot;
          - name: HASURA_GRAPHQL_DATABASE_URL
            value: postgres://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@$(POSTGRES_HOST):$(POSTGRES_PORT)/$(POSTGRES_DATABASE)
          - name: FUNCTION_NAMESPACE
            value: dev
          ports:
          - name: http
            containerPort: 8080
            protocol: TCP
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
          resources:
              {}
  ---
  # Source: api/templates/ingress.yaml
  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    name: api
    labels:
      app.kubernetes.io/name: api
      helm.sh/chart: api-0.0.0
      app.kubernetes.io/instance: api
      app.kubernetes.io/version: &quot;0.0&quot;
      app.kubernetes.io/managed-by: Helm
    annotations:
      kubernetes.io/ingress.class: nginx
  spec:
    rules:
      - host: &quot;api.shopozor&quot;
        http:
          paths:
            - path: /
              backend:
                serviceName: api
                servicePort: 8080
  </code></pre>
  <p>Now, I have a nuxt frontend that tries to make use of hasura's websocket. I configured apollo in the standard way</p>
  <pre class="lang-js prettyprint-override"><code>//---
  // nuxt.config.js
  [...]
    // Give apollo module options
    apollo: {
      cookieAttributes: {
        expires: 7
      },
      includeNodeModules: true, 
      authenticationType: 'Basic', 
      clientConfigs: {
        default: '~/apollo/clientConfig.js'
      }
    },
  [...]
  //---
  // apollo/clientConfig.js
  import { InMemoryCache } from 'apollo-cache-inmemory'
  export default function (context) {
    return {
      httpLinkOptions: {
        uri: 'http://api.shopozor/v1/graphql',
        credentials: 'same-origin'
      },
      cache: new InMemoryCache(),
      wsEndpoint: 'ws://localhost:8080/v1/graphql'
    }
  }
  </code></pre>
  <p>Notice I need no particular header currently. I should be able to access the websocket without authorization token.</p>
  <p>Now, when I start my application, the websocket connection tries to initialize. If I port-forward my hasura service, then the above configuration is fine. The websocket connection seems to be working. At least, the hasura log shows</p>
  <pre class="lang-sh prettyprint-override"><code>2020-07-16T06:49:59.937386882Z {&quot;type&quot;:&quot;websocket-log&quot;,&quot;timestamp&quot;:&quot;2020-07-16T06:49:59.937+0000&quot;,&quot;level&quot;:&quot;info&quot;,&quot;detail&quot;:{&quot;event&quot;:{&quot;type&quot;:&quot;accepted&quot;},&quot;connection_info&quot;:{&quot;websocket_id&quot;:&quot;8437b784-1fce-4430-9ca9-a9e7517307f0&quot;,&quot;token_expiry&quot;:null,&quot;msg&quot;:null},&quot;user_vars&quot;:null}}
  </code></pre>
  <p>If I, however, change the <code>wsEndpoint</code> in the above configuration to use the ingress into my hasura instance,</p>
  <pre class="lang-js prettyprint-override"><code>wsEndpoint: 'ws://api.shopozor/v1/graphql'
  </code></pre>
  <p>then it doesn't work anymore. Instead, I continuously get a <code>404 Not Found</code>. I can, however, access the hasura console through <code>http://api.shopozor</code>. The hasura log shows</p>
  <pre class="lang-sh prettyprint-override"><code>2020-07-16T10:37:53.564657244Z {&quot;type&quot;:&quot;websocket-log&quot;,&quot;timestamp&quot;:&quot;2020-07-16T10:37:53.564+0000&quot;,&quot;level&quot;:&quot;error&quot;,&quot;detail&quot;:{&quot;event&quot;:{&quot;type&quot;:&quot;rejected&quot;,&quot;detail&quot;:{&quot;path&quot;:&quot;$&quot;,&quot;error&quot;:&quot;only '/v1/graphql', '/v1alpha1/graphql' are supported on websockets&quot;,&quot;code&quot;:&quot;not-found&quot;}},&quot;connection_info&quot;:{&quot;websocket_id&quot;:&quot;5e031467-fb5c-460d-b2a5-11f1e21f22e7&quot;,&quot;token_expiry&quot;:null,&quot;msg&quot;:null},&quot;user_vars&quot;:null}}
  </code></pre>
  <p>So I googled a lot, found some information on annotations I should use in my ingresses, but nothing worked. What am I missing here? Do I need a particular nginx ingress controller configuration? Do I need to pass some special annotations to my hasura ingress? What do I need to do to make it work?</p>
  <h2>EDIT</h2>
  <p>To answer a question to this post, here's my the ingress to hasura I have applied on my cluster:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: api
      meta.helm.sh/release-namespace: dev
    labels:
      app.kubernetes.io/instance: api
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: api
      app.kubernetes.io/version: '0.0'
      helm.sh/chart: api-0.0.0
    name: api
  spec:
    rules:
      - host: api.shopozor
        http:
          paths:
            - backend:
                serviceName: api
                servicePort: 8080
              path: /
  </code></pre>
  <h2>EDIT 2</h2>
  <p>With the following ingress for my hasura instance</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: api
      meta.helm.sh/release-namespace: dev
      nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_http_version 1.1;
        proxy_set_header Upgrade &quot;websocket&quot;;
        proxy_set_header Connection &quot;Upgrade&quot;;
    labels:
      app.kubernetes.io/instance: api
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: api
      app.kubernetes.io/version: '0.0'
      helm.sh/chart: api-0.0.0
    name: api
  spec:
    rules:
      - host: api.shopozor
        http:
          paths:
            - backend:
                serviceName: api
                servicePort: 8080
              path: /
  </code></pre>
  <p>the frontend application still has the same issue with websockets. In addition, it cannot really connect to hasura anymore. Instead, I get the following errors:</p>
  <pre><code> ERROR  Network error: Unexpected token &lt; in JSON at position 0                                                                                                                                         23:17:06
  
    at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
    at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1588:34
    at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2008:15
    at Set.forEach (&lt;anonymous&gt;)
    at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2006:26
    at Map.forEach (&lt;anonymous&gt;)
    at QueryManager.broadcastQueries (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2004:20)
    at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1483:29
    at runMicrotasks (&lt;anonymous&gt;)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
  
  Global error handler                                                                                                                                                                                    23:17:06
  
   ERROR  Network error: Unexpected token &lt; in JSON at position 0                                                                                                                                         23:17:06
  
    at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
    at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1486:27
    at runMicrotasks (&lt;anonymous&gt;)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
  </code></pre>
  <p>as well as</p>
  <pre><code>client.js?06a0:49 ApolloError: Network error: Unexpected token &lt; in JSON at position 0
      at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
      at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1486:27
      at runMicrotasks (&lt;anonymous&gt;)
      at processTicksAndRejections (internal/process/task_queues.js:97:5) {
    graphQLErrors: [],
    networkError: SyntaxError [ServerParseError]: Unexpected token &lt; in JSON at position 0
        at JSON.parse (&lt;anonymous&gt;)
        at D:\workspace\shopozor\services\node_modules\apollo-link-http-common\lib\index.js:35:25
        at runMicrotasks (&lt;anonymous&gt;)
        at processTicksAndRejections (internal/process/task_queues.js:97:5) {
      name: 'ServerParseError',
      response: Body {
        url: 'http://api.shopozor/v1/graphql/',
        status: 404,
        statusText: 'Not Found',
        headers: [Headers],
        ok: false,
        body: [PassThrough],
        bodyUsed: true,
        size: 0,
        timeout: 0,
        _raw: [Array],
        _abort: false,
        _bytes: 153
      },
      statusCode: 404,
      bodyText: '&lt;html&gt;\r\n' +
        '&lt;head&gt;&lt;title&gt;404 Not Found&lt;/title&gt;&lt;/head&gt;\r\n' +
        '&lt;body&gt;\r\n' +
        '&lt;center&gt;&lt;h1&gt;404 Not Found&lt;/h1&gt;&lt;/center&gt;\r\n' +
        '&lt;hr&gt;&lt;center&gt;nginx/1.17.8&lt;/center&gt;\r\n' +
        '&lt;/body&gt;\r\n' +
        '&lt;/html&gt;\r\n'
    },
    message: 'Network error: Unexpected token &lt; in JSON at position 0',
    extraInfo: undefined
  }
  </code></pre>
  <p>and</p>
  <pre><code>index.js?a6d6:111 OPTIONS http://api.shopozor/v1/graphql/ net::ERR_ABORTED 404 (Not Found)
  </code></pre>
  <p>and</p>
  <pre><code>Access to fetch at 'http://api.shopozor/v1/graphql/' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
  </code></pre>
  <p>Without the new annotations</p>
  <pre class="lang-yaml prettyprint-override"><code>nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_http_version 1.1;
        proxy_set_header Upgrade &quot;websocket&quot;;
        proxy_set_header Connection &quot;Upgrade&quot;;
  </code></pre>
  <p>on my hasura ingress, the connection between my frontend and hasura is working fine, except for the websockets.</p>
  <h2>EDIT 3</h2>
  <p>I tried the following two ingresses, with no success:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: api
      meta.helm.sh/release-namespace: dev
      nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
      nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
      nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_http_version 1.1;
        proxy_set_header Upgrade &quot;websocket&quot;;
        proxy_set_header Connection &quot;Upgrade&quot;;
    labels:
      app.kubernetes.io/instance: api
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: api
      app.kubernetes.io/version: '0.0'
      helm.sh/chart: api-0.0.0
    name: api
  spec:
    rules:
      - host: api.shopozor
        http:
          paths:
            - backend:
                serviceName: api
                servicePort: 8080
              path: /
  </code></pre>
  <p>and</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: api
      meta.helm.sh/release-namespace: dev
      nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
      nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
    labels:
      app.kubernetes.io/instance: api
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: api
      app.kubernetes.io/version: '0.0'
      helm.sh/chart: api-0.0.0
    name: api
  spec:
    rules:
      - host: api.shopozor
        http:
          paths:
            - backend:
                serviceName: api
                servicePort: 8080
              path: /
  </code></pre>
  <p>In the latter case, I just get the error</p>
  <pre><code>WebSocket connection to 'ws://api.shopozor/v1/graphql/' failed: Error during WebSocket handshake: Unexpected response code: 404
  </code></pre>
  <p>while the graphql endpoint is functional. In the former case, I can't access to anything on the hasura instance and I get the websocket handshake issue above (so no graphql and no websocket working).</p>
  <h2>EDIT 4</h2>
  <p>With my api ingress configuration (without any additional nginx annotations like the above: <code>nginx.ingress.kubernetes.io/proxy-read-timeout</code>, <code> nginx.ingress.kubernetes.io/proxy-send-timeout</code>, <code>nginx.ingress.kubernetes.io/configuration-snippet</code>), if I do this:</p>
  <pre class="lang-sh prettyprint-override"><code>curl -i -N -H &quot;Connection: Upgrade&quot; \
   -H &quot;Upgrade: websocket&quot; \
   -H &quot;Origin: http://localhost:3000&quot; \
   -H &quot;Host: api.shopozor&quot; \ 
   -H &quot;Sec-Websocket-Version: 13&quot; \
   -H &quot;Sec-WebSocket-Key: B8KgbaRLCMNCREjE5Kvg1w==&quot; \ 
   -H &quot;Sec-WebSocket-Protocol: graphql-ws&quot; \
   -H &quot;Accept-Encoding: gzip, deflate&quot; \
   -H &quot;Accept-Language: en-US,en;q=0.9&quot; \
   -H &quot;Cache-Control: no-cache&quot; \
   -H &quot;Pragma: no-cache&quot; \
   -H &quot;Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits&quot; \
   -H &quot;User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.49 Safari/537.36&quot; \
   http://api.shopozor/v1/graphql
  </code></pre>
  <p>then hasura is happy:</p>
  <pre><code>2020-07-28T07:42:28.903877263Z {&quot;type&quot;:&quot;websocket-log&quot;,&quot;timestamp&quot;:&quot;2020-07-28T07:42:28.894+0000&quot;,&quot;level&quot;:&quot;info&quot;,&quot;detail&quot;:{&quot;event&quot;:{&quot;type&quot;:&quot;accepted&quot;},&quot;connection_info&quot;:{&quot;websocket_id&quot;:&quot;94243bde-41c4-42c8-8d8f-355c47a3492e&quot;,&quot;token_expiry&quot;:null,&quot;msg&quot;:null},&quot;user_vars&quot;:null}}
  </code></pre>
  <p>The headers in my curl above are the very same headers as those sent by my frontend app. Any clue on what I am doing wrong? The difference between the frontend calls and this curl is that in the frontend I define the websocket endpoint to be <code>ws://api.shopozor/v1/graphql</code> while I curl <code>http://api.shopozor/v1/graphql</code>. I cannot, in apollo vue, set the <code>wsEndpoint</code> to <code>http://api.shopozor/v1/graphql</code>. I get an error.</p>