title: istio getting "RBAC: access denied" even the servicerolebinding checked to be allowed
body: <p>I've been struggleing with istio... So here I am seeking help from the experts!</p>
  <h1>Background</h1>
  <p>I'm trying to deploy my kubeflow application for <strong>multi-tenency</strong> with <em>dex</em>.
  Refering to the <a href="https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/" rel="nofollow noreferrer">kubeflow offical document</a> with the <a href="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml" rel="nofollow noreferrer">manifest file from github</a></p>
  <p>Here is a list of component/version information</p>
  <ul>
  <li>I'm running kubernetes 1.15 on GKE</li>
  <li>Istio 1.1.6 been used in kubeflow for service meth</li>
  <li>Trying to deploy kubeflow 1.0 for ML</li>
  <li>Deployed dex 1.0 for authn</li>
  </ul>
  <p>With the manifest file I successfully deployed the kubeflow on my cluster. Here's what I've done.</p>
  <ul>
  <li>Deploy the kubeflow application on the cluster</li>
  <li>Deploy Dex with OIDC service to enable authn to google Oauth2.0</li>
  <li>Enable the RBAC</li>
  <li>create envoy filter to append header &quot;kubeflow-userid&quot; as the login user</li>
  </ul>
  <p>Here is a verification of step 3 and 4
  Check RBAC enabled and envoyfilter added for kubeflow-userid</p>
  <pre><code>[root@gke-client-tf leilichao]# k get clusterrbacconfigs -o yaml
  apiVersion: v1
  items:
  - apiVersion: rbac.istio.io/v1alpha1
    kind: ClusterRbacConfig
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {&quot;apiVersion&quot;:&quot;rbac.istio.io/v1alpha1&quot;,&quot;kind&quot;:&quot;ClusterRbacConfig&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;default&quot;},&quot;spec&quot;:{&quot;mode&quot;:&quot;ON&quot;}}
      creationTimestamp: &quot;2020-07-04T01:28:52Z&quot;
      generation: 2
      name: default
      resourceVersion: &quot;5986075&quot;
      selfLink: /apis/rbac.istio.io/v1alpha1/clusterrbacconfigs/default
      uid: db70920e-f364-40ec-a93b-a3364f88650f
    spec:
      mode: &quot;ON&quot;
  kind: List
  metadata:
    resourceVersion: &quot;&quot;
    selfLink: &quot;&quot;
  [root@gke-client-tf leilichao]# k get envoyfilter -n istio-system -o yaml
  apiVersion: v1
  items:
  - apiVersion: networking.istio.io/v1alpha3
    kind: EnvoyFilter
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {&quot;apiVersion&quot;:&quot;networking.istio.io/v1alpha3&quot;,&quot;kind&quot;:&quot;EnvoyFilter&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;labels&quot;:{&quot;app.kubernetes.io/component&quot;:&quot;oidc-authservice&quot;,&quot;app.kubernetes.io/instance&quot;:&quot;oidc-authservice-v1.0.0&quot;,&quot;app.kubernetes.io/managed-by&quot;:&quot;kfctl&quot;,&quot;app.kubernetes.io/name&quot;:&quot;oidc-authservice&quot;,&quot;app.kubernetes.io/part-of&quot;:&quot;kubeflow&quot;,&quot;app.kubernetes.io/version&quot;:&quot;v1.0.0&quot;},&quot;name&quot;:&quot;authn-filter&quot;,&quot;namespace&quot;:&quot;istio-system&quot;},&quot;spec&quot;:{&quot;filters&quot;:[{&quot;filterConfig&quot;:{&quot;httpService&quot;:{&quot;authorizationRequest&quot;:{&quot;allowedHeaders&quot;:{&quot;patterns&quot;:[{&quot;exact&quot;:&quot;cookie&quot;},{&quot;exact&quot;:&quot;X-Auth-Token&quot;}]}},&quot;authorizationResponse&quot;:{&quot;allowedUpstreamHeaders&quot;:{&quot;patterns&quot;:[{&quot;exact&quot;:&quot;kubeflow-userid&quot;}]}},&quot;serverUri&quot;:{&quot;cluster&quot;:&quot;outbound|8080||authservice.istio-system.svc.cluster.local&quot;,&quot;failureModeAllow&quot;:false,&quot;timeout&quot;:&quot;10s&quot;,&quot;uri&quot;:&quot;http://authservice.istio-system.svc.cluster.local&quot;}},&quot;statusOnError&quot;:{&quot;code&quot;:&quot;GatewayTimeout&quot;}},&quot;filterName&quot;:&quot;envoy.ext_authz&quot;,&quot;filterType&quot;:&quot;HTTP&quot;,&quot;insertPosition&quot;:{&quot;index&quot;:&quot;FIRST&quot;},&quot;listenerMatch&quot;:{&quot;listenerType&quot;:&quot;GATEWAY&quot;}}],&quot;workloadLabels&quot;:{&quot;istio&quot;:&quot;ingressgateway&quot;}}}
      creationTimestamp: &quot;2020-07-04T01:40:43Z&quot;
      generation: 1
      labels:
        app.kubernetes.io/component: oidc-authservice
        app.kubernetes.io/instance: oidc-authservice-v1.0.0
        app.kubernetes.io/managed-by: kfctl
        app.kubernetes.io/name: oidc-authservice
        app.kubernetes.io/part-of: kubeflow
        app.kubernetes.io/version: v1.0.0
      name: authn-filter
      namespace: istio-system
      resourceVersion: &quot;4715289&quot;
      selfLink: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/envoyfilters/authn-filter
      uid: e599ba82-315a-4fc1-9a5d-e8e35d93ca26
    spec:
      filters:
      - filterConfig:
          httpService:
            authorizationRequest:
              allowedHeaders:
                patterns:
                - exact: cookie
                - exact: X-Auth-Token
            authorizationResponse:
              allowedUpstreamHeaders:
                patterns:
                - exact: kubeflow-userid
            serverUri:
              cluster: outbound|8080||authservice.istio-system.svc.cluster.local
              failureModeAllow: false
              timeout: 10s
              uri: http://authservice.istio-system.svc.cluster.local
          statusOnError:
            code: GatewayTimeout
        filterName: envoy.ext_authz
        filterType: HTTP
        insertPosition:
          index: FIRST
        listenerMatch:
          listenerType: GATEWAY
      workloadLabels:
        istio: ingressgateway
  kind: List
  metadata:
    resourceVersion: &quot;&quot;
    selfLink: &quot;&quot;
  
  </code></pre>
  <h1>RBAC Issue problem analysis</h1>
  <p>After I finished my deployment. I performed below functional testing:</p>
  <ul>
  <li>I can login with my google account with google oauth</li>
  <li>I was able to create my own profile/namespace</li>
  <li>I was able to create a notebook server</li>
  <li>However I can <strong>NOT</strong> connect to the notebook server</li>
  </ul>
  <h2>RBAC Issue investigation</h2>
  <p>I'm getting &quot;RBAC: access denied&quot; error after I successfully created the notebook server on kubeflow and trying to connect the notebook server.
  I managed to updated the envoy log level and get the log below.</p>
  <pre><code>[2020-08-06 13:32:43.290][26][debug][rbac] [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:64] checking request: remoteAddress: 10.1.1.2:58012, localAddress: 10.1.2.66:8888, ssl: none, headers: ':authority', 'compliance-kf-system.ml'
  ':path', '/notebook/roger-l-c-lei/aug06/'
  ':method', 'GET'
  'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'
  'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
  'accept-encoding', 'gzip, deflate'
  'accept-language', 'en,zh-CN;q=0.9,zh;q=0.8'
  'cookie', 'authservice_session=MTU5NjY5Njk0MXxOd3dBTkZvMldsVllVMUZPU0VaR01sSk5RVlJJV2xkRFVrRTFTVUl5V0RKV1EwdEhTMU5QVjFCVlUwTkpSVFpYUlVoT1RGVlBUa0U9fN3lPBXDDSZMT9MTJRbG8jv7AtblKTE3r84ayeCYuKOk; _xsrf=2|1e6639f2|10d3ea0a904e0ae505fd6425888453f8|1596697030'
  'referer', 'http://compliance-kf-system.ml/jupyter/'
  'upgrade-insecure-requests', '1'
  'x-forwarded-for', '10.10.10.230'
  'x-forwarded-proto', 'http'
  'x-request-id', 'babbf884-4cec-93fd-aea6-2fc60d3abb83'
  'kubeflow-userid', 'roger.l.c.lei@XXXX.com'
  'x-istio-attributes', 'CjAKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEg8SDXJvZ2VyLWwtYy1sZWkKIwoYZGVzdGluYXRpb24uc2VydmljZS5uYW1lEgcSBWF1ZzA2Ck4KCnNvdXJjZS51aWQSQBI+a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTg5Y2Q0YmQ0Yy1kdnF3dC5pc3Rpby1zeXN0ZW0KQQoXZGVzdGluYXRpb24uc2VydmljZS51aWQSJhIkaXN0aW86Ly9yb2dlci1sLWMtbGVpL3NlcnZpY2VzL2F1ZzA2CkMKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBInEiVhdWcwNi5yb2dlci1sLWMtbGVpLnN2Yy5jbHVzdGVyLmxvY2Fs'
  'x-envoy-expected-rq-timeout-ms', '300000'
  'x-b3-traceid', '3bf35cca1f7b75e7a42a046b1c124b1f'
  'x-b3-spanid', 'a42a046b1c124b1f'
  'x-b3-sampled', '1'
  'x-envoy-original-path', '/notebook/roger-l-c-lei/aug06/'
  'content-length', '0'
  'x-envoy-internal', 'true'
  , dynamicMetadata: filter_metadata {
    key: &quot;istio_authn&quot;
    value {
    }
  }
  
  [2020-08-06 13:32:43.290][26][debug][rbac] [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:108] enforced denied
  </code></pre>
  <p>From the source code it looks like the allowed function is returnning false so it's giving the &quot;RBAC: access denied&quot; response.</p>
  <pre><code>  if (engine.has_value()) {
      if (engine-&gt;allowed(*callbacks_-&gt;connection(), headers,
                          callbacks_-&gt;streamInfo().dynamicMetadata(), nullptr)) {
        ENVOY_LOG(debug, &quot;enforced allowed&quot;);
        config_-&gt;stats().allowed_.inc();
        return Http::FilterHeadersStatus::Continue;
      } else {
        ENVOY_LOG(debug, &quot;enforced denied&quot;);
        callbacks_-&gt;sendLocalReply(Http::Code::Forbidden, &quot;RBAC: access denied&quot;, nullptr,
                                   absl::nullopt);
        config_-&gt;stats().denied_.inc();
        return Http::FilterHeadersStatus::StopIteration;
      }
    }
  </code></pre>
  <p>I took a search on the dumped envoy, it looks like the rule should be allowing any request with a header key as my mail address. Now I can confirm I've got that in my header from above log.</p>
  <pre><code>{
   &quot;name&quot;: &quot;envoy.filters.http.rbac&quot;,
   &quot;config&quot;: {
    &quot;rules&quot;: {
     &quot;policies&quot;: {
      &quot;ns-access-istio&quot;: {
       &quot;permissions&quot;: [
        {
         &quot;and_rules&quot;: {
          &quot;rules&quot;: [
           {
            &quot;any&quot;: true
           }
          ]
         }
        }
       ],
       &quot;principals&quot;: [
        {
         &quot;and_ids&quot;: {
          &quot;ids&quot;: [
           {
            &quot;header&quot;: {
             &quot;exact_match&quot;: &quot;roger.l.c.lei@XXXX.com&quot;
            }
           }
          ]
         }
        }
       ]
      }
     }
    }
   }
  }
  </code></pre>
  <p>With the understand that the envoy config that's been used to validate RBAC authz is from this config. And it's distributed to the sidecar by mixer, The log and code leads me to the rbac.istio.io config of servicerolebinding.</p>
  <pre><code>[root@gke-client-tf leilichao]# k get servicerolebinding -n roger-l-c-lei -o yaml
  apiVersion: v1
  items:
  - apiVersion: rbac.istio.io/v1alpha1
    kind: ServiceRoleBinding
    metadata:
      annotations:
        role: admin
        user: roger.l.c.lei@XXXX.com
      creationTimestamp: &quot;2020-07-04T01:35:30Z&quot;
      generation: 5
      name: owner-binding-istio
      namespace: roger-l-c-lei
      ownerReferences:
      - apiVersion: kubeflow.org/v1
        blockOwnerDeletion: true
        controller: true
        kind: Profile
        name: roger-l-c-lei
        uid: 689c9f04-08a6-4c51-a1dc-944db1a66114
      resourceVersion: &quot;23201026&quot;
      selfLink: /apis/rbac.istio.io/v1alpha1/namespaces/roger-l-c-lei/servicerolebindings/owner-binding-istio
      uid: bbbffc28-689c-4099-837a-87a2feb5948f
    spec:
      roleRef:
        kind: ServiceRole
        name: ns-access-istio
      subjects:
      - properties:
          request.headers[]: roger.l.c.lei@XXXX.com
    status: {}
  kind: List
  metadata:
    resourceVersion: &quot;&quot;
    selfLink: &quot;&quot;
  </code></pre>
  <p>I wanted to have a try updating this ServiceRoleBinding to validate some assumption since I can't debug the envoy source code and there's not enough log to show why exactly is the &quot;allow&quot; method returnning false.</p>
  <p>However I find myself cannot update the servicerolebinding. It resumes to its orriginal version everytime right after I finish editing it.</p>
  <p>I find that there's this istio-galley validatingAdmissionConfiguration(Code block below) that monitors these istio rbac resources.</p>
  <pre><code>[root@gke-client-tf leilichao]# k get validatingwebhookconfigurations istio-galley -oyaml
  apiVersion: admissionregistration.k8s.io/v1beta1
  kind: ValidatingWebhookConfiguration
  metadata:
    creationTimestamp: &quot;2020-08-04T15:00:59Z&quot;
    generation: 1
    labels:
      app: galley
      chart: galley
      heritage: Tiller
      istio: galley
      release: istio
    name: istio-galley
    ownerReferences:
    - apiVersion: extensions/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Deployment
      name: istio-galley
      uid: 11fef012-4145-49ac-a43c-2e1d0a460ea4
    resourceVersion: &quot;22484680&quot;
    selfLink: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations/istio-galley
    uid: 6f485e28-3b5a-4a3b-b31f-a5c477c82619
  webhooks:
  - admissionReviewVersions:
    - v1beta1
    clientConfig:
      caBundle: 
      .
      .
      .
      service:
        name: istio-galley
        namespace: istio-system
        path: /admitpilot
        port: 443
    failurePolicy: Fail
    matchPolicy: Exact
    name: pilot.validation.istio.io
    namespaceSelector: {}
    objectSelector: {}
    rules:
    - apiGroups:
      - config.istio.io
      apiVersions:
      - v1alpha2
      operations:
      - CREATE
      - UPDATE
      resources:
      - httpapispecs
      - httpapispecbindings
      - quotaspecs
      - quotaspecbindings
      scope: '*'
    - apiGroups:
      - rbac.istio.io
      apiVersions:
      - '*'
      operations:
      - CREATE
      - UPDATE
      resources:
      - '*'
      scope: '*'
    - apiGroups:
      - authentication.istio.io
      apiVersions:
      - '*'
      operations:
      - CREATE
      - UPDATE
      resources:
      - '*'
      scope: '*'
    - apiGroups:
      - networking.istio.io
      apiVersions:
      - '*'
      operations:
      - CREATE
      - UPDATE
      resources:
      - destinationrules
      - envoyfilters
      - gateways
      - serviceentries
      - sidecars
      - virtualservices
      scope: '*'
    sideEffects: Unknown
    timeoutSeconds: 30
  - admissionReviewVersions:
    - v1beta1
    clientConfig:
      caBundle: 
      .
      .
      .
      service:
        name: istio-galley
        namespace: istio-system
        path: /admitmixer
        port: 443
    failurePolicy: Fail
    matchPolicy: Exact
    name: mixer.validation.istio.io
    namespaceSelector: {}
    objectSelector: {}
    rules:
    - apiGroups:
      - config.istio.io
      apiVersions:
      - v1alpha2
      operations:
      - CREATE
      - UPDATE
      resources:
      - rules
      - attributemanifests
      - circonuses
      - deniers
      - fluentds
      - kubernetesenvs
      - listcheckers
      - memquotas
      - noops
      - opas
      - prometheuses
      - rbacs
      - solarwindses
      - stackdrivers
      - cloudwatches
      - dogstatsds
      - statsds
      - stdios
      - apikeys
      - authorizations
      - checknothings
      - listentries
      - logentries
      - metrics
      - quotas
      - reportnothings
      - tracespans
      scope: '*'
    sideEffects: Unknown
    timeoutSeconds: 30
  
  </code></pre>
  <h1>Long stroy short</h1>
  <p>I've been banging my head over this istio issue for more than 2 weeks. I'm sure there's planty of people felting the same trying to trouble shoot istio on k8s. Any suggestion is welcomed!
  Here's how I understand the problem, please correct me if I'm wrong:</p>
  <ul>
  <li>The log evidence showed the rbac rules is not allowing my access to the resource</li>
  <li>I need to update the rbac rules</li>
  <li>rules are distributed by mixer to the envoy container according to ServiceRoleBinding</li>
  <li>So I need to update the ServiceRoleBinding instead</li>
  <li>I cannot update the ServiceRoleBinding because either the validating admission webhook or the istio mixer is preventing me from doing it</li>
  </ul>
  <p>I've run into below problems where</p>
  <h2>I cannot update the ServiceRoleBinding even after I deleted the validating webhook</h2>
  <p>I tried to delete this validating webhook to update the servicerolebinding. The resource resumes right after I save the edit.
  The validating webhook is actually generated automatically from a configmap so I had to update that to update the webhook.</p>
  <h2>Is there some kind of cache in galley that mixer uses to distribute the config</h2>
  <p>I can't find any relevent log that indicates the rbac.istio.io resource is protected/validated by any service in the istio-system namespace.</p>
  <h2>How can I get the log of the MIXER</h2>
  <p>I need to understand which component exactly controls the policy. I managed to update the log level but failed to find anything useful</p>
  <h2>Most importantly How do I debug an envoy container</h2>
  <p>I need to debug the envoy app to understand why it's returnning false for the allow function.
  If we can not debug it easily. Is there a document that lets me update the code to add more log and build a new image to GCR so I can have another run and based on the log to see what's going on behind the scene.</p>