title: KrbException connecting to Hadoop cluster with Zookeeper client - UNKNOWN_SERVER
body: <p>My Zookeeper client is having trouble connecting to the Hadoop cluster.</p>
  
  <p>This works fine from a Linux VM, but I am using a Mac.</p>
  
  <p>I set the <code>-Dsun.security.krb5.debug=true</code> flag on the JVM and get the following output:</p>
  
  <pre><code>Found ticket for solr@DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM@DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
  Entered Krb5Context.initSecContext with state=STATE_NEW
  Found ticket for solr@DDA.MYCO.COM to go to krbtgt/DDA.MYCO.COM@DDA.MYCO.COM expiring on Sat Apr 29 03:15:04 BST 2017
  Service ticket not found in the subject
  &gt;&gt;&gt; Credentials acquireServiceCreds: same realm
  Using builtin default etypes for default_tgs_enctypes
  default etypes for default_tgs_enctypes: 17 16 23.
  &gt;&gt;&gt; CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
  &gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
  &gt;&gt;&gt; KrbKdcReq send: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000, number of retries =3, #bytes=682
  &gt;&gt;&gt; KDCCommunication: kdc=oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com UDP:88, timeout=30000,Attempt =1, #bytes=682
  &gt;&gt;&gt; KrbKdcReq send: #bytes read=217
  &gt;&gt;&gt; KdcAccessibility: remove oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
  &gt;&gt;&gt; KDCRep: init() encoding tag is 126 req type is 13
  &gt;&gt;&gt;KRBError:
       cTime is Thu Dec 24 11:18:15 GMT 2015 1450955895000
       sTime is Fri Apr 28 15:15:06 BST 2017 1493388906000
       suSec is 925863
       error code is 7
       error Message is Server not found in Kerberos database
       cname is solr@DDA.MYCO.COM
       sname is zookeeper/oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com@DDA.MYCO.COM
       msgType is 30
  KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
      at sun.security.krb5.KrbTgsRep.&lt;init&gt;(KrbTgsRep.java:73)
      at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
      at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
      at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
      at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
      at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
      at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
      at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:366)
      at org.apache.zookeeper.client.ZooKeeperSaslClient$2.run(ZooKeeperSaslClient.java:363)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:362)
      at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:348)
      at org.apache.zookeeper.client.ZooKeeperSaslClient.sendSaslPacket(ZooKeeperSaslClient.java:420)
      at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:458)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1057)
  Caused by: KrbException: Identifier doesn't match expected value (906)
      at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
      at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
      at sun.security.krb5.internal.TGSRep.&lt;init&gt;(TGSRep.java:60)
      at sun.security.krb5.KrbTgsRep.&lt;init&gt;(KrbTgsRep.java:55)
      ... 18 more
  ERROR   2017-04-28 15:15:07,046 5539    org.apache.zookeeper.client.ZooKeeperSaslClient [main-SendThread(oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com:2181)]    
  An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed 
  [Caused by GSSException: No valid credentials provided 
  (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]) 
  occurred when evaluating Zookeeper Quorum Member's  received SASL token. 
  This may be caused by Java's being unable to resolve the Zookeeper Quorum Member's hostname correctly. 
  You may want to try to adding '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment. 
  Zookeeper Client will go to AUTH_FAILED state.
  </code></pre>
  
  <p>I've tested Kerberos config as follows:</p>
  
  <pre><code>&gt;kinit -kt /etc/security/keytabs/solr.headless.keytab solr
  &gt;klist
  Credentials cache: API:3451691D-7D5E-49FD-A27C-135816F33E4D
          Principal: solr@DDA.MYCO.COM
  
    Issued                Expires               Principal
  Apr 28 16:58:02 2017  Apr 29 04:58:02 2017  krbtgt/DDA.MYCO.COM@DDA.MYCO.COM
  </code></pre>
  
  <p>Following the instructions <a href="https://community.hortonworks.com/articles/75253/connect-hadoop-client-on-mac-os-x-to-kerberized-hd.html" rel="nofollow noreferrer">from hortonworks</a> I managed to get the kerberos ticket stored in a file:</p>
  
  <pre><code> &gt;klist -c FILE:/tmp/krb5cc_501
  Credentials cache: FILE:/tmp/krb5cc_501
      Principal: solr@DDA.MYCO.COM
  
  Issued                Expires               Principal
  Apr 28 17:10:25 2017  Apr 29 05:10:25 2017  krbtgt/DDA.MYCO.COM@DDA.MYCO.COM
  </code></pre>
  
  <p>Also I tried the suggested JVM option suggested in the stack trace (<code>-Dsun.net.spi.nameservice.provider.1=dns,sun</code>), but this led to a different error along the lines of <code>Client session timed out</code>, which suggests that this JVM param is preventing the client from connecting correctly in the first place.</p>
  
  <p>==EDIT==</p>
  
  <p>Seems that the Mac version of Kerberos is not the latest:</p>
  
  <pre><code>&gt; krb5-config --version
  Kerberos 5 release 1.7-prerelease
  </code></pre>
  
  <p>I just tried <code>brew install krb5</code> to install a newer version, then adjusting the path to point to the new version.</p>
  
  <pre><code>&gt; krb5-config --version
  Kerberos 5 release 1.15.1
  </code></pre>
  
  <p>This has had no effect whatsoever on the outcome.</p>
  
  <p>NB this works fine from a linux VM on my Mac, using exactly the same jaas.conf, keytab files, and krb5.conf.</p>
  
  <p>krb5.conf:</p>
  
  <pre><code>[libdefaults]
  renew_lifetime = 7d
  forwardable = true
    default_realm = DDA.MYCO.COM
    ticket_lifetime = 24h
    dns_lookup_realm = false
    dns_lookup_kdc = false
  
  
  [realms]
    DDA.MYCO.COM = {
      admin_server = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
      kdc = oc-10-252-132-139.nat-ucfc2z3b.usdv1.mycloud.com
    }
  </code></pre>
  
  <p>Reverse DNS:
  I checked that the FQDN hostname I'm connecting to can be found using a reverse DNS lookup:</p>
  
  <pre><code>&gt; host 10.252.132.160
  160.132.252.10.in-addr.arpa domain name pointer oc-10-252-132-160.nat-ucfc2z3b.usdv1.mycloud.com.
  </code></pre>
  
  <p>This is exactly as per the response to the same command from the linux VM.</p>
  
  <p>===WIRESHARK ANALYSIS===</p>
  
  <p>Using Wireshark configured to use the system key tabs allows a bit more detail in the analysis.</p>
  
  <p>Here I have found that a failed call looks like this:</p>
  
  <pre><code>client -&gt; host  AS-REQ
  host -&gt; client  AS-REP 
  client -&gt; host  AS-REQ
  host -&gt; client  AS-REP 
  client -&gt; host TGS-REQ  &lt;-- this call is detailed below
  host -&gt; client KRB error KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
  </code></pre>
  
  <p>The erroneous TGS-REQ call shows the following:</p>
  
  <pre><code>Kerberos
  tgs-req
      pvno: 5
      msg-type: krb-tgs-req (12)
      padata: 1 item
      req-body
          Padding: 0
          kdc-options: 40000000 (forwardable)
          realm: DDA.MYCO.COM
          sname
              name-type: kRB5-NT-UNKNOWN (0)
              sname-string: 2 items
                  SNameString: zookeeper
                  SNameString: oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
          till: 1970-01-01 00:00:00 (UTC)
          nonce: 797021964
          etype: 3 items
              ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
              ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
              ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
  </code></pre>
  
  <p>Here is the corresponding successful call from the linux box, which is followed by several more exchanges.</p>
  
  <pre><code>Kerberos
      tgs-req
          pvno: 5
          msg-type: krb-tgs-req (12)
          padata: 1 item
          req-body
              Padding: 0
              kdc-options: 40000000 (forwardable)
              realm: DDA.MYCO.COM
              sname
                  name-type: kRB5-NT-UNKNOWN (0)
                  sname-string: 2 items
                      SNameString: zookeeper
                      SNameString: d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
              till: 1970-01-01 00:00:00 (UTC)
              nonce: 681936272
              etype: 3 items
                  ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
                  ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
                  ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
  </code></pre>
  
  <p>So it looks like the client is sending </p>
  
  <pre><code>oc-10-252-134-51.nat-ucfc2z3b.usdv1.mycloud.com
  </code></pre>
  
  <p>as the server host, when it should be sending:</p>
  
  <pre><code>d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
  </code></pre>
  
  <p>So the question is, how do I fix that?  Bear in mind this is a Java piece of code.</p>
  
  <p>My /etc/hosts has the following:</p>
  
  <pre><code>10.252.132.160 b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
  10.252.134.51  d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
  10.252.132.139 d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
  </code></pre>
  
  <p>And my krb5.conf file has:</p>
  
  <pre><code>kdc = d7cc18.ddapoc.ucfc2z3b.usdv1.mycloud.com
  kdc = b3e073.ddapoc.ucfc2z3b.usdv1.mycloud.com
  kdc = d59407.ddapoc.ucfc2z3b.usdv1.mycloud.com
  </code></pre>
  
  <p>I tried adding <code>-Dsun.net.spi.nameservice.provider.1=file,dns</code> as a JVM param but got the same result.  </p>