title: Zookeeper/SASL Checksum failed
body: <p>How do I fix the problem that generates this error:</p>
  
  <pre><code>WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1040] - Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
  javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
      at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199)
      at org.apache.zookeeper.server.ZooKeeperSaslServer.evaluateResponse(ZooKeeperSaslServer.java:50)
  </code></pre>
  
  <p>I have set up Zookeeper on an AWS EC2 instance. I have outlined the steps I followed to set up Kerberos and Zookeeper <a href="https://github.com/shafiquejamal/kafka-zookeeper-kerberos" rel="nofollow noreferrer">here</a>. Zookeeper seems to be working:</p>
  
  <pre><code>zookeeper@zookeeper-server-01:~/zk/zookeeper-3.4.11$ JVMFLAGS="-Djava.security.auth.login.config=/home/zookeeper/jaas/jaas.conf -Dsun.security.krb5.debug=true" bin/zkServer.sh start-foreground
  ...
  &gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  &gt;&gt;&gt; KrbAsRep cons in KrbAsReq.getReply zookeeper/zookeeper-server-01
  2017-12-22 00:21:52,308 [myid:] - INFO  [main:Login@297] - Server successfully logged in.
  2017-12-22 00:21:52,312 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
  2017-12-22 00:21:52,313 [myid:] - INFO  [Thread-1:Login$1@130] - TGT refresh thread started.
  2017-12-22 00:21:52,313 [myid:] - INFO  [Thread-1:Login@305] - TGT valid starting at:        Fri Dec 22 00:21:52 UTC 2017
  2017-12-22 00:21:52,313 [myid:] - INFO  [Thread-1:Login@306] - TGT expires:                  Fri Dec 22 10:21:52 UTC 2017
  2017-12-22 00:21:52,314 [myid:] - INFO  [Thread-1:Login$1@185] - TGT refresh sleeping until: Fri Dec 22 08:25:59 UTC 2017
  </code></pre>
  
  <p>When I try, however, to connect a <code>zkCli.sh</code> (running on a different EC2 instance) to it, the server closes the connection and outputs the checksum error above.</p>
  
  <p>The Zookeeper client appears to be able to connect to the Zookeeper server:</p>
  
  <pre><code>JVMFLAGS="-Djava.security.auth.login.config=/home/admin/Downloads/zookeeper-3.4.11/conf/zookeeper-test-client-jaas.conf -Dsun.security.krb5.debug=true" bin/zkCli.sh -server zookeeper-server-01.eigenroute.com:2181
  Connecting to zookeeper-server-01.eigenroute.com:2181
  2017-12-22 00:27:12,779 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=
  3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
  ...
  2017-12-22 00:27:12,788 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/admin/Downloads/zookeeper-3.4.11
  2017-12-22 00:27:12,789 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=zookeeper-server-01.eigenroute.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1de0aca6
  Welcome to ZooKeeper!
  JLine support is enabled
  ...
  &gt;&gt;&gt; KrbAsReq creating message
  [zk: zookeeper-server-01.eigenroute.com:2181(CONNECTING) 0] &gt;&gt;&gt; KrbKdcReq send: kdc=kerberos-server-01.eigenroute.com UDP:88, timeout=30000, number of retries =3, #bytes=166
  &gt;&gt;&gt; KDCCommunication: kdc=kerberos-server-01.eigenroute.com UDP:88, timeout=30000,Attempt =1, #bytes=166
  &gt;&gt;&gt; KrbKdcReq send: #bytes read=310
  &gt;&gt;&gt;Pre-Authentication Data:
  ...
  </code></pre>
  
  <p>The client receives an error about needing preauthorization, but then appears to be successfully logged in (does this mean successfully authenticated?) to ...the Zookeeper server? Or logged into Kerberos?:</p>
  
  <pre><code>...
  KRBError received: NEEDED_PREAUTH
  KrbAsReqBuilder: PREAUTH FAILED/REQ, re-send AS-REQ
  Using builtin default etypes for default_tkt_enctypes
  default etypes for default_tkt_enctypes: 18 17 16 23.
  Looking for keys for: zktestclient/eigenroute.com@EIGENROUTE.COM
  Added key: 17version: 3
  Added key: 18version: 3
  Looking for keys for: zktestclient/eigenroute.com@EIGENROUTE.COM
  Added key: 17version: 3
  Added key: 18version: 3
  Using builtin default etypes for default_tkt_enctypes
  default etypes for default_tkt_enctypes: 18 17 16 23.
  &gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  &gt;&gt;&gt; KrbAsReq creating message
  &gt;&gt;&gt; KrbKdcReq send: kdc=kerberos-server-01.eigenroute.com UDP:88, timeout=30000, number of retries =3, #bytes=253
  &gt;&gt;&gt; KDCCommunication: kdc=kerberos-server-01.eigenroute.com UDP:88, timeout=30000,Attempt =1, #bytes=253
  &gt;&gt;&gt; KrbKdcReq send: #bytes read=742
  &gt;&gt;&gt; KdcAccessibility: remove kerberos-server-01.eigenroute.com
  Looking for keys for: zktestclient/eigenroute.com@EIGENROUTE.COM
  Added key: 17version: 3
  Added key: 18version: 3
  &gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  &gt;&gt;&gt; KrbAsRep cons in KrbAsReq.getReply zktestclient/eigenroute.com
  2017-12-22 00:27:13,286 [myid:] - INFO  [main-SendThread(35.169.37.216:2181):Login@297] - Client successfully logged in.
  ...
  </code></pre>
  
  <p>The client then opens a socket connection to the Zookeeper server, and attempts to SASL authenticate to it:</p>
  
  <pre><code>...
  2017-12-22 00:27:13,312 [myid:] - INFO  [main-SendThread(35.169.37.216:2181):ClientCnxn$SendThread@103
  5] - Opening socket connection to server 35.169.37.216/35.169.37.216:2181. Will attempt to SASL-authen
  ticate using Login Context section 'Client'
  2017-12-22 00:27:13,317 [myid:] - INFO  [main-SendThread(35.169.37.216:2181):ClientCnxn$SendThread@877
  ] - Socket connection established to 35.169.37.216/35.169.37.216:2181, initiating session
  2017-12-22 00:27:13,359 [myid:] - INFO  [main-SendThread(35.169.37.216:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server 35.169.37.216/35.169.37.216:2181, sessionid = 0x1000436873a0001, negotiated timeout = 30000
  
  WATCHER::
  
  WatchedEvent state:SyncConnected type:None path:null
  Found ticket for zktestclient/eigenroute.com@EIGENROUTE.COM to go to krbtgt/EIGENROUTE.COM@EIGENROUTE.
  COM expiring on Fri Dec 22 10:27:13 UTC 2017
  Entered Krb5Context.initSecContext with state=STATE_NEW
  Found ticket for zktestclient/eigenroute.com@EIGENROUTE.COM to go to krbtgt/EIGENROUTE.COM@EIGENROUTE.
  COM expiring on Fri Dec 22 10:27:13 UTC 2017
  Service ticket not found in the subject
  &gt;&gt;&gt; Credentials acquireServiceCreds: same realm
  Using builtin default etypes for default_tgs_enctypes
  default etypes for default_tgs_enctypes: 18 17 16 23.
  &gt;&gt;&gt; CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
  &gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  &gt;&gt;&gt; KrbKdcReq send: kdc=kerberos-server-01.eigenroute.com UDP:88, timeout=30000, number of retries =3, #bytes=712
  &gt;&gt;&gt; KDCCommunication: kdc=kerberos-server-01.eigenroute.com UDP:88, timeout=30000,Attempt =1, #bytes=712
  &gt;&gt;&gt; KrbKdcReq send: #bytes read=678
  &gt;&gt;&gt; KdcAccessibility: remove kerberos-server-01.eigenroute.com
  &gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  &gt;&gt;&gt; KrbApReq: APOptions are 00000000 00000000 00000000 00000000
  &gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  Krb5Context setting mySeqNumber to: 50687702
  Krb5Context setting peerSeqNumber to: 0
  Created InitSecContextToken:
  0000: 01 00 6E 82 02 6B 30 82   02 67 A0 03 02 01 05 A1  ..n..k0..g......
  ...
  0260: 33 25 94 1F 60 93 E9 CF   7E EF 15 82 F8 6D ED 06  3%..`........m..
  0270: 43                                                 C
  
  2017-12-22 00:27:13,405 [myid:] - INFO  [main-SendThread(35.169.37.216:2181):ClientCnxn$SendThread@1161] - Unable to read additional data from server sessionid 0x1000436873a0001, likely server has closed socket, closing socket connection and attempting reconnect
  
  WATCHER::
  
  WatchedEvent state:Disconnected type:None path:null
  </code></pre>
  
  <p>So SASL authentication is not a complete failure, but the Zookeeper server closes the connection (on account of a checksum failure). </p>
  
  <p>UPDATE #1. In response to T-Heron's comment, the result of <code>nslookup zookeeper-server-01.eigenroute.com</code> on the client machine is:</p>
  
  <pre><code>Server:     172.31.0.2
  Address:    172.31.0.2#53
  
  Non-authoritative answer:
  Name:   zookeeper-server-01.eigenroute.com
  Address: 35.169.37.216
  </code></pre>
  
  <p>The DNS entry for <code>zookeeper-server-01.eigenroute.com</code> is:</p>
  
  <pre><code>zookeeper-server-01.eigenroute.com  30 minutes  A       
  35.169.37.216
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/bS2M5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bS2M5.png" alt="enter image description here"></a></p>
  
  <p>On the client machine, <code>/etc/hosts</code> contains:</p>
  
  <pre><code>127.0.1.1 ip-172-31-95-211.ec2.internal ip-172-31-95-211
  127.0.0.1 localhost
  34.239.197.36 kerberos-server-02
  
  # The following lines are desirable for IPv6 capable hosts
  ::1 ip6-localhost ip6-loopback
  fe00::0 ip6-localnet
  ff00::0 ip6-mcastprefix
  ff02::1 ip6-allnodes
  ff02::2 ip6-allrouters
  ff02::3 ip6-allhosts
  </code></pre>
  
  <p>(<code>kerberos-server-02</code> is misnamed, it is not a KDC, when I comment this line out the result is the same) and on the ZooKeeper server, <code>zookeeper-server-01.eigenroute.com</code>, <code>/etc/hosts</code> contains:</p>
  
  <pre><code>127.0.1.1 ip-172-31-88-14.ec2.internal ip-172-31-88-14
  127.0.0.1 localhost
  34.225.180.212 kerberos-server-01
  
  # The following lines are desirable for IPv6 capable hosts
  ::1 ip6-localhost ip6-loopback
  fe00::0 ip6-localnet
  ff00::0 ip6-mcastprefix
  ff02::1 ip6-allnodes
  ff02::2 ip6-allrouters
  ff02::3 ip6-allhosts
  </code></pre>
  
  <p>(the entry for <code>kerberos-server-01</code> doesn't need to be there - when I remove it the result is the same). </p>
  
  <p>Can someone explain how to solve the checksum failure? Thanks!</p>