title: Kerberized Hadoop Login failure for user ... LoginException: Checksum failed
body: <h2>Environment:</h2>
  
  <ul>
  <li>Hadoop 2.9.2</li>
  <li>Kerberos 5 release 1.15.1</li>
  <li>RHEL 7</li>
  </ul>
  
  <h2>Error</h2>
  
  <p>Exception in Hadoop datanode log that prevents startup.</p>
  
  <p>The log entry is:</p>
  
  <pre><code>INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@&lt;REALM&gt; from keytab /etc/security/keytabs/&lt;file.keytab&gt; javax.security.auth.login.LoginException: Checksum failed
  </code></pre>
  
  <p>Full Stack Trace:</p>
  
  <pre><code>org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@&lt;REALM&gt; from keytab /etc/security/keytabs/datanode.keytab javax.security.auth.login.LoginException: Checksum failed
          at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1104)
          at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:312)
          at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2596)
          at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2645)
          at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2789)
          at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:77)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
  Caused by: javax.security.auth.login.LoginException: Checksum failed
          at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:808)
          at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
          at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
          at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
          at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
          at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
          at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1095)
          ... 10 more
  Caused by: KrbException: Checksum failed
          at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)
          at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)
          at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
          at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:150)
          at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
          at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:308)
          at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447)
          at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:780)
          ... 23 more
  Caused by: java.security.GeneralSecurityException: Checksum failed
          at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
          at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
          at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)
          at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100)
          ... 30 more
  </code></pre>
  
  <h2>Diagnose</h2>
  
  <p>If <strong>kdiag</strong> is run to diagnose the problem:</p>
  
  <pre><code>bin/hadoop org.apache.hadoop.security.KDiag --principal namenode/_HOST@&lt;REALM&gt; --keytab /etc/security/keytab/namenode.keytab
  </code></pre>
  
  <p>The same exception as above is encountered...</p>
  
  <p>The last line of useful output is:</p>
  
  <pre><code>&gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  </code></pre>
  
  <p>Normal healthy output should be:</p>
  
  <pre><code>&gt;&gt;&gt; EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
  &gt;&gt;&gt; CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
  &gt;&gt;&gt; KrbAsRep cons in KrbAsReq.getReply datanode/_HOST
  </code></pre>
  
  <h2>/etc/krb5.conf</h2>
  
  <p>Contents of Kerberos Configuration</p>
  
  <pre><code>[logging]
   default = FILE:/var/log/krb5libs.log
   kdc = FILE:/var/log/krb5kdc.log
   admin_server = FILE:/var/log/kadmind.log
  
  [libdefaults]
    dns_lookup_realm = false
    dns_lookup_kdc = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    rdns = false
    forwardable = true
    # pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
    default_realm = EXAMPLE.COM
    default_ccache_name = KEYRING:persistent:%{uid}
  
  [realms]
    EXAMPLE.COM = {
    kdc = kdc.example.com
    admin_server = kdc.example.com
    dict_file = /usr/share/dict/words
  }
  [domain_realm]
  .example.com = EXAMPLE.COM
  example.com = EXAMPLE.COM
  </code></pre>