title: spark yarn cluster Stack trace: ExitCodeException exitCode=13
body: <p>I try to run my program in <code>yarn cluster mode</code>.I'm 100% sure the class in fat jar by sbt exists.
  I have no clue why the spark always throw <code>Stack trace: ExitCodeException exitCode=13 error.</code></p>
  
  <p>And I follow the <code>trackping</code> page and see <code>java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test.</code></p>
  
  <p>Then I run the spark PI example in yarn cluster and make it. 
  In yarn client/local mode,it still failed with the same error：<code>java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test</code></p>
  
  <p>P.S. The spark conf in program looks like this:</p>
  
  <pre><code>  var sparkConf = new SparkConf()
      .setAppName("TiffDN2TOA")
  //    .setIfMissing("spark.master", masterUrl)
      .set("spark.executor.memory", "10g")
      .set("spark.kryoserializer.buffer.max", "1024")
  
    implicit val sc = new SparkContext(sparkConf)
  </code></pre>
  
  <pre><code>object test {
    val masterUrl = "local[*]"
    var sparkConf = new SparkConf()
      .setAppName("TiffDN2TOA")
      //    .setIfMissing("spark.master", masterUrl)
      .set("spark.executor.memory", "10g")
      .set("spark.kryoserializer.buffer.max", "1024")
  
    implicit val sc = new SparkContext(sparkConf)
  
    def main(args: Array[String]): Unit = {
      HadoopLandsatDN2ToaMethods.scenesDn2Toa(args(0),args(1))
    }
  }
  
  </code></pre>
  
  <pre><code>spark2-submit \
  &gt; --master yarn \
  &gt; --deploy-mode cluster \
  &gt; --class org.air.ebds.organize.geotrellisETLtoa.LandsatDN2Toa \
  &gt; --num-executors 4 \
  &gt; --executor-cores 4 \
  &gt; --executor-memory 10G \
  &gt; --driver-memory 12g \
  &gt; --conf "spark.kryoserializer.buffer.max=1024m spark.kryoserializer.buffer=1024m" \
  &gt; /root/Desktop/toa.jar \
  &gt; /root/Desktop/ebds_landsat8/LC08/122/031/LC08_L1TP_122031_20140727,/root/Desktop/ebds_landsat8/LC08/122/031/LC08_L1TP_122031_20140913,LC08_L1TP_122031_20141116 \
  &gt; file:///
  19/07/10 16:40:21 INFO client.RMProxy: Connecting to ResourceManager at bigdataone/192.168.1.151:8032
  19/07/10 16:40:21 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
  19/07/10 16:40:21 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (65536 MB per container)
  19/07/10 16:40:21 INFO yarn.Client: Will allocate AM container, with 13516 MB memory including 1228 MB overhead
  19/07/10 16:40:21 INFO yarn.Client: Setting up container launch context for our AM
  19/07/10 16:40:21 INFO yarn.Client: Setting up the launch environment for our AM container
  19/07/10 16:40:21 INFO yarn.Client: Preparing resources for our AM container
  19/07/10 16:40:21 INFO yarn.Client: Uploading resource file:/root/Desktop/toa.jar -&gt; hdfs://bigdataone:8020/user/root/.sparkStaging/application_1561542066113_0061/toa.jar
  19/07/10 16:40:33 INFO yarn.Client: Uploading resource file:/tmp/spark-0ccc5b92-4ef5-4f5e-944b-386abcbb5938/__spark_conf__3393474382108225503.zip -&gt; hdfs://bigdataone:8020/user/root/.sparkStaging/application_1561542066113_0061/__spark_conf__.zip
  19/07/10 16:40:33 INFO spark.SecurityManager: Changing view acls to: root
  19/07/10 16:40:33 INFO spark.SecurityManager: Changing modify acls to: root
  19/07/10 16:40:33 INFO spark.SecurityManager: Changing view acls groups to: 
  19/07/10 16:40:33 INFO spark.SecurityManager: Changing modify acls groups to: 
  19/07/10 16:40:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
  19/07/10 16:40:34 INFO yarn.Client: Submitting application application_1561542066113_0061 to ResourceManager
  19/07/10 16:40:34 INFO impl.YarnClientImpl: Submitted application application_1561542066113_0061
  19/07/10 16:40:35 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:35 INFO yarn.Client: 
       client token: N/A
       diagnostics: N/A
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: root.users.root
       start time: 1562748034023
       final status: UNDEFINED
       tracking URL: http://bigdataone:8088/proxy/application_1561542066113_0061/
       user: root
  19/07/10 16:40:36 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:37 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:38 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:39 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:40 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:41 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:42 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:43 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:44 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:45 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:46 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:47 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:48 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:49 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:50 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
  19/07/10 16:40:51 INFO yarn.Client: Application report for application_1561542066113_0061 (state: FAILED)
  19/07/10 16:40:51 INFO yarn.Client: 
       client token: N/A
       diagnostics: Application application_1561542066113_0061 failed 2 times due to AM Container for appattempt_1561542066113_0061_000002 exited with  exitCode: 13
  For more detailed output, check application tracking page:http://bigdataone:8088/proxy/application_1561542066113_0061/Then, click on links to logs of each attempt.
  Diagnostics: Exception from container-launch.
  Container id: container_1561542066113_0061_02_000001
  Exit code: 13
  Stack trace: ExitCodeException exitCode=13: 
      at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
      at org.apache.hadoop.util.Shell.run(Shell.java:507)
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
      at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  
  
  Container exited with a non-zero exit code 13
  Failing this attempt. Failing the application.
       ApplicationMaster host: N/A
       ApplicationMaster RPC port: -1
       queue: root.users.root
       start time: 1562748034023
       final status: FAILED
       tracking URL: http://bigdataone:8088/cluster/app/application_1561542066113_0061
       user: root
  Exception in thread "main" org.apache.spark.SparkException: Application application_1561542066113_0061 finished with failed status
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:1153)
      at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1568)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:892)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  19/07/10 16:40:51 INFO util.ShutdownHookManager: Shutdown hook called
  19/07/10 16:40:51 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-5e6eb641-9f2a-4351-947d-a3b4cf578f6d
  19/07/10 16:40:51 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0ccc5b92-4ef5-4f5e-944b-386abcbb5938
  
  </code></pre>
  
  <p>The application tracking page:<a href="http://bigdataone:8088/proxy/application_1561542066113_0061" rel="nofollow noreferrer">http://bigdataone:8088/proxy/application_1561542066113_0061</a> shows:</p>
  
  <pre><code>19/07/10 15:14:10 INFO util.SignalUtils: Registered signal handler for TERM
  19/07/10 15:14:10 INFO util.SignalUtils: Registered signal handler for HUP
  19/07/10 15:14:10 INFO util.SignalUtils: Registered signal handler for INT
  19/07/10 15:14:10 INFO spark.SecurityManager: Changing view acls to: yarn,root
  19/07/10 15:14:10 INFO spark.SecurityManager: Changing modify acls to: yarn,root
  19/07/10 15:14:10 INFO spark.SecurityManager: Changing view acls groups to: 
  19/07/10 15:14:10 INFO spark.SecurityManager: Changing modify acls groups to: 
  19/07/10 15:14:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, root); groups with view permissions: Set(); users  with modify permissions: Set(yarn, root); groups with modify permissions: Set()
  19/07/10 15:14:10 INFO yarn.ApplicationMaster: Preparing Local resources
  19/07/10 15:14:11 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1561542066113_0055_000002
  19/07/10 15:14:11 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
  19/07/10 15:14:11 ERROR yarn.ApplicationMaster: Uncaught exception: 
  java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:682)
      at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:448)
      at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:301)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:241)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:241)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:241)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:782)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
      at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:781)
      at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:240)
      at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:806)
      at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
  19/07/10 15:14:11 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test)
  19/07/10 15:14:11 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://bigdataone:8020/user/root/.sparkStaging/application_1561542066113_0055
  19/07/10 15:14:11 INFO util.ShutdownHookManager: Shutdown hook called
  </code></pre>