title: Run Spark in IntelliJ Idea on Standalone Cluster with Master on same Windows Machine
body: <p>I have been able to successfully run a Spark application in IntelliJ Idea when setting master as local[*]. However, when I set master to a separate instance of Spark an exception occurs.</p>
  <p>The SparkPi App I am trying to execute is below.</p>
  <pre><code>import scala.math.random
  
  import org.apache.spark.SparkContext
  import org.apache.spark.SparkConf
  
  /** Computes an approximation to pi */
  object SparkPi {
    def main(args: Array[String]) {
      val conf = new SparkConf().setMaster(&quot;spark tjvrlaptop:7077&quot;).setAppName(&quot;Spark Pi&quot;) //.set(&quot;spark.scheduler.mode&quot;, &quot;FIFO&quot;).set(&quot;spark.cores.max&quot;, &quot;8&quot;)
      val spark = new SparkContext(conf)
      val slices = if (args.length &gt; 0) args(0).toInt else 20
      val n = math.max(100000000L * slices, Int.MaxValue).toInt // avoid overflow
  
      for(j &lt;- 1 to 1000000) {
        val count = spark.parallelize(1 until n, slices).map { i =&gt;
          val x = random * 2 - 1
          val y = random * 2 - 1
          if (x * x + y * y &lt; 1) 1 else 0
        }.reduce(_ + _)
        println(&quot;Pi is roughly &quot; + 4.0 * count / n)
      }
      spark.stop()
    }
  }
  </code></pre>
  <p>Here is my build.sbt contents:</p>
  <pre><code>name := &quot;SBTScalaSparkPi&quot;
  
  version := &quot;1.0&quot;
  
  scalaVersion := &quot;2.10.6&quot;
  
  libraryDependencies += &quot;org.apache.spark&quot; %% &quot;spark-core&quot; % &quot;1.6.1&quot;
  </code></pre>
  <p>Here is my plugins.sbt contents:</p>
  <pre><code>logLevel := Level.Warn
  </code></pre>
  <p>I executed the Spark Master and a worker by using the following commands in different command prompts on the same machine.</p>
  <pre><code>spark-1.6.1-bin-hadoop2.6\bin&gt;spark-class org.apache.spark.deploy.master.Master --host tjvrlaptop
  
  spark-1.6.1-bin-hadoop2.6\bin&gt;spark-class org.apache.spark.deploy.worker.Worker spark tjvrlaptop:7077
  </code></pre>
  <p>[The Master and Worker seems to be up and running without any issues][1]</p>
  <p>[1]: http i.stack.imgur.com/B3BDZ.png</p>
  <p>Next I tried to run the program in IntelliJ. It fails after a while with the following errors:</p>
  <pre><code>**Command Promt where Master is running**
  
  &gt;     16/03/27 14:44:33 INFO Master: Registering app Spark Pi
  &gt;     16/03/27 14:44:33 INFO Master: Registered app Spark Pi with ID app-20160327144433-0000
  &gt;     16/03/27 14:44:33 INFO Master: Launching executor app-20160327144433-0000/0 on worker
  &gt; worker-20160327140440-192.168.56.1-52701
  &gt;     16/03/27 14:44:38 INFO Master: Received unregister request from application app-20160327144433-0000
  &gt;     16/03/27 14:44:38 INFO Master: Removing app app-20160327144433-0000
  &gt;     16/03/27 14:44:38 INFO Master: TJVRLAPTOP:55368 got disassociated, removing it.
  &gt;     16/03/27 14:44:38 INFO Master: 192.168.56.1:55350 got disassociated, removing it.
  &gt;     16/03/27 14:44:38 WARN Master: Got status update for unknown executor app-20160327144433-0000/0
  
  **Command Prompt where the Worker is running**
  
  &gt; 16/03/27 14:44:34 INFO Worker: Asked to launch executor
  &gt; app-20160327144433-0000/0 for Spark Pi 16/03/27 14:44:34 INFO
  &gt; SecurityManager: Changing view acls to: tjoha 16/03/27 14:44:34 INFO
  &gt; SecurityManager: Changing modify acls to: tjoha 16/03/27 14:44:34 INFO
  &gt; SecurityManager: SecurityManager: authentication disabled; ui acls
  &gt; disabled; users with view permissions: Set(tjoha); users with modify
  &gt; permissions: Set(tjoha) 16/03/27 14:44:34 INFO ExecutorRunner: Launch
  &gt; command: &quot;C Program Files\Java\jre1.8.0_77\bin\java&quot; &quot;-cp&quot;
  &gt; &quot;C Users\tjoha\Documents\spark-1.6.1-bin-hadoop2.6\bin\..\conf\;C Users\tjoha\Documents\spark-1.6.1-bin-hadoop2.6\bin\..\lib\spark-assembly-1.6.1-hadoop2.6.0.jar;C Users\tjoha\Documents\spark-1.6.1-bin-hadoop2.6\bin\..\lib\datanucleus-api-jdo-3.2.6.jar;C Users\tjoha\Documents\spark-1.6.1-bin-hadoop2.6\bin\..\lib\datanucleus-core-3.2.10.jar;C Users\tjoha\Documents\spark-1.6.1-bin-hadoop2.6\bin\..\lib\datanucleus-rdbms-3.2.9.jar&quot;
  &gt; &quot;-Xms1024M&quot; &quot;-Xmx1024M&quot; &quot;-Dspark.driver.port=55350&quot;
  &gt; &quot;org.apache.spark.executor.CoarseGrainedExecutorBackend&quot;
  &gt; &quot;--driver-url&quot; &quot;spark CoarseGrainedScheduler@192.168.56.1:55350&quot;
  &gt; &quot;--executor-id&quot; &quot;0&quot; &quot;--hostname&quot; &quot;192.168.56.1&quot; &quot;--cores&quot; &quot;8&quot;
  &gt; &quot;--app-id&quot; &quot;app-20160327144433-0000&quot; &quot;--worker-url&quot;
  &gt; &quot;spark Worker@192.168.56.1:52701&quot; 16/03/27 14:44:38 INFO Worker:
  &gt; Asked to kill executor app-20160327144433-0000/0 16/03/27 14:44:38
  &gt; INFO ExecutorRunner: Runner thread for executor
  &gt; app-20160327144433-0000/0 interrupted 16/03/27 14:44:38 INFO
  &gt; ExecutorRunner: Killing process! 16/03/27 14:44:38 INFO Worker:
  &gt; Executor app-20160327144433-0000/0 finished with state KILLED
  &gt; exitStatus 1 16/03/27 14:44:38 INFO Worker: Cleaning up local
  &gt; directories for application app-20160327144433-0000 16/03/27 14:44:38
  &gt; INFO ExternalShuffleBlockResolver: Application app-20160327144433-0000
  &gt; removed, cleanupLocalDirs = true
  
  **IntelliJ Idea Output**
  
  &gt; Using Spark's default log4j profile:
  &gt; org/apache/spark/log4j-defaults.properties 16/03/27 15:06:04 INFO
  &gt; SparkContext: Running Spark version 1.6.1 16/03/27 15:06:05 WARN
  &gt; NativeCodeLoader: Unable to load native-hadoop library for your
  &gt; platform... using builtin-java classes where applicable 16/03/27
  &gt; 15:06:05 INFO SecurityManager: Changing view acls to: tjoha 16/03/27
  &gt; 15:06:05 INFO SecurityManager: Changing modify acls to: tjoha 16/03/27
  &gt; 15:06:05 INFO SecurityManager: SecurityManager: authentication
  &gt; disabled; ui acls disabled; users with view permissions: Set(tjoha);
  &gt; users with modify permissions: Set(tjoha) 16/03/27 15:06:06 INFO
  &gt; Utils: Successfully started service 'sparkDriver' on port 56183.
  &gt; 16/03/27 15:06:07 INFO Slf4jLogger: Slf4jLogger started 16/03/27
  &gt; 15:06:07 INFO Remoting: Starting remoting 16/03/27 15:06:07 INFO
  &gt; Remoting: Remoting started; listening on addresses
  &gt; :[akka tcp sparkDriverActorSystem@192.168.56.1:56196] 16/03/27
  &gt; 15:06:07 INFO Utils: Successfully started service
  &gt; 'sparkDriverActorSystem' on port 56196. 16/03/27 15:06:07 INFO
  &gt; SparkEnv: Registering MapOutputTracker 16/03/27 15:06:07 INFO
  &gt; SparkEnv: Registering BlockManagerMaster 16/03/27 15:06:07 INFO
  &gt; DiskBlockManager: Created local directory at
  &gt; C Users\tjoha\AppData\Local\Temp\blockmgr-9623b0f9-81f5-4a10-bbc7-ba077d53a2e5
  &gt; 16/03/27 15:06:07 INFO MemoryStore: MemoryStore started with capacity
  &gt; 2.4 GB 16/03/27 15:06:07 INFO SparkEnv: Registering OutputCommitCoordinator 16/03/27 15:06:07 WARN Utils: Service
  &gt; 'SparkUI' could not bind on port 4040. Attempting port 4041. 16/03/27
  &gt; 15:06:07 INFO Utils: Successfully started service 'SparkUI' on port
  &gt; 4041. 16/03/27 15:06:07 INFO SparkUI: Started SparkUI at http 192.168.56.1:4041 16/03/27 15:06:08 INFO
  &gt; AppClient$ClientEndpoint: Connecting to master
  &gt; spark tjvrlaptop:7077... 16/03/27 15:06:09 INFO
  &gt; SparkDeploySchedulerBackend: Connected to Spark cluster with app ID
  &gt; app-20160327150608-0002 16/03/27 15:06:09 INFO
  &gt; AppClient$ClientEndpoint: Executor added: app-20160327150608-0002/0 on
  &gt; worker-20160327150550-192.168.56.1-56057 (192.168.56.1:56057) with 8
  &gt; cores 16/03/27 15:06:09 INFO SparkDeploySchedulerBackend: Granted
  &gt; executor ID app-20160327150608-0002/0 on hostPort 192.168.56.1:56057
  &gt; with 8 cores, 1024.0 MB RAM 16/03/27 15:06:09 INFO
  &gt; AppClient$ClientEndpoint: Executor updated: app-20160327150608-0002/0
  &gt; is now RUNNING 16/03/27 15:06:09 INFO Utils: Successfully started
  &gt; service 'org.apache.spark.network.netty.NettyBlockTransferService' on
  &gt; port 56234. 16/03/27 15:06:09 INFO NettyBlockTransferService: Server
  &gt; created on 56234 16/03/27 15:06:09 INFO BlockManagerMaster: Trying to
  &gt; register BlockManager 16/03/27 15:06:09 INFO
  &gt; BlockManagerMasterEndpoint: Registering block manager
  &gt; 192.168.56.1:56234 with 2.4 GB RAM, BlockManagerId(driver, 192.168.56.1, 56234) 16/03/27 15:06:09 INFO BlockManagerMaster: Registered BlockManager 16/03/27 15:06:09 INFO
  &gt; SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling
  &gt; beginning after reached minRegisteredResourcesRatio: 0.0 16/03/27
  &gt; 15:06:10 INFO SparkContext: Starting job: reduce at SparkPi.scala:37
  &gt; 16/03/27 15:06:10 INFO DAGScheduler: Got job 0 (reduce at
  &gt; SparkPi.scala:37) with 20 output partitions 16/03/27 15:06:10 INFO
  &gt; DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:37)
  &gt; 16/03/27 15:06:10 INFO DAGScheduler: Parents of final stage: List()
  &gt; 16/03/27 15:06:10 INFO DAGScheduler: Missing parents: List() 16/03/27
  &gt; 15:06:10 INFO DAGScheduler: Submitting ResultStage 0
  &gt; (MapPartitionsRDD[1] at map at SparkPi.scala:33), which has no missing
  &gt; parents 16/03/27 15:06:10 INFO MemoryStore: Block broadcast_0 stored
  &gt; as values in memory (estimated size 1880.0 B, free 1880.0 B) 16/03/27
  &gt; 15:06:10 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in
  &gt; memory (estimated size 1212.0 B, free 3.0 KB) 16/03/27 15:06:10 INFO
  &gt; BlockManagerInfo: Added broadcast_0_piece0 in memory on
  &gt; 192.168.56.1:56234 (size: 1212.0 B, free: 2.4 GB) 16/03/27 15:06:10 INFO SparkContext: Created broadcast 0 from broadcast at
  &gt; DAGScheduler.scala:1006 16/03/27 15:06:10 INFO DAGScheduler:
  &gt; Submitting 20 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at
  &gt; map at SparkPi.scala:33) 16/03/27 15:06:10 INFO TaskSchedulerImpl:
  &gt; Adding task set 0.0 with 20 tasks 16/03/27 15:06:14 INFO
  &gt; SparkDeploySchedulerBackend: Registered executor
  &gt; NettyRpcEndpointRef(null) (TJVRLAPTOP:56281) with ID 0 16/03/27
  &gt; 15:06:14 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0,
  &gt; TJVRLAPTOP, partition 0,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:14
  
  &gt; ...
  
  &gt; TJVRLAPTOP, partition 6,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:14
  &gt; INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7,
  &gt; TJVRLAPTOP, partition 7,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:14
  &gt; INFO BlockManagerMasterEndpoint: Registering block manager
  &gt; TJVRLAPTOP:56319 with 511.1 MB RAM, BlockManagerId(0, TJVRLAPTOP,
  &gt; 56319) 16/03/27 15:06:15 INFO BlockManagerInfo: Added
  &gt; broadcast_0_piece0 in memory on TJVRLAPTOP:56319 (size: 1212.0 B,
  &gt; free: 511.1 MB) 16/03/27 15:06:15 INFO TaskSetManager: Starting task
  &gt; 8.0 in stage 0.0 (TID 8, TJVRLAPTOP, partition 8,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:15 INFO TaskSetManager: Starting task 9.0 in
  &gt; stage 0.0 (TID 9, TJVRLAPTOP, partition 9,PROCESS_LOCAL, 2078 bytes)
  &gt; 16/03/27 15:06:15 INFO TaskSetManager: Starting task 10.0 in stage 0.0
  &gt; (TID 10, TJVRLAPTOP, partition 10,PROCESS_LOCAL, 2078 bytes) 16/03/27
  &gt; 15:06:15 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11,
  &gt; TJVRLAPTOP, partition 11,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:15
  
  &gt; ...
  
  &gt; java.lang.ClassNotFoundException: SparkPi$$anonfun$main$1$$anonfun$1
  &gt;   at java.net.URLClassLoader.findClass(Unknown Source)    at
  &gt; java.lang.ClassLoader.loadClass(Unknown Source)   at
  &gt; java.lang.ClassLoader.loadClass(Unknown Source)   at
  &gt; java.lang.Class.forName0(Native Method)   at
  &gt; java.lang.Class.forName(Unknown Source)   at
  &gt; org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
  &gt;   at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)   at
  &gt; java.io.ObjectInputStream.readClassDesc(Unknown Source)   at
  &gt; java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)  at
  &gt; java.io.ObjectInputStream.readObject0(Unknown Source)     at
  &gt; java.io.ObjectInputStream.defaultReadFields(Unknown Source)   at
  &gt; java.io.ObjectInputStream.readSerialData(Unknown Source)  at
  &gt; java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)  at
  &gt; java.io.ObjectInputStream.readObject0(Unknown Source)     at
  &gt; java.io.ObjectInputStream.defaultReadFields(Unknown Source)   at
  &gt; java.io.ObjectInputStream.readSerialData(Unknown Source)  at
  &gt; java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)  at
  &gt; java.io.ObjectInputStream.readObject0(Unknown Source)     at
  &gt; java.io.ObjectInputStream.defaultReadFields(Unknown Source)   at
  &gt; java.io.ObjectInputStream.readSerialData(Unknown Source)  at
  &gt; java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)  at
  &gt; java.io.ObjectInputStream.readObject0(Unknown Source)     at
  &gt; java.io.ObjectInputStream.readObject(Unknown Source)  at
  &gt; org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
  &gt;   at
  &gt; org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
  &gt;   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
  &gt;   at org.apache.spark.scheduler.Task.run(Task.scala:89)   at
  &gt; org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
  &gt;   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  &gt;   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  &gt;   at java.lang.Thread.run(Unknown Source) 16/03/27 15:06:15 INFO
  &gt; TaskSetManager: Lost task 5.0 in stage 0.0 (TID 5) on executor
  &gt; TJVRLAPTOP: java.lang.ClassNotFoundException
  &gt; (SparkPi$$anonfun$main$1$$anonfun$1) [duplicate 1] 16/03/27 15:06:15
  &gt; INFO TaskSetManager: Lost task 3.0 in stage 0.0 (TID 3) on executor
  &gt; TJVRLAPTOP: java.lang.ClassNotFoundException
  
  &gt; ...
   
  &gt; INFO TaskSetManager: Starting task 10.1 in stage 0.0 (TID 20,
  &gt; TJVRLAPTOP, partition 10,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:15
  
  &gt; ...
  
  &gt; TJVRLAPTOP, partition 3,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:15
  &gt; INFO TaskSetManager: Lost task 4.0 in stage 0.0 (TID 4) on executor
  &gt; TJVRLAPTOP: java.lang.ClassNotFoundException
  &gt; (SparkPi$$anonfun$main$1$$anonfun$1) [duplicate 8] 16/03/27 15:06:15
  &gt; INFO TaskSetManager: Lost task 12.0 in stage 0.0 (TID 12) on executor
  &gt; TJVRLAPTOP: java.lang.ClassNotFoundException
  
  &gt; ...
  
  &gt; INFO TaskSetManager: Starting task 2.3 in stage 0.0 (TID 39,
  &gt; TJVRLAPTOP, partition 2,PROCESS_LOCAL, 2078 bytes) 16/03/27 15:06:16
  &gt; MapOutputTrackerMasterEndpoint stopped! 16/03/27 15:06:16 WARN
  &gt; TransportChannelHandler: Exception in connection from
  &gt; TJVRLAPTOP/192.168.56.1:56281 java.io.IOException: An existing
  &gt; connection was forcibly closed by the remote host 16/03/27 15:06:17
  &gt; INFO MemoryStore: MemoryStore cleared 16/03/27 15:06:17 INFO
  &gt; BlockManager: BlockManager stopped 16/03/27 15:06:17 INFO
  &gt; BlockManagerMaster: BlockManagerMaster stopped 16/03/27 15:06:17 INFO
  &gt; OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
  &gt; OutputCommitCoordinator stopped! 16/03/27 15:06:17 INFO SparkContext:
  &gt; Successfully stopped SparkContext 16/03/27 15:06:17 INFO
  &gt; ShutdownHookManager: Shutdown hook called 16/03/27 15:06:17 INFO
  &gt; RemoteActorRefProvider$RemotingTerminator: Shutting down remote
  &gt; daemon. 16/03/27 15:06:17 INFO ShutdownHookManager: Deleting directory
  &gt; C Users\tjoha\AppData\Local\Temp\spark-11f8184f-23fb-43be-91bb-113fb74aa8b9
  </code></pre>