title: Python worker failed to connect back
body: <p>I'm a newby with Spark and trying to complete a Spark tutorial:
  <a href="https://www.youtube.com/watch?v=3CPI2D_QD44&amp;index=4&amp;list=PLot-YkcC7wZ_2sxmRTZr2c121rjcaleqv" rel="noreferrer">link to tutorial</a></p>
  
  <p>After installing it on local machine (Win10 64, Python 3, Spark 2.4.0) and setting all env variables (HADOOP_HOME, SPARK_HOME etc) I'm trying to run a simple Spark job via WordCount.py file:</p>
  
  <pre><code>from pyspark import SparkContext, SparkConf
  
  if __name__ == "__main__":
      conf = SparkConf().setAppName("word count").setMaster("local[2]")
      sc = SparkContext(conf = conf)
  
      lines = sc.textFile("C:/Users/mjdbr/Documents/BigData/python-spark-tutorial/in/word_count.text")
      words = lines.flatMap(lambda line: line.split(" "))
      wordCounts = words.countByValue()
  
      for word, count in wordCounts.items():
          print("{} : {}".format(word, count))
  </code></pre>
  
  <p>After running it from terminal:</p>
  
  <pre><code>spark-submit WordCount.py
  </code></pre>
  
  <p>I get below error.
  I checked (by commenting out line by line) that it crashes at </p>
  
  <pre><code>wordCounts = words.countByValue()
  </code></pre>
  
  <p>Any idea what should I check to make it work?</p>
  
  <pre><code>Traceback (most recent call last):
    File "C:\Users\mjdbr\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "C:\Users\mjdbr\Anaconda3\lib\runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\worker.py", line 25, in &lt;module&gt;
  ModuleNotFoundError: No module named 'resource'
  18/11/10 23:16:58 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
  org.apache.spark.SparkException: Python worker failed to connect back.
          at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:170)
          at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:97)
          at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
          at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108)
          at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
          at org.apache.spark.scheduler.Task.run(Task.scala:121)
          at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
          at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
  Caused by: java.net.SocketTimeoutException: Accept timed out
          at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
          at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
          at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
          at java.net.PlainSocketImpl.accept(Unknown Source)
          at java.net.ServerSocket.implAccept(Unknown Source)
          at java.net.ServerSocket.accept(Unknown Source)
          at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:164)
          ... 14 more
  18/11/10 23:16:58 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
  Traceback (most recent call last):
    File "C:/Users/mjdbr/Documents/BigData/python-spark-tutorial/rdd/WordCount.py", line 19, in &lt;module&gt;
      wordCounts = words.countByValue()
    File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\rdd.py", line 1261, in countByValue
    File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\rdd.py", line 844, in reduce
    File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\rdd.py", line 816, in collect
    File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py", line 1257, in __call__
    File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py", line 328, in get_return_value
  py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure:
  Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): org.apache.spark.SparkException: Python worker failed to connect back.
          at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:170)
          at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:97)
          at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
          at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108)
          at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
          at org.apache.spark.scheduler.Task.run(Task.scala:121)
          at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
          at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
  Caused by: java.net.SocketTimeoutException: Accept timed out
          at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
          at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
          at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
          at java.net.PlainSocketImpl.accept(Unknown Source)
          at java.net.ServerSocket.implAccept(Unknown Source)
          at java.net.ServerSocket.accept(Unknown Source)
          at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:164)
          ... 14 more
  
  Driver stacktrace:
          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874)
          at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
          at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
          at scala.Option.foreach(Option.scala:257)
          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
          at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
          at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
          at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
          at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
          at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
          at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:166)
          at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
          at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
          at py4j.Gateway.invoke(Gateway.java:282)
          at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
          at py4j.commands.CallCommand.execute(CallCommand.java:79)
          at py4j.GatewayConnection.run(GatewayConnection.java:238)
          at java.lang.Thread.run(Unknown Source)
  Caused by: org.apache.spark.SparkException: Python worker failed to connect back.
          at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:170)
          at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:97)
          at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
          at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108)
          at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
          at org.apache.spark.scheduler.Task.run(Task.scala:121)
          at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
          at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          ... 1 more
  Caused by: java.net.SocketTimeoutException: Accept timed out
          at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
          at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
          at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
          at java.net.PlainSocketImpl.accept(Unknown Source)
          at java.net.ServerSocket.implAccept(Unknown Source)
          at java.net.ServerSocket.accept(Unknown Source)
          at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:164)
          ... 14 more
  </code></pre>
  
  <p>As suggested by theplatypus - checked if the 'resource' module can be imported directly from terminal - apparently not:</p>
  
  <pre><code>&gt;&gt;&gt; import resource
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  ModuleNotFoundError: No module named 'resource'
  </code></pre>
  
  <p>In terms of installation resources - I followed instructions from <a href="https://www.youtube.com/watch?v=iarn1KHeouc&amp;index=3&amp;list=PLot-YkcC7wZ_2sxmRTZr2c121rjcaleqv" rel="noreferrer">this tutorial</a>:</p>
  
  <ol>
  <li>downloaded spark-2.4.0-bin-hadoop2.7.tgz from <a href="https://spark.apache.org/downloads.html" rel="noreferrer">Apache Spark website</a>  </li>
  <li>un-zipped it to my C-drive  </li>
  <li>already had Python_3 installed (Anaconda distribution) as well as Java</li>
  <li>created local 'C:\hadoop\bin' folder to store winutils.exe  </li>
  <li>created 'C:\tmp\hive' folder and gave Spark access to it</li>
  <li>added environment variables (SPARK_HOME, HADOOP_HOME etc)  </li>
  </ol>
  
  <p>Is there any extra resource I should install?</p>