title: PySpark cannot run program "python"
body: <p>Currently I am trying to run a pyspark script and when I try to convert my spark dataframe to a pandas dataframe it throws the following error:</p>
  <pre><code>Py4JJavaError: An error occurred while calling o110.collectToPython.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 12.0 failed 1 times, most recent failure: Lost task 0.0 in stage 12.0 (TID 158, 192.168.0.103, executor driver): java.io.IOException: Cannot run program &quot;python&quot;: error=2, No such file or directory
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
      at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:209)
      at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:132)
      at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:105)
      at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:119)
      at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:131)
      at org.apache.spark.sql.execution.python.PandasGroupUtils$.executePython(PandasGroupUtils.scala:44)
      at org.apache.spark.sql.execution.python.FlatMapGroupsInPandasExec.$anonfun$doExecute$1(FlatMapGroupsInPandasExec.scala:94)
      at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
      at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:127)
      at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: java.io.IOException: error=2, No such file or directory
      at java.lang.UNIXProcess.forkAndExec(Native Method)
      at java.lang.UNIXProcess.(UNIXProcess.java:247)
      at java.lang.ProcessImpl.start(ProcessImpl.java:134)
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
  
  </code></pre>
  <p>I have changed my env variables in the <code>bashrc</code> to the following:</p>
  <pre><code>export SPARK_HOME=&quot;/usr/local/spark/spark-3.0.0-bin-hadoop2.7&quot;
  export PATH=&quot;${PATH}:${SPARK_HOME}/bin&quot;
  
  export PYSPARK_PYTHON=&quot;/usr/bin/python3&quot;
  export PYSPARK_DRIVER_PYTHON=&quot;/usr/bin/python3&quot;
  
  alias python='/usr/bin/python3'
  export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH
  
  
  </code></pre>
  <p>but I still have the error. However I don't know what else to do and what could be the source of the error. If anyone could guide me to what could be the problem since I have spent almost a week trying to figure it out but still with no solution I will appreciate it.</p>