title: Configuring Spark to work with Jupyter Notebook and Anaconda
body: <p>I've spent a few days now trying to make Spark work with my Jupyter Notebook and Anaconda. Here's what my .bash_profile looks like:</p>
  <pre><code>PATH=&quot;/my/path/to/anaconda3/bin:$PATH&quot;
  
  export JAVA_HOME=&quot;/my/path/to/jdk&quot;
  export PYTHON_PATH=&quot;/my/path/to/anaconda3/bin/python&quot;
  export PYSPARK_PYTHON=&quot;/my/path/to/anaconda3/bin/python&quot;
  
  export PATH=$PATH:/my/path/to/spark-2.1.0-bin-hadoop2.7/bin
  export PYSPARK_DRIVER_PYTHON=jupyter
  export PYSPARK_DRIVER_PYTHON_OPTS=&quot;notebook&quot; pyspark
  export SPARK_HOME=/my/path/to/spark-2.1.0-bin-hadoop2.7
  alias pyspark=&quot;pyspark --conf spark.local.dir=/home/puifais --num-executors 30 --driver-memory 128g --executor-memory 6g --packages com.databricks:spark-csv_2.11:1.5.0&quot;
  </code></pre>
  <p>When I type <code>/my/path/to/spark-2.1.0-bin-hadoop2.7/bin/spark-shell</code>, I can launch Spark just fine in my command line shell. And the output <code>sc</code> is not empty. It seems to work fine.</p>
  <p>When I type <code>pyspark</code>, it launches my Jupyter Notebook fine. When I create a new Python3 notebook, this error appears:</p>
  <pre><code>[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /my/path/to/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py: 
  </code></pre>
  <p>And <code>sc</code> in my Jupyter Notebook is empty.</p>
  <p>Can anyone help solve this situation?</p>
  <hr />
  <p>Just want to clarify:  There is nothing after the colon at the end of the error. I also tried to create my own start-up file using this <a href="https://stackoverflow.com/a/43523771/7275398">post</a> and I quote here so you don't have to go look there:</p>
  <blockquote>
  <p>I created a short initialization script init_spark.py as follows:</p>
  <pre><code>from pyspark import SparkConf, SparkContext
  conf = SparkConf().setMaster(&quot;yarn-client&quot;)
  sc = SparkContext(conf = conf)
  </code></pre>
  <p>and placed it in the ~/.ipython/profile_default/startup/ directory</p>
  </blockquote>
  <p>When I did this, the error then became:</p>
  <pre><code>[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /my/path/to/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py:
  [IPKernelApp] WARNING | Unknown error in handling startup files:
  </code></pre>