title: Increase Number of Executors for a spark instance
body: <p>I am launching a spark instance via pyspark. The system I am using is r3.8x on EC2 with 244gb ram and 32 vCPU. The code I am using is</p>
  <pre><code>#Import Packages
  import os
  import sys
  import pandas as pd
  import numpy as np
  import glob
  import shutil
  
  #Initialize Spark Environment
  spark_path = &quot;C:\spark&quot;
  os.environ['SPARK_HOME'] = spark_path
  os.environ['HADOOP_HOME'] = spark_path
  
  sys.path.append(spark_path + &quot;/bin&quot;)
  sys.path.append(spark_path + &quot;/python&quot;)
  sys.path.append(spark_path + &quot;/python/pyspark/&quot;)
  sys.path.append(spark_path + &quot;/python/lib&quot;)
  sys.path.append(spark_path + &quot;/python/lib/pyspark.zip&quot;)
  sys.path.append(spark_path + &quot;/python/lib/py4j-0.10.4-src.zip&quot;)
  
  from pyspark import SparkContext
  from pyspark import SparkConf
  
  sc =  SparkContext(&quot;local&quot;, &quot;test&quot;) 
  #sc =  SparkContext(&quot;local&quot;, &quot;test&quot;) 
  
  sc.defaultParallelism
  1
  </code></pre>
  <p>I have set a few parameters using spark-defaults.conf file as follows</p>
  <pre><code>spark.driver.memory 20g
  spark.executor.cores 2
  spark.executor.instances 16
  spark.executor.memory 220g
  spark.memory.fraction 0.07
  spark.worker.cleanup.enabled true
  </code></pre>
  <p>This is confirmed from this code in pyspark</p>
  <pre><code>&gt; sc._conf.getAll()
  [('spark.executor.memory', '220g'),
   ('spark.app.id', 'local-1500885816084'),
   ('spark.app.name', 'test'),
   ('spark.executor.id', 'driver'),
   ('spark.driver.port', '61076'),
   ('spark.executor.instances', '16'),
   ('spark.executor.cores', '2'),
   ('spark.master', 'local[32]'),
   ('spark.driver.memory', '20g'),
   ('spark.rdd.compress', 'True'),
   ('spark.serializer.objectStreamReset', '100'),
   ('spark.submit.deployMode', 'client'),
   ('spark.memory.fraction', '0.07'),
   ('spark.worker.cleanup.enabled', 'true')]
  </code></pre>
  <p>Now when I check my resource manager using localhost:4041 I get this table</p>
  <p><a href="https://i.stack.imgur.com/4i6hU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4i6hU.png" alt="enter image description here" /></a></p>
  <p>It seems as if only one executor is running. How can I set this up right?</p>