title: How to get the actual configuration that spark used?
body: <p>I have a cluster with 3 nodes (1 master and 2 workers). All the nodes are 64 GB and 16 cores. I start pyspark shell with specifying the following configuration</p>
  <pre><code>'spark.driver.memory=19g'
  'spark.executor.memory=40g'
  'spark.executor.instances=8'
  'spark.executor.cores=5'
  'spark.driver.cores=5'
  'spark.default.parallelism=80'
  
  </code></pre>
  <p>Clearly, the executor memory and number executor cores are asking for resources that simply are not there. But, I am able to successfully start the shell without any exceptions. Now, when I check the number of executor instances and executor memory allocated they are 8 and 40 respectively</p>
  <pre><code>spark.sparkContext.getConf().get('spark.executor.instances')
  output:
  '8'
  spark.sparkContext.getConf().get('spark.executor.memory')
  output:
  '40g'
  </code></pre>
  <p>Which cannot be True.</p>
  <p>How can I know what the actual configuration is that my cluster got set up to use?</p>