title: How will spark load a huge csv file if the entire file is present on a single node?
body: <p>If I have a huge CSV file of 50GB on a single HDFS node and I am trying to read the file using spark.read as below:</p>
  <pre><code>file_df = spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('/hdfspath/customer.csv')
  </code></pre>
  <p>and I am submitting the spark job using the below spark-submit:</p>
  <pre><code>spark-submit --master yarn --deploy-mode cluster --num-executors 4 --executor-memory 3G --executor-cores 5 --driver-memory 1G load_csv.py
  </code></pre>
  <p>I understand that until there is an action, spark will not load any data into memory. But what happens when an action is triggered and the first thing to be done is reading the file into memory to kick start the transformations. How does spark read the 50GB file in small parts based on the cores and executors I mentioned ?</p>
  <p>For example: I mentioned 4 executors and 3GB memory for each executor. While reading, will spark convert the main customer.csv file into chunks of 3GB for each executor and load the file which is:</p>
  <p>For the first 12gb:</p>
  <pre><code>Executor 1: 3GB
  Executor 2: 3GB
  Executor 3: 3GB
  Executor 4: 3GB
  </code></pre>
  <p>and so on until the whole file is completed processing ?</p>
  <p>Or will it split the file based on HDFS block size and read block by block for ex: 128MB and it tries to fit in as many blocks as it can in each executor of 3GB ?</p>
  <p>If a file is entirely present on a single cluster (which in my case it is), how will spark process a file ?</p>
  <p>I understand its a bit broader and cumbersome to explain, but any help would be greatly appreciated.</p>