title: PySpark:Processing 100000 column dataset
body: <p>I am using Cloudera Spark 2. We have Kerberos implemented in the cluster.</p>
  
  <p>I have a CSV dataset of 400000x100000. It has 500GB.
  I have done the following :</p>
  
  <ul>
  <li><code>df = spark.read.csv('largefile.csv',header=Ture,maxCoulmns=100000)</code> </li>
  <li><code>print(df.count())</code> and saved in spark_test.py</li>
  <li>In terminal <code>spark2-submit --master yarn --deploy-mode cluster spark_test.py</code></li>
  </ul>
  
  <hr>
  
  <p>The spark job is running, I am able to track it in Spark UI and it is keep on running, after 30 minutes or so it is failing. </p>
  
  <hr>
  
  <p>For testing purpose I have tried the above steps with 10 columns dataset and the job completed successfully.</p>
  
  <hr>
  
  <p>Is there any restrictions or configurations to increase the columns that Spark process ?</p>