title: Spark Dataframe vs Pandas Dataframe memory usage comparison
body: <p>I'm new to spark, and had written some sample code, to check whether using spark is feasible (reducing memory usage), so I had created a sample dataframe, converted to spark DF, and was comparing the memory usage of both. The sample code is:</p>
  <pre><code>emp = [(1, &quot;Smith&quot;, -1, &quot;2018&quot;, &quot;10&quot;, &quot;M&quot;, 3000),
         (2, &quot;Rose&quot;, 1, &quot;2010&quot;, &quot;20&quot;, &quot;M&quot;, 4000),
         (3, &quot;Williams&quot;, 1, &quot;2010&quot;, &quot;10&quot;, &quot;M&quot;, 1000),
         (4, &quot;Jones&quot;, 2, &quot;2005&quot;, &quot;10&quot;, &quot;F&quot;, 2000),
         (5, &quot;Brown&quot;, 2, &quot;2010&quot;, &quot;40&quot;, &quot;&quot;, -1),
         (6, &quot;Brown&quot;, 2, &quot;2010&quot;, &quot;50&quot;, &quot;&quot;, -1)]*800000
  empColumns = [&quot;emp_id&quot;, &quot;name&quot;, &quot;superior_emp_id&quot;, &quot;year_joined&quot;,
                &quot;emp_dept_id&quot;, &quot;gender&quot;, &quot;salary&quot;]
  
  df = pd.DataFrame(emp, columns=empColumns)
  print(sys.getsizeof(df))
  
  empDF = spark.createDataFrame(data=emp, schema=empColumns)
  print(sys.getsizeof(empDF))
  print(sys.getsizeof(empDF.collect()))
  print(sys.getsizeof(empDF.toPandas()))
  </code></pre>
  <p>The values I'm getting is:</p>
  <pre><code>1267200144
  48
  42915440
  1267200144
  </code></pre>
  <p>So now, I have 2 questions:</p>
  <ol>
  <li>What is the meaning of <code>sys.getsizeof(empDF)</code>and which part is of 48 bytes here.</li>
  <li>Why is there such a difference between spark and pandas df?</li>
  </ol>