title: Compute size of Spark dataframe - SizeEstimator gives unexpected results
body: <p>I am trying to find a reliable way to compute the size (in bytes) of a Spark dataframe programmatically. </p>
  
  <p>The reason is that I would like to have a method to compute an "optimal" number of partitions ("optimal" could mean different things here: it could mean  <a href="https://www.slideshare.net/hadooparchbook/top-5-mistakes-when-writing-spark-applications-66374492/39" rel="noreferrer">having an optimal partition size</a>, or <a href="https://stackoverflow.com/questions/45058368/how-to-deal-with-large-number-of-parquet-files">resulting in an optimal file size</a> when writing to Parquet tables - but both can be assumed to be some linear function of the dataframe size). In other words, I would like to call <code>coalesce(n)</code> or <code>repartition(n)</code> on the dataframe, where <code>n</code> is not a fixed number but rather a function of the dataframe size.</p>
  
  <p>Other topics on SO suggest using <code>SizeEstimator.estimate</code> from <code>org.apache.spark.util</code> to get the size in bytes of the dataframe, but the results I'm getting are inconsistent.</p>
  
  <p>First of all, I'm persisting my dataframe to memory:</p>
  
  <pre><code>df.cache().count 
  </code></pre>
  
  <p>The Spark UI shows a size of 4.8GB in the Storage tab. Then, I run the following command to get the size from <code>SizeEstimator</code>:</p>
  
  <pre><code>import org.apache.spark.util.SizeEstimator
  SizeEstimator.estimate(df)
  </code></pre>
  
  <p>This gives a result of 115'715'808 bytes =~ 116MB. However, applying <code>SizeEstimator</code> to different objects leads to very different results. For instance, I try computing the size separately for each row in the dataframe and sum them:</p>
  
  <pre><code>df.map(row =&gt; SizeEstimator.estimate(row.asInstanceOf[ AnyRef ])).reduce(_+_)
  </code></pre>
  
  <p>This results in a size of 12'084'698'256 bytes =~ 12GB. Or, I can try to apply <code>SizeEstimator</code> to every partition: </p>
  
  <pre><code>df.mapPartitions(
      iterator =&gt; Seq(SizeEstimator.estimate(
          iterator.toList.map(row =&gt; row.asInstanceOf[ AnyRef ]))).toIterator
  ).reduce(_+_)
  </code></pre>
  
  <p>which results again in a different size of 10'792'965'376 bytes =~ 10.8GB. </p>
  
  <p>I understand there are memory optimizations / memory overhead involved, but after performing these tests I don't see how <code>SizeEstimator</code> can be used to get a sufficiently good estimate of the dataframe size (and consequently of the partition size, or resulting Parquet file sizes). </p>
  
  <p>What is the appropriate way (if any) to apply <code>SizeEstimator</code> in order to get a good estimate of a dataframe size or of its partitions? If there isn't any, what is the suggested approach here?</p>