title: Hive Partition Performance
body: <p>I would like to ask about Hive partitioning performance.
  How many records (in term of rows) would I need in order to see the partitioning performance?</p>
  
  <p>Currently, I have 2 million+ records and I have split the table into 2 partitions.
  My partitioning condition is as follow:</p>
  
  <ul>
  <li><code>ADD PARTITION (year_month=’2017_07’)</code></li>
  <li><code>ADD PARTITION (year_month=’2017_08’)</code></li>
  <li><code>INSERT OVERWRITE TABLE T PARTITION (year_month='2017_07')
  SELECT * FROM T WHERE st_time &lt; '2017_08_01 00:00:00.0';</code></li>
  <li><code>INSERT OVERWRITE TABLE T PARTITION (year_month='2017_08')
  SELECT * FROM T WHERE st_time &gt;= '2017_08_01 00:00:00.0';</code></li>
  </ul>
  
  <p>When I tried to do some performance testing between the partitioned table and non-partitioned table, the performance of both tables is relatively the same.
  My performance testing query is as follows:</p>
  
  <ul>
  <li><code>SELECT * FROM T WHERE st_time &lt; '2017_08_01 00:00:00.0';</code></li>
  <li><code>SELECT * FROM non_part_table WHERE st_time &lt; '2017_08_01 00:00:00.0';</code></li>
  </ul>
  
  <p>Am I missing out on some important points? </p>
  
  <p>Also, does partition work on derived column/function? E.g. date_format(st_time, yyyy-MM)</p>
  
  <p>Any help or advice is greatly appreciated.</p>