title: Data from partitioned table does not show up when queried from Hive
body: <p><strong>Note that</strong> this question <strong>is not</strong> a duplicate of <a href="https://stackoverflow.com/questions/42476731/spark-sql-saveastable-returns-empty-result">this one</a>! I do not use Spark SQL for partitioning! I am saving individual parquet files!</p>
  
  <p><strong>I am also using Databricks</strong> which does not support Hive-flavored SQL.</p>
  
  <p>I have a table in Hive (I'm using Databricks) which contains two partitions. It looks like this:</p>
  
  <pre class="lang-sql prettyprint-override"><code>CREATE TABLE foo_test (`col0` STRING, `col1` STRING, `col2` STRING, `datestamp` STRING)
  USING parquet
  OPTIONS (
    `serialization.format` '1'
  )
  PARTITIONED BY (datestamp)
  </code></pre>
  
  <p><strong>Edit:</strong> *this is also the result of calling <code>show create table foo_test;</code></p>
  
  <p>I've added two partitions with Spark to this table by hand:</p>
  
  <pre class="lang-py prettyprint-override"><code>df = spark.read.csv(file_path.format(datestamp), header=True, inferSchema=True)
  
  partitionPath = "/mnt/foo_test/datestamp={}/".format(datestamp)
  
  df.coalesce(1).write.parquet(partitionPath, mode="overwrite")
  
  
  /mnt/foo_test/datestamp=20180101/
  /mnt/foo_test/datestamp=20180102/
  </code></pre>
  
  <p>If I load the data with Spark I can see it is there:</p>
  
  <pre class="lang-py prettyprint-override"><code>spark.read.option("mergeSchema", True).parquet("/mnt/foo_test").show()
  
  +----+----+----+----+---------+
  |col0|col1|col2|col3|datestamp|
  +----+----+----+----+---------+
  | foo| bar| baz|   1| 20180102|
  | xul| qux| wom|   2| 20180102|
  | bar| foo| baz|   1| 20180102|
  | qux| xul| wom|   2| 20180102|
  | foo| bar| baz|null| 20180101|
  | xul| qux| wom|null| 20180101|
  | bar| foo| baz|null| 20180101|
  | qux| xul| wom|null| 20180101|
  +----+----+----+----+---------+
  </code></pre>
  
  <p>My problem is that if I run an SQL query against this Hive table it returns nothing:</p>
  
  <pre class="lang-sql prettyprint-override"><code>SELECT * FROM foo_test;
  
  OK
  </code></pre>
  
  <p>Even after adding the partition by hand:</p>
  
  <pre class="lang-py prettyprint-override"><code>spark.sql("ALTER TABLE foo_test ADD IF NOT EXISTS PARTITION (datestamp=20180102)")
  </code></pre>
  
  <p>and repairing the table:</p>
  
  <pre><code>MSCK REPAIR TABLE foo_test;
  </code></pre>
  
  <p>I can see that the partitions are present according to Hive:</p>
  
  <pre class="lang-sql prettyprint-override"><code>SHOW PARTITIONS foo_test;
  
  partition
  datestamp=20180102
  datestamp=20180101
  </code></pre>
  
  <p>but the <code>SELECT</code> returns nothing.</p>
  
  <p>This is the description of my table:</p>
  
  <pre><code>col0    string  null
  col1    string  null
  col2    string  null
  datestamp   string  null
  # Partition Information     
  # col_name  data_type   comment
  datestamp   string  null
  # Detailed Table Information        
  Database    default 
  Table   foo_test    
  Owner   root    
  Created Thu Apr 26 12:25:06 UTC 2018    
  Last Access Thu Jan 01 00:00:00 UTC 1970    
  Type    MANAGED 
  Provider    parquet 
  Table Properties    [transient_lastDdlTime=1524745506]  
  Location    dbfs:/user/hive/warehouse/foo_test  
  Serde Library   org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe 
  InputFormat org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat   
  OutputFormat    org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat  
  Storage Properties  [serialization.format=1]    
  Partition Provider  Catalog
  </code></pre>
  
  <p><strong>What could be the problem here?</strong></p>