{"action":"created","issue":{"url":"https://api.github.com/repos/prestodb/presto/issues/7713","repository_url":"https://api.github.com/repos/prestodb/presto","labels_url":"https://api.github.com/repos/prestodb/presto/issues/7713/labels{/name}","comments_url":"https://api.github.com/repos/prestodb/presto/issues/7713/comments","events_url":"https://api.github.com/repos/prestodb/presto/issues/7713/events","html_url":"https://github.com/prestodb/presto/issues/7713","id":218541813,"number":7713,"title":"Parquet ROW Type Does Not Support Schema Evolution","user":{"login":"fbertsch","id":20819040,"avatar_url":"https://avatars2.githubusercontent.com/u/20819040?v=3","gravatar_id":"","url":"https://api.github.com/users/fbertsch","html_url":"https://github.com/fbertsch","followers_url":"https://api.github.com/users/fbertsch/followers","following_url":"https://api.github.com/users/fbertsch/following{/other_user}","gists_url":"https://api.github.com/users/fbertsch/gists{/gist_id}","starred_url":"https://api.github.com/users/fbertsch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fbertsch/subscriptions","organizations_url":"https://api.github.com/users/fbertsch/orgs","repos_url":"https://api.github.com/users/fbertsch/repos","events_url":"https://api.github.com/users/fbertsch/events{/privacy}","received_events_url":"https://api.github.com/users/fbertsch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-03-31T15:47:16Z","updated_at":"2017-03-31T15:55:41Z","closed_at":null,"body":"We use `ROW` types for a lot of Json data the we put into parquet format and read from Presto. One issue we have run into is that general schema evolution rules do not apply to ROW types; that is, we can't add new fields to a `ROW` type and expect previous schemas to still be applicable.\r\n\r\nNote: we set `hive.parquet.use-column-names=true`.\r\n\r\n# Test Case\r\n\r\nFor an example of this, see [0]. We have outlined many of the forms of schema evolution we use, and the last two are adding and removing elements from a `ROW` type. Note the schema evolution for the \"Nested Row Type - Adding a Subcolumn\" is the following:\r\n\r\n```\r\nmessage example {\r\n    required group nested {\r\n        required binary id (UTF8);\r\n        required binary exists (UTF8);\r\n    }\r\n}\r\n```\r\n\r\nto\r\n\r\n```\r\nmessage example {\r\n    required group nested {\r\n        required binary id (UTF8);\r\n        required binary exists (UTF8);\r\n        required binary added (UTF8);\r\n    }\r\n}\r\n```\r\n\r\n# Result\r\n\r\nit errors with `Schema mismatch, metastore schema for row column nested has 3 fields but parquet schema has 2 fields`.\r\n\r\nThe issue comes from [1], which prevents differing `STRUCT` types from having different numbers of elements from the Hive metastore schema.\r\n\r\n# Expected Result\r\n\r\nWe expected that the `ROW` type have schema evolution similar to that of adding a new column; old parquet files without the column just have `NULL` for the value. New files with the column can be read with the old.\r\n\r\nWe propose letting nested `ROW` types have the same rules of schema evolution as general parquet schemas; possibly by using a recursive definition for `ROW` types.\r\n\r\n[0] https://github.com/fbertsch/schema_evolution_exploration/blob/master/presto_col_names/Schema_Evolution_Exploration.ipynb\r\n[1] https://github.com/prestodb/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/parquet/ParquetHiveRecordCursor.java#L745"},"comment":{"url":"https://api.github.com/repos/prestodb/presto/issues/comments/290752392","html_url":"https://github.com/prestodb/presto/issues/7713#issuecomment-290752392","issue_url":"https://api.github.com/repos/prestodb/presto/issues/7713","id":290752392,"user":{"login":"nezihyigitbasi","id":1223839,"avatar_url":"https://avatars0.githubusercontent.com/u/1223839?v=3","gravatar_id":"","url":"https://api.github.com/users/nezihyigitbasi","html_url":"https://github.com/nezihyigitbasi","followers_url":"https://api.github.com/users/nezihyigitbasi/followers","following_url":"https://api.github.com/users/nezihyigitbasi/following{/other_user}","gists_url":"https://api.github.com/users/nezihyigitbasi/gists{/gist_id}","starred_url":"https://api.github.com/users/nezihyigitbasi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nezihyigitbasi/subscriptions","organizations_url":"https://api.github.com/users/nezihyigitbasi/orgs","repos_url":"https://api.github.com/users/nezihyigitbasi/repos","events_url":"https://api.github.com/users/nezihyigitbasi/events{/privacy}","received_events_url":"https://api.github.com/users/nezihyigitbasi/received_events","type":"User","site_admin":false},"created_at":"2017-03-31T15:55:41Z","updated_at":"2017-03-31T15:55:41Z","body":"@fbertsch the new Parquet reader should have support for schema evolution for ROW types, can you please give it a try? (set `hive.parquet-optimized-reader.enabled=true`)\r\n\r\n"}}
{"actor":{"display_login":"nezihyigitbasi"}}