{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16944","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16944/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16944/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16944/events","html_url":"https://github.com/apache/spark/pull/16944","id":207944873,"number":16944,"title":"[SPARK-19611][SQL] Introduce configurable table schema inference","user":{"login":"budde","id":798699,"avatar_url":"https://avatars0.githubusercontent.com/u/798699?v=3","gravatar_id":"","url":"https://api.github.com/users/budde","html_url":"https://github.com/budde","followers_url":"https://api.github.com/users/budde/followers","following_url":"https://api.github.com/users/budde/following{/other_user}","gists_url":"https://api.github.com/users/budde/gists{/gist_id}","starred_url":"https://api.github.com/users/budde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/budde/subscriptions","organizations_url":"https://api.github.com/users/budde/orgs","repos_url":"https://api.github.com/users/budde/repos","events_url":"https://api.github.com/users/budde/events{/privacy}","received_events_url":"https://api.github.com/users/budde/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2017-02-15T22:12:21Z","updated_at":"2017-03-01T19:32:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16944","html_url":"https://github.com/apache/spark/pull/16944","diff_url":"https://github.com/apache/spark/pull/16944.diff","patch_url":"https://github.com/apache/spark/pull/16944.patch"},"body":"*Update: Accidentally closed #16942. Opening a replacement PR.*\r\n\r\nReplaces #16797. See the discussion in this PR for more details/justification for this change.\r\n\r\n## Summary of changes\r\n[JIRA for this change](https://issues.apache.org/jira/browse/SPARK-19611)\r\n\r\n- Add spark.sql.hive.caseSensitiveInferenceMode param to SQLConf\r\n- Add schemaPreservesCase field to CatalogTable (set to false when schema can't\r\n  successfully be read from Hive table props)\r\n- Perform schema inference in HiveMetastoreCatalog if schemaPreservesCase is\r\n  false, depending on spark.sql.hive.caseSensitiveInferenceMode\r\n- Update table metadata properties in HiveExternalCatalog.alterTable()\r\n- Add HiveSchemaInferenceSuite tests\r\n\r\n## How was this patch tested?\r\n\r\nThe tests in HiveSchemaInferenceSuite should verify that schema inference is working as expected.\r\n\r\n## Open issues\r\n\r\n- Should \"INFER_AND_SAVE\" be the default behavior? This restores the out-of-the-box compatibility that was present prior to 2.1.0 but changes the behavior of 2.1.0 (which is essentially \"NEVER_INFER\").\r\n- Is ```HiveExternalCatalog.alterTable()``` the appropriate place to write back the table metadata properties outside of createTable()? Should a new external catalog method like updateTableMetadata() be introduced?\r\n- All partition columns will still be treated as case-insensitive even after inferring. As far as I remember, this has always been the case with schema inference prior to Spark 2.1.0 and I haven't made any attempts to reconcile this since it doesn't cause the same problems that case sensitive data fields do. Should we attempt to restore case sensitivity by inspecting file paths or leave this as-is?"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/283444510","html_url":"https://github.com/apache/spark/pull/16944#issuecomment-283444510","issue_url":"https://api.github.com/repos/apache/spark/issues/16944","id":283444510,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars2.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-03-01T19:32:37Z","updated_at":"2017-03-01T19:32:37Z","body":"**[Test build #73707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73707/testReport)** for PR 16944 at commit [`a2658fc`](https://github.com/apache/spark/commit/a2658fced7886f82701739bc3a1ff5c7335a5bc8)."}}
{"actor":{"display_login":"SparkQA"}}