{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19322","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19322/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19322/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19322/events","html_url":"https://github.com/apache/spark/pull/19322","id":259821439,"number":19322,"title":"[SPARK-22102][SQL] Set ConfVars.METASTOREWAREHOUSE before constructor CliSessionState","user":{"login":"wangyum","id":5399861,"avatar_url":"https://avatars0.githubusercontent.com/u/5399861?v=4","gravatar_id":"","url":"https://api.github.com/users/wangyum","html_url":"https://github.com/wangyum","followers_url":"https://api.github.com/users/wangyum/followers","following_url":"https://api.github.com/users/wangyum/following{/other_user}","gists_url":"https://api.github.com/users/wangyum/gists{/gist_id}","starred_url":"https://api.github.com/users/wangyum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wangyum/subscriptions","organizations_url":"https://api.github.com/users/wangyum/orgs","repos_url":"https://api.github.com/users/wangyum/repos","events_url":"https://api.github.com/users/wangyum/events{/privacy}","received_events_url":"https://api.github.com/users/wangyum/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-09-22T13:50:25Z","updated_at":"2017-09-23T10:37:21Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19322","html_url":"https://github.com/apache/spark/pull/19322","diff_url":"https://github.com/apache/spark/pull/19322.diff","patch_url":"https://github.com/apache/spark/pull/19322.patch"},"body":"## What changes were proposed in this pull request? \r\n\r\nAs described in [SPARK-22102](https://issues.apache.org/jira/browse/SPARK-22102). spark-sql shows the warehouse dir is `file:/root/create/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse`, but actually the warehouse dir is `/user/hive/warehouse` when create table. the reason is that set `ConfVars.METASTOREWAREHOUSE` after constructor CliSessionState is invalid.\r\n\r\nThis PR set `ConfVars.METASTOREWAREHOUSE` before constructor CliSessionState\r\n\r\n\r\n\r\n## How was this patch tested?\r\n\r\nmanual tests, seems didn't have a good way to do unit test.\r\n\r\n```\r\nYumings-MacBook-Pro:spark-2.3.0-SNAPSHOT-bin-2.6.5 ming$ bin/spark-sql\r\n17/09/22 21:59:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).\r\nlog4j:WARN Please initialize the log4j system properly.\r\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\r\nUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\r\n17/09/22 22:00:00 INFO SQLUtils: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse').\r\n17/09/22 22:00:00 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore\r\n17/09/22 22:00:00 INFO ObjectStore: ObjectStore, initialize called\r\n17/09/22 22:00:00 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored\r\n17/09/22 22:00:00 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored\r\n17/09/22 22:00:02 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=\"Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order\"\r\n17/09/22 22:00:03 INFO Datastore: The class \"org.apache.hadoop.hive.metastore.model.MFieldSchema\" is tagged as \"embedded-only\" so does not have its own datastore table.\r\n17/09/22 22:00:03 INFO Datastore: The class \"org.apache.hadoop.hive.metastore.model.MOrder\" is tagged as \"embedded-only\" so does not have its own datastore table.\r\n17/09/22 22:00:03 INFO Datastore: The class \"org.apache.hadoop.hive.metastore.model.MFieldSchema\" is tagged as \"embedded-only\" so does not have its own datastore table.\r\n17/09/22 22:00:03 INFO Datastore: The class \"org.apache.hadoop.hive.metastore.model.MOrder\" is tagged as \"embedded-only\" so does not have its own datastore table.\r\n17/09/22 22:00:03 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY\r\n17/09/22 22:00:03 INFO ObjectStore: Initialized ObjectStore\r\n17/09/22 22:00:03 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0\r\n17/09/22 22:00:04 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException\r\n17/09/22 22:00:04 INFO HiveMetaStore: Added admin role in metastore\r\n17/09/22 22:00:04 INFO HiveMetaStore: Added public role in metastore\r\n17/09/22 22:00:04 INFO HiveMetaStore: No user is added in admin role, since config is empty\r\n17/09/22 22:00:04 INFO HiveMetaStore: 0: get_all_databases\r\n17/09/22 22:00:04 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_all_databases   \r\n17/09/22 22:00:04 INFO HiveMetaStore: 0: get_functions: db=default pat=*\r\n17/09/22 22:00:04 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_functions: db=default pat=*     \r\n17/09/22 22:00:04 INFO Datastore: The class \"org.apache.hadoop.hive.metastore.model.MResourceUri\" is tagged as \"embedded-only\" so does not have its own datastore table.\r\n17/09/22 22:00:04 INFO SessionState: Created local directory: /var/folders/gl/xfd53h993834l25kxpnjksy40000gn/T/a202faad-8ea0-4019-8051-f31828db413a_resources\r\n17/09/22 22:00:04 INFO SessionState: Created HDFS directory: /tmp/hive/ming/a202faad-8ea0-4019-8051-f31828db413a\r\n17/09/22 22:00:04 INFO SessionState: Created local directory: /var/folders/gl/xfd53h993834l25kxpnjksy40000gn/T/ming/a202faad-8ea0-4019-8051-f31828db413a\r\n17/09/22 22:00:04 INFO SessionState: Created HDFS directory: /tmp/hive/ming/a202faad-8ea0-4019-8051-f31828db413a/_tmp_space.db\r\n17/09/22 22:00:04 INFO SparkContext: Running Spark version 2.3.0-SNAPSHOT\r\n17/09/22 22:00:04 INFO SparkContext: Submitted application: SparkSQL::10.8.44.55\r\n17/09/22 22:00:04 INFO SecurityManager: Changing view acls to: ming\r\n17/09/22 22:00:04 INFO SecurityManager: Changing modify acls to: ming\r\n17/09/22 22:00:04 INFO SecurityManager: Changing view acls groups to: \r\n17/09/22 22:00:04 INFO SecurityManager: Changing modify acls groups to: \r\n17/09/22 22:00:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ming); groups with view permissions: Set(); users  with modify permissions: Set(ming); groups with modify permissions: Set()\r\n17/09/22 22:00:04 INFO Utils: Successfully started service 'sparkDriver' on port 57139.\r\n17/09/22 22:00:04 INFO SparkEnv: Registering MapOutputTracker\r\n17/09/22 22:00:05 INFO SparkEnv: Registering BlockManagerMaster\r\n17/09/22 22:00:05 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information\r\n17/09/22 22:00:05 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up\r\n17/09/22 22:00:05 INFO DiskBlockManager: Created local directory at /private/var/folders/gl/xfd53h993834l25kxpnjksy40000gn/T/blockmgr-d3520e8e-9879-4329-8dfb-71a429dec945\r\n17/09/22 22:00:05 INFO MemoryStore: MemoryStore started with capacity 366.3 MB\r\n17/09/22 22:00:05 INFO SparkEnv: Registering OutputCommitCoordinator\r\n17/09/22 22:00:05 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.\r\n17/09/22 22:00:05 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.\r\n17/09/22 22:00:05 INFO Utils: Successfully started service 'SparkUI' on port 4042.\r\n17/09/22 22:00:05 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.8.44.55:4042\r\n17/09/22 22:00:05 INFO Executor: Starting executor ID driver on host localhost\r\n17/09/22 22:00:05 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57140.\r\n17/09/22 22:00:05 INFO NettyBlockTransferService: Server created on 10.8.44.55:57140\r\n17/09/22 22:00:05 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy\r\n17/09/22 22:00:05 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.8.44.55, 57140, None)\r\n17/09/22 22:00:05 INFO BlockManagerMasterEndpoint: Registering block manager 10.8.44.55:57140 with 366.3 MB RAM, BlockManagerId(driver, 10.8.44.55, 57140, None)\r\n17/09/22 22:00:05 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.8.44.55, 57140, None)\r\n17/09/22 22:00:05 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.8.44.55, 57140, None)\r\n17/09/22 22:00:05 INFO SQLUtils: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse').\r\n17/09/22 22:00:05 INFO SharedState: Warehouse path is 'file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse'.\r\n17/09/22 22:00:05 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.\r\n17/09/22 22:00:05 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse\r\n17/09/22 22:00:05 INFO HiveMetaStore: 0: get_database: default\r\n17/09/22 22:00:05 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_database: default       \r\n17/09/22 22:00:05 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse\r\n17/09/22 22:00:06 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint\r\nspark-sql> create table t(id int);\r\n17/09/22 22:00:32 INFO HiveMetaStore: 0: get_database: global_temp\r\n17/09/22 22:00:32 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_database: global_temp   \r\n17/09/22 22:00:32 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException\r\n17/09/22 22:00:32 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse\r\n17/09/22 22:00:33 INFO HiveMetaStore: 0: get_database: default\r\n17/09/22 22:00:33 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_database: default       \r\n17/09/22 22:00:33 INFO HiveMetaStore: 0: get_database: default\r\n17/09/22 22:00:33 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_database: default       \r\n17/09/22 22:00:33 INFO HiveMetaStore: 0: get_table : db=default tbl=t\r\n17/09/22 22:00:33 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_table : db=default tbl=t        \r\n17/09/22 22:00:33 INFO HiveMetaStore: 0: get_database: default\r\n17/09/22 22:00:33 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=get_database: default       \r\n17/09/22 22:00:34 INFO HiveMetaStore: 0: create_table: Table(tableName:t, dbName:default, owner:ming, createTime:1506088832, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id, type:int, comment:null)], location:file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse/t, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{spark.sql.sources.schema.part.0={\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}, spark.sql.sources.schema.numParts=1, spark.sql.create.version=2.3.0-SNAPSHOT}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null))\r\n17/09/22 22:00:34 INFO audit: ugi=ming  ip=unknown-ip-addr      cmd=create_table: Table(tableName:t, dbName:default, owner:ming, createTime:1506088832, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id, type:int, comment:null)], location:file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse/t, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{spark.sql.sources.schema.part.0={\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}, spark.sql.sources.schema.numParts=1, spark.sql.create.version=2.3.0-SNAPSHOT}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null))       \r\n17/09/22 22:00:34 WARN HiveMetaStore: Location: file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse/t specified for non-external table:t\r\n17/09/22 22:00:34 INFO FileUtils: Creating directory if it doesn't exist: file:/Users/ming/tmp/spark/spark-2.3.0-SNAPSHOT-bin-2.6.5/spark-warehouse/t\r\nTime taken: 2.156 seconds\r\n17/09/22 22:00:34 INFO SparkSQLCLIDriver: Time taken: 2.156 seconds\r\nspark-sql> \r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/331625645","html_url":"https://github.com/apache/spark/pull/19322#issuecomment-331625645","issue_url":"https://api.github.com/repos/apache/spark/issues/19322","id":331625645,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-09-23T10:37:21Z","updated_at":"2017-09-23T10:37:21Z","author_association":"NONE","body":"**[Test build #82113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82113/testReport)** for PR 19322 at commit [`cfc31f7`](https://github.com/apache/spark/commit/cfc31f76dc2a13ec2422963615254209a86af4ca)."}}
{"actor":{"display_login":"SparkQA"}}