{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/17248","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/17248/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/17248/comments","events_url":"https://api.github.com/repos/apache/spark/issues/17248/events","html_url":"https://github.com/apache/spark/pull/17248","id":213440969,"number":17248,"title":"[SPARK-19909][SS] Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS","user":{"login":"sarutak","id":4736016,"avatar_url":"https://avatars0.githubusercontent.com/u/4736016?v=3","gravatar_id":"","url":"https://api.github.com/users/sarutak","html_url":"https://github.com/sarutak","followers_url":"https://api.github.com/users/sarutak/followers","following_url":"https://api.github.com/users/sarutak/following{/other_user}","gists_url":"https://api.github.com/users/sarutak/gists{/gist_id}","starred_url":"https://api.github.com/users/sarutak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sarutak/subscriptions","organizations_url":"https://api.github.com/users/sarutak/orgs","repos_url":"https://api.github.com/users/sarutak/repos","events_url":"https://api.github.com/users/sarutak/events{/privacy}","received_events_url":"https://api.github.com/users/sarutak/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-03-10T20:14:40Z","updated_at":"2017-03-10T22:14:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/17248","html_url":"https://github.com/apache/spark/pull/17248","diff_url":"https://github.com/apache/spark/pull/17248.diff","patch_url":"https://github.com/apache/spark/pull/17248.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen we try to run Structured Streaming in local mode but use HDFS for the storage, batches will be fail because of error like as follows.\r\n\r\n```\r\nval handle = stream.writeStream.format(\"console\").start()\r\n17/03/09 16:54:45 ERROR StreamMetadata: Error writing stream metadata StreamMetadata(fc07a0b1-5423-483e-a59d-b2206a49491e) to /private/var/folders/4y/tmspvv353y59p3w4lknrf7cc0000gn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata\r\norg.apache.hadoop.security.AccessControlException: Permission denied: user=kou, access=WRITE, inode=\"/private/var/folders/4y/tmspvv353y59p3w4lknrf7cc0000gn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata\":hdfs:supergroup:drwxr-xr-x\r\n```\r\n\r\nIt's because that a temporary checkpoint directory is created on local file system but metadata whose path is based on the checkpoint directory will be created on HDFS.\r\n\r\nThis PR will fixe this issue.\r\n\r\n## How was this patch tested?\r\n\r\nI tested manually in local mode with HDFS.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/285799564","html_url":"https://github.com/apache/spark/pull/17248#issuecomment-285799564","issue_url":"https://api.github.com/repos/apache/spark/issues/17248","id":285799564,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars2.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-03-10T22:14:11Z","updated_at":"2017-03-10T22:14:11Z","body":"**[Test build #74334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74334/testReport)** for PR 17248 at commit [`cc7a3f8`](https://github.com/apache/spark/commit/cc7a3f8474975e49a5b4e87c9ae9d8ea0185fc9f).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}