{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16912","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16912/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16912/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16912/events","html_url":"https://github.com/apache/spark/pull/16912","id":207197303,"number":16912,"title":"[SPARK-19576] [Core] Task attempt paths exist in output path after saveAsNewAPIHadoopFile completes with speculation enabled","user":{"login":"sharkdtu","id":3926682,"avatar_url":"https://avatars.githubusercontent.com/u/3926682?v=3","gravatar_id":"","url":"https://api.github.com/users/sharkdtu","html_url":"https://github.com/sharkdtu","followers_url":"https://api.github.com/users/sharkdtu/followers","following_url":"https://api.github.com/users/sharkdtu/following{/other_user}","gists_url":"https://api.github.com/users/sharkdtu/gists{/gist_id}","starred_url":"https://api.github.com/users/sharkdtu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sharkdtu/subscriptions","organizations_url":"https://api.github.com/users/sharkdtu/orgs","repos_url":"https://api.github.com/users/sharkdtu/repos","events_url":"https://api.github.com/users/sharkdtu/events{/privacy}","received_events_url":"https://api.github.com/users/sharkdtu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-02-13T11:59:57Z","updated_at":"2017-02-13T12:02:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16912","html_url":"https://github.com/apache/spark/pull/16912","diff_url":"https://github.com/apache/spark/pull/16912.diff","patch_url":"https://github.com/apache/spark/pull/16912.patch"},"body":"`writeShard` in `saveAsNewAPIHadoopDataset` always committed its tasks without question. The problem is that when speculation is enabled sometimes this can result in multiple tasks committing their output to the same path, which may lead to task temporary paths exist in output path after `saveAsNewAPIHadoopFile` completes. \r\n\r\n```scala\r\n-rw-r--r--    3   user group       0   2017-02-11 19:36 hdfs://.../output/_SUCCESS\r\ndrwxr-xr-x    -   user group       0   2017-02-11 19:36 hdfs://.../output/attempt_201702111936_32487_r_000044_0\r\n-rw-r--r--    3   user group    8952   2017-02-11 19:36 hdfs://.../output/part-r-00000\r\n-rw-r--r--    3   user group    7878   2017-02-11 19:36 hdfs://.../output/part-r-00001\r\n```\r\nAssume there are two attempt tasks that commit at the same time, The two attempt tasks maybe rename their task attempt paths to task committed path at the same time. When one task's `rename` operation completes, the other task's `rename` operation will let its task attempt path under the task committed path.\r\n\r\nAnyway, it is not recommended that `writeShard` in `saveAsNewAPIHadoopDataset` always committed its tasks without question. Similar question in SPARK-4879 triggered by calling saveAsHadoopFile has been solved. Newest master has solved it too. This PR just fix 2.1\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/279368455","html_url":"https://github.com/apache/spark/pull/16912#issuecomment-279368455","issue_url":"https://api.github.com/repos/apache/spark/issues/16912","id":279368455,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2017-02-13T12:02:19Z","updated_at":"2017-02-13T12:02:19Z","body":"Can one of the admins verify this patch?"}}
{"actor":{"display_login":"AmplabJenkins"}}