{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16585","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16585/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16585/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16585/events","html_url":"https://github.com/apache/spark/pull/16585","id":200786009,"number":16585,"title":"[SPARK-19223][SQL][PySpark] Fix InputFileBlockHolder for datasources which are based on HadoopRDD or NewHadoopRDD","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-01-14T06:44:43Z","updated_at":"2017-01-14T08:11:16Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16585","html_url":"https://github.com/apache/spark/pull/16585","diff_url":"https://github.com/apache/spark/pull/16585.diff","patch_url":"https://github.com/apache/spark/pull/16585.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nFor some datasources which are based on HadoopRDD or NewHadoopRDD, such as spark-xml, InputFileBlockHolder doesn't work with Python UDF.\r\n\r\nThe method to reproduce it is, running the following codes with `bin/pyspark --packages com.databricks:spark-xml_2.11:0.4.1`:\r\n\r\n    from pyspark.sql.functions import udf,input_file_name\r\n    from pyspark.sql.types import StringType\r\n    from pyspark.sql import SparkSession\r\n\r\n    def filename(path):\r\n        return path\r\n\r\n    session = SparkSession.builder.appName('APP').getOrCreate()\r\n\r\n    session.udf.register('sameText',filename)\r\n    sameText = udf(filename, StringType())\r\n\r\n    df = session.read.format('xml').load('a.xml', rowTag='root').select('*',input_file_name().alias('file'))\r\n    df.select('file').show() # works\r\n    df.select(sameText(df['file'])).show()   # returns empty content\r\n\r\nThe issue is because in `HadoopRDD` and `NewHadoopRDD` we set the file block's info in `InputFileBlockHolder` before the returned iterator begins consuming. `InputFileBlockHolder` will record this info into thread local variable. When running Python UDF in batch, we set up another thread to consume the iterator from child plan's output rdd, so we can't read the info back in another thread.\r\n\r\nTo fix this, we have to set the info in `InputFileBlockHolder` after the iterator begins consuming. So the info can be read in correct thread.\r\n\r\n## How was this patch tested?\r\n\r\nManual test with above example codes for spark-xml package on pyspark: `bin/pyspark --packages com.databricks:spark-xml_2.11:0.4.1`.\r\n\r\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/272609654","html_url":"https://github.com/apache/spark/pull/16585#issuecomment-272609654","issue_url":"https://api.github.com/repos/apache/spark/issues/16585","id":272609654,"user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"created_at":"2017-01-14T08:11:16Z","updated_at":"2017-01-14T08:11:16Z","body":"retest this please."}}
{"actor":{"display_login":"viirya"}}