{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19607","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19607/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19607/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19607/events","html_url":"https://github.com/apache/spark/pull/19607","id":269486176,"number":19607,"title":"[SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp values for Pandas to respect session timezone","user":{"login":"ueshin","id":506656,"avatar_url":"https://avatars1.githubusercontent.com/u/506656?v=4","gravatar_id":"","url":"https://api.github.com/users/ueshin","html_url":"https://github.com/ueshin","followers_url":"https://api.github.com/users/ueshin/followers","following_url":"https://api.github.com/users/ueshin/following{/other_user}","gists_url":"https://api.github.com/users/ueshin/gists{/gist_id}","starred_url":"https://api.github.com/users/ueshin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ueshin/subscriptions","organizations_url":"https://api.github.com/users/ueshin/orgs","repos_url":"https://api.github.com/users/ueshin/repos","events_url":"https://api.github.com/users/ueshin/events{/privacy}","received_events_url":"https://api.github.com/users/ueshin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":70,"created_at":"2017-10-30T05:32:49Z","updated_at":"2017-11-28T08:47:33Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19607","html_url":"https://github.com/apache/spark/pull/19607","diff_url":"https://github.com/apache/spark/pull/19607.diff","patch_url":"https://github.com/apache/spark/pull/19607.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen converting Pandas DataFrame/Series from/to Spark DataFrame using `toPandas()` or pandas udfs, timestamp values behave to respect Python system timezone instead of session timezone.\r\n\r\nFor example, let's say we use `\"America/Los_Angeles\"` as session timezone and have a timestamp value `\"1970-01-01 00:00:01\"` in the timezone. Btw, I'm in Japan so Python timezone would be `\"Asia/Tokyo\"`.\r\n\r\nThe timestamp value from current `toPandas()` will be the following:\r\n\r\n```\r\n>>> spark.conf.set(\"spark.sql.session.timeZone\", \"America/Los_Angeles\")\r\n>>> df = spark.createDataFrame([28801], \"long\").selectExpr(\"timestamp(value) as ts\")\r\n>>> df.show()\r\n+-------------------+\r\n|                 ts|\r\n+-------------------+\r\n|1970-01-01 00:00:01|\r\n+-------------------+\r\n\r\n>>> df.toPandas()\r\n                   ts\r\n0 1970-01-01 17:00:01\r\n```\r\n\r\nAs you can see, the value becomes `\"1970-01-01 17:00:01\"` because it respects Python timezone.\r\nAs we discussed in #18664, we consider this behavior is a bug and the value should be `\"1970-01-01 00:00:01\"`.\r\n\r\n## How was this patch tested?\r\n\r\nAdded tests and existing tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/347453556","html_url":"https://github.com/apache/spark/pull/19607#issuecomment-347453556","issue_url":"https://api.github.com/repos/apache/spark/issues/19607","id":347453556,"user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2017-11-28T08:47:33Z","updated_at":"2017-11-28T08:47:33Z","author_association":"MEMBER","body":"Yup, I was testing and trying to produce details. Let me describe this in the JIRA, not here :D."}}
{"actor":{"display_login":"HyukjinKwon"}}