{"action":"created","issue":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/14052","repository_url":"https://api.github.com/repos/pandas-dev/pandas","labels_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14052/labels{/name}","comments_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14052/comments","events_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14052/events","html_url":"https://github.com/pandas-dev/pandas/issues/14052","id":172272637,"number":14052,"title":"Inconsistent .values for tz-aware columns","user":{"login":"sdementen","id":1304950,"avatar_url":"https://avatars1.githubusercontent.com/u/1304950?v=3","gravatar_id":"","url":"https://api.github.com/users/sdementen","html_url":"https://github.com/sdementen","followers_url":"https://api.github.com/users/sdementen/followers","following_url":"https://api.github.com/users/sdementen/following{/other_user}","gists_url":"https://api.github.com/users/sdementen/gists{/gist_id}","starred_url":"https://api.github.com/users/sdementen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdementen/subscriptions","organizations_url":"https://api.github.com/users/sdementen/orgs","repos_url":"https://api.github.com/users/sdementen/repos","events_url":"https://api.github.com/users/sdementen/events{/privacy}","received_events_url":"https://api.github.com/users/sdementen/received_events","type":"User","site_admin":false},"labels":[{"id":35818298,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/API%20Design","name":"API Design","color":"AD7FA8","default":false},{"id":76811,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/Bug","name":"Bug","color":"e10c02","default":false},{"id":195647922,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/Difficulty%20Intermediate","name":"Difficulty Intermediate","color":"fbca04","default":false},{"id":195648009,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/Effort%20Low","name":"Effort Low","color":"006b75","default":false},{"id":13098779,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/Reshaping","name":"Reshaping","color":"02d7e1","default":false},{"id":60458168,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/Timezones","name":"Timezones","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/pandas-dev/pandas/milestones/32","html_url":"https://github.com/pandas-dev/pandas/milestone/32","labels_url":"https://api.github.com/repos/pandas-dev/pandas/milestones/32/labels","id":933188,"number":32,"title":"Next Major Release","description":"after 0.18.0 of course","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars1.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":1077,"closed_issues":173,"state":"open","created_at":"2015-01-13T10:53:19Z","updated_at":"2017-03-17T21:20:53Z","due_on":"2017-08-31T07:00:00Z","closed_at":null},"comments":26,"created_at":"2016-08-20T14:05:04Z","updated_at":"2017-03-19T05:57:35Z","closed_at":null,"body":"When getting the values of a dataframe with multiple columns with a single type datetime64[ns,UTC], it returns a numpy array with dtype=object instead of dtype=datetime64[ns,UTC].\nWhen returning a single column with dtype=datetime64[ns,UTC] or returning a multiple columns of dtype=datetime64[ns] (ie without tz), it works.\nLooking at the code, it appears that SingleBlockManager.is_mixed_type returns False in the first case (datetime64[ns,UTC]) and True in the two other cases (single datetime64[ns,UTC] or multiple datetime64[ns]).\n#### Code Sample, a copy-pastable example if possible\n\n```\nimport pandas\n\nidx = pandas.date_range(\"2010\", \"2011\", tz=\"UTC\")\ndf = pandas.DataFrame(index=idx)\ndf[\"A\"] = idx\ndf[\"B\"] = idx\nprint(df[[\"B\"]].values.dtype)\nprint(df[[\"A\", \"B\"]].values.dtype)\n```\n#### Actual Output\n\n```\ndatetime64[ns, UTC]\nobject\n```\n#### Expected Output\n\n```\ndatetime64[ns, UTC]\ndatetime64[ns, UTC]\n```\n#### output of `pd.show_versions()`\n\nINSTALLED VERSIONS\n\ncommit: None\npython: 3.5.2.final.0\npython-bits: 32\nOS: Windows\nOS-release: 7\nmachine: AMD64\nprocessor: Intel64 Family 6 Model 61 Stepping 4, GenuineIntel\nbyteorder: little\nLC_ALL: None\nLANG: None\n\npandas: 0.18.1\nnose: None\npip: 8.1.2\nsetuptools: 23.0.0\nCython: None\nnumpy: 1.11.0\nscipy: 0.17.1\nstatsmodels: None\nxarray: None\nIPython: 4.2.0\nsphinx: None\npatsy: None\ndateutil: 2.5.3\npytz: 2016.4\nblosc: None\nbottleneck: None\ntables: None\nnumexpr: 2.6.1\nmatplotlib: 1.5.1\nopenpyxl: 2.3.2\nxlrd: 1.0.0\nxlwt: None\nxlsxwriter: 0.9.2\nlxml: 3.6.0\nbs4: None\nhtml5lib: None\nhttplib2: 0.9.2\napiclient: None\nsqlalchemy: 1.0.12\npymysql: None\npsycopg2: 2.6.2 (dt dec pq3 ext)\njinja2: 2.8\nboto: None\npandas_datareader: None\n"},"comment":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/comments/287596559","html_url":"https://github.com/pandas-dev/pandas/issues/14052#issuecomment-287596559","issue_url":"https://api.github.com/repos/pandas-dev/pandas/issues/14052","id":287596559,"user":{"login":"sdementen","id":1304950,"avatar_url":"https://avatars1.githubusercontent.com/u/1304950?v=3","gravatar_id":"","url":"https://api.github.com/users/sdementen","html_url":"https://github.com/sdementen","followers_url":"https://api.github.com/users/sdementen/followers","following_url":"https://api.github.com/users/sdementen/following{/other_user}","gists_url":"https://api.github.com/users/sdementen/gists{/gist_id}","starred_url":"https://api.github.com/users/sdementen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdementen/subscriptions","organizations_url":"https://api.github.com/users/sdementen/orgs","repos_url":"https://api.github.com/users/sdementen/repos","events_url":"https://api.github.com/users/sdementen/events{/privacy}","received_events_url":"https://api.github.com/users/sdementen/received_events","type":"User","site_admin":false},"created_at":"2017-03-19T05:57:35Z","updated_at":"2017-03-19T05:57:35Z","body":"sorry to be not that clear in my previous post \r\n\r\nI was illustrating indeed what Series, a DatetimeIndex and a DataFrame were giving as a result of using their property `values`. In terms of API design, I would expect these 3 to be \"consistent/coherent\" in some way.\r\nAs both Series and DatetimeIndex returns ndarray of dtype `datetime64[ns]` in UTC but without the tz information (hence, we loose the fact that the `datetime64[ns]` were tz aware vs naive in pandas world, but OK, given the constrains of numpy, it looks a reasonable assumption), I would expect the `df[['B']]` or `df[['A','B']]` to also return ndarray of dtype `datetime64[ns]` in UTC. This would be equivalent to applying values to each columns (as a series) in the dataframe and then concatenating the results. This would also mean that, if the dataframe contains only `datetime64[ns]` and/or  `datetime64[ns, any tz]`, it would be converted to a ndarray of dtype  `datetime64[ns]`.\r\n\r\nHence, I am a bit far from the previous goal of df[['B']] returning an ndarray of dtype object with tz-aware Timestamp inside.\r\n\r\nTo make everything even clearer, the following script shows the 3 cases ([\"A\"], [\"B\"], [\"A\", \"B\"]) for which values return 3 different kind of objects, as well as 2 scenarios to return consistent values.\r\n```\r\nimport numpy\r\nfrom pandas import date_range, DataFrame\r\n\r\ndef to_values(df, as_dt64=False):\r\n    return numpy.array(list(df.itertuples(index=False)),\r\n                       dtype=\"<M8[ns]\" if as_dt64 else object)\r\n\r\ndf = DataFrame({\"A\": date_range(\"20120101\", periods=2),\r\n                \"B\": date_range(\"20120101\", periods=2, tz=\"CET\")})\r\n\r\nfor col in [[\"A\"], [\"B\"], [\"A\", \"B\"]]:\r\n    print(\"Case : \", col)\r\n    print(\"1. Current .values\")\r\n    print(repr(df[col].values))\r\n    print(\"2. Exporting Timestamp 'object'\")\r\n    arr = to_values(df[col])\r\n    print(repr(arr))\r\n    print(\"3. Exporting datetime64[ns]\")\r\n    arr = to_values(df[col], as_dt64=True)\r\n    print(repr(arr))\r\n    print()\r\n```\r\nwith output\r\n```\r\nCase :  ['A']\r\n1. Current .values\r\narray([['2012-01-01T00:00:00.000000000'],\r\n       ['2012-01-02T00:00:00.000000000']], dtype='datetime64[ns]')\r\n2. Exporting Timestamp 'object'\r\narray([[Timestamp('2012-01-01 00:00:00')],\r\n       [Timestamp('2012-01-02 00:00:00')]], dtype=object)\r\n3. Exporting datetime64[ns]\r\narray([['2012-01-01T00:00:00.000000000'],\r\n       ['2012-01-02T00:00:00.000000000']], dtype='datetime64[ns]')\r\n\r\nCase :  ['B']\r\n1. Current .values\r\nDatetimeIndex(['2012-01-01 00:00:00+01:00', '2012-01-02 00:00:00+01:00'], dtype='datetime64[ns, CET]', freq='D')\r\n2. Exporting Timestamp 'object'\r\narray([[Timestamp('2012-01-01 00:00:00+0100', tz='CET')],\r\n       [Timestamp('2012-01-02 00:00:00+0100', tz='CET')]], dtype=object)\r\n3. Exporting datetime64[ns]\r\narray([['2011-12-31T23:00:00.000000000'],\r\n       ['2012-01-01T23:00:00.000000000']], dtype='datetime64[ns]')\r\n\r\nCase :  ['A', 'B']\r\n1. Current .values\r\narray([[Timestamp('2012-01-01 00:00:00'),\r\n        Timestamp('2012-01-01 00:00:00+0100', tz='CET')],\r\n       [Timestamp('2012-01-02 00:00:00'),\r\n        Timestamp('2012-01-02 00:00:00+0100', tz='CET')]], dtype=object)\r\n2. Exporting Timestamp 'object'\r\narray([[Timestamp('2012-01-01 00:00:00'),\r\n        Timestamp('2012-01-01 00:00:00+0100', tz='CET')],\r\n       [Timestamp('2012-01-02 00:00:00'),\r\n        Timestamp('2012-01-02 00:00:00+0100', tz='CET')]], dtype=object)\r\n3. Exporting datetime64[ns]\r\narray([['2012-01-01T00:00:00.000000000', '2011-12-31T23:00:00.000000000'],\r\n       ['2012-01-02T00:00:00.000000000', '2012-01-01T23:00:00.000000000']], dtype='datetime64[ns]')\r\n```\r\n\r\nAs this can lead to backward incompatible changes, I am not sure which option is the best but probably the `to_values(df, as_dt64=True)` interpretation is the more consistent with Series and DatetimeIndex."}}
{"actor":{"display_login":"sdementen"}}