{"action":"closed","issue":{"url":"https://api.github.com/repos/amacd31/phildb/issues/16","repository_url":"https://api.github.com/repos/amacd31/phildb","labels_url":"https://api.github.com/repos/amacd31/phildb/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/amacd31/phildb/issues/16/comments","events_url":"https://api.github.com/repos/amacd31/phildb/issues/16/events","html_url":"https://github.com/amacd31/phildb/issues/16","id":248645364,"number":16,"title":"Loss of datetime precision with Pandas < 0.20","user":{"login":"amacd31","id":906888,"avatar_url":"https://avatars1.githubusercontent.com/u/906888?v=4","gravatar_id":"","url":"https://api.github.com/users/amacd31","html_url":"https://github.com/amacd31","followers_url":"https://api.github.com/users/amacd31/followers","following_url":"https://api.github.com/users/amacd31/following{/other_user}","gists_url":"https://api.github.com/users/amacd31/gists{/gist_id}","starred_url":"https://api.github.com/users/amacd31/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amacd31/subscriptions","organizations_url":"https://api.github.com/users/amacd31/orgs","repos_url":"https://api.github.com/users/amacd31/repos","events_url":"https://api.github.com/users/amacd31/events{/privacy}","received_events_url":"https://api.github.com/users/amacd31/received_events","type":"User","site_admin":false},"labels":[{"id":89144163,"url":"https://api.github.com/repos/amacd31/phildb/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-08-08T08:56:00Z","updated_at":"2017-08-08T11:55:27Z","closed_at":"2017-08-08T11:55:27Z","body":"With Pandas < 0.20 the precision of datetimes can be lost when writing irregular series.\r\n\r\nThis can occur when the data to be written can be stored as float32 and Pandas treats the data as float32 instead of float64. The below snippet of code demonstrates the problem and the final assert will fail with older versions of Pandas.\r\n\r\n```import pandas as pd\r\n\r\nsample = pd.DataFrame(\r\n    pd.np.array(\r\n        [x + 0.1 for x in range(10)],\r\n        dtype=pd.np.float32\r\n    ),\r\n    index=pd.date_range(\r\n        '2017-08-06 06:50:00+00:00',\r\n        periods=10,\r\n        freq='1T'\r\n    )\r\n)\r\n\r\nsample['datestamp'] = pd.Series(\r\n    sample.index.map(\r\n        lambda dateval: dateval.value // 1000000000\r\n    ),\r\n    index=sample.index\r\n)\r\n\r\nsample.values.astype(pd.np.int64)\r\n\r\nprint(sample.values.astype(pd.np.int64)[0][1])\r\nassert sample.values.astype(pd.np.int64)[0][1] == 1502002200\r\n```\r\n\r\nThis issue stems from the fact that calling `.values` on a `pandas.DataFrame` should cast the data types to the a capable common data type. However the bug in pandas results in the int64 being cast to a float32, losing precision in the process, instead of correctly upcasting everything to float64 (which can hold both types successfully).\r\n\r\nThis bug can be worked around by either ensuring that the time series to be written has a dtype of float64 or you are using Pandas >= 0.20."}}
{"actor":{"display_login":"amacd31"}}