{"action":"created","issue":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/17540","repository_url":"https://api.github.com/repos/pandas-dev/pandas","labels_url":"https://api.github.com/repos/pandas-dev/pandas/issues/17540/labels{/name}","comments_url":"https://api.github.com/repos/pandas-dev/pandas/issues/17540/comments","events_url":"https://api.github.com/repos/pandas-dev/pandas/issues/17540/events","html_url":"https://github.com/pandas-dev/pandas/issues/17540","id":258016045,"number":17540,"title":"`read_hdf` throws UnicodeDecodeError with Python 3.5 and 3.6 but not with Python 2.7","user":{"login":"zoof","id":1737483,"avatar_url":"https://avatars0.githubusercontent.com/u/1737483?v=4","gravatar_id":"","url":"https://api.github.com/users/zoof","html_url":"https://github.com/zoof","followers_url":"https://api.github.com/users/zoof/followers","following_url":"https://api.github.com/users/zoof/following{/other_user}","gists_url":"https://api.github.com/users/zoof/gists{/gist_id}","starred_url":"https://api.github.com/users/zoof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zoof/subscriptions","organizations_url":"https://api.github.com/users/zoof/orgs","repos_url":"https://api.github.com/users/zoof/repos","events_url":"https://api.github.com/users/zoof/events{/privacy}","received_events_url":"https://api.github.com/users/zoof/received_events","type":"User","site_admin":false},"labels":[{"id":49747336,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/2/3%20Compat","name":"2/3 Compat","color":"c7eef8","default":false},{"id":47229190,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/IO%20HDF5","name":"IO HDF5","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-09-15T11:58:58Z","updated_at":"2017-09-16T16:17:39Z","closed_at":null,"author_association":"NONE","body":"#### Code Sample, a copy-pastable example if possible\r\n\r\n```python\r\nimport pandas as pd\r\ndf = pd.read_hdf('data.h5')\r\n```\r\n#### Problem description\r\nThe HDF5 dataset was created with `pandas`, `to_hdf` in Python 2.7 and can be read in by Python 2.7. When I try to read it in with Python 3.5 or Python 3.6, I get the following:\r\n\r\n    ---------------------------------------------------------------------------\r\n    UnicodeDecodeError                        Traceback (most recent call last)\r\n    <ipython-input-2-53006689fd2c> in <module>()\r\n    ----> 1 df = pd.read_hdf(data.h5')\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in read_hdf(path_or_buf, key, **kwargs)\r\n        356                                      'contains multiple datasets.')\r\n        357             key = candidate_only_group._v_pathname\r\n    --> 358         return store.select(key, auto_close=auto_close, **kwargs)\r\n        359     except:\r\n        360         # if there is an error, close the store\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in select(self, key, where, start, stop, columns, iterator, chunksize, auto_close, **kwargs)\r\n        720                            chunksize=chunksize, auto_close=auto_close)\r\n        721 \r\n    --> 722         return it.get_result()\r\n        723 \r\n        724     def select_as_coordinates(\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in get_result(self, coordinates)\r\n       1426 \r\n       1427         # directly return the result\r\n    -> 1428         results = self.func(self.start, self.stop, where)\r\n       1429         self.close()\r\n       1430         return results\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in func(_start, _stop, _where)\r\n        713             return s.read(start=_start, stop=_stop,\r\n        714                           where=_where,\r\n    --> 715                           columns=columns, **kwargs)\r\n        716 \r\n        717         # create the iterator\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in read(self, start, stop, **kwargs)\r\n       2864             blk_items = self.read_index('block%d_items' % i)\r\n       2865             values = self.read_array('block%d_values' % i,\r\n    -> 2866                                      start=_start, stop=_stop)\r\n       2867             blk = make_block(values,\r\n       2868                              placement=items.get_indexer(blk_items))\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in read_array(self, key, start, stop)\r\n       2413         import tables\r\n       2414         node = getattr(self.group, key)\r\n    -> 2415         data = node[start:stop]\r\n       2416         attrs = node._v_attrs\r\n       2417 \r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/vlarray.py in __getitem__(self, key)\r\n        673             start, stop, step = self._process_range(\r\n        674                 key.start, key.stop, key.step)\r\n    --> 675             return self.read(start, stop, step)\r\n        676         # Try with a boolean or point selection\r\n        677         elif type(key) in (list, tuple) or isinstance(key, numpy.ndarray):\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/vlarray.py in read(self, start, stop, step)\r\n        813         atom = self.atom\r\n        814         if not hasattr(atom, 'size'):  # it is a pseudo-atom\r\n    --> 815             outlistarr = [atom.fromarray(arr) for arr in listarr]\r\n        816         else:\r\n        817             # Convert the list to the right flavor\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/vlarray.py in <listcomp>(.0)\r\n        813         atom = self.atom\r\n        814         if not hasattr(atom, 'size'):  # it is a pseudo-atom\r\n    --> 815             outlistarr = [atom.fromarray(arr) for arr in listarr]\r\n        816         else:\r\n        817             # Convert the list to the right flavor\r\n\r\n    /home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/atom.py in fromarray(self, array)\r\n       1226         if array.size == 0:\r\n       1227             return None\r\n    -> 1228         return six.moves.cPickle.loads(array.tostring())\r\n\r\n    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 23: ordinal not in range(128)\r\n\r\n**Note**: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates!\r\n\r\n**Note**: Many problems can be resolved by simply upgrading `pandas` to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if `master` addresses this issue, but that is not necessary.\r\n\r\nFor documentation-related issues, you can check the latest versions of the docs on `master` here:\r\n\r\nhttps://pandas-docs.github.io/pandas-docs-travis/\r\n\r\nIf the issue has not been resolved there, go ahead and file it in the issue tracker.\r\n\r\n#### Expected Output\r\n    In [1]: import pandas as pd\r\n    In [2]: df = pd.read_hdf('data.h5')\r\n\r\n#### Output of ``pd.show_versions()``\r\n\r\n\r\n[paste the output of ``pd.show_versions()`` here below this line]\r\n\r\n<details py36>\r\n\r\n    INSTALLED VERSIONS\r\n    ------------------\r\n    commit: None\r\n    python: 3.6.1.final.0\r\n    python-bits: 64\r\n    OS: Linux\r\n    OS-release: 4.9.0-3-amd64\r\n    machine: x86_64\r\n    processor: \r\n    byteorder: little\r\n    LC_ALL: None\r\n    LANG: en_US.UTF-8\r\n    LOCALE: en_US.UTF-8\r\n\r\n    pandas: 0.20.1\r\n    pytest: 3.0.7\r\n    pip: 9.0.1\r\n    setuptools: 27.2.0\r\n    Cython: 0.25.2\r\n    numpy: 1.12.1\r\n    scipy: 0.19.0\r\n    xarray: None\r\n    IPython: 5.3.0\r\n    sphinx: 1.5.6\r\n    patsy: 0.4.1\r\n    dateutil: 2.6.0\r\n    pytz: 2017.2\r\n    blosc: None\r\n    bottleneck: 1.2.1\r\n    tables: 3.3.0\r\n    numexpr: 2.6.2\r\n    feather: None\r\n    matplotlib: 2.0.2\r\n    openpyxl: 2.4.7\r\n    xlrd: 1.0.0\r\n    xlwt: 1.2.0\r\n    xlsxwriter: 0.9.6\r\n    lxml: 3.7.3\r\n    bs4: 4.6.0\r\n    html5lib: 0.999\r\n    sqlalchemy: 1.1.9\r\n    pymysql: None\r\n    psycopg2: None\r\n    jinja2: 2.9.6\r\n    s3fs: None\r\n    pandas_gbq: None\r\n    pandas_datareader: None\r\n\r\n</details py36>\r\n\r\n<details py27>\r\n\r\n    INSTALLED VERSIONS\r\n    ------------------\r\n    commit: None\r\n    python: 2.7.13.final.0\r\n    python-bits: 64\r\n    OS: Linux\r\n    OS-release: 4.9.0-3-amd64\r\n    machine: x86_64\r\n    processor: \r\n    byteorder: little\r\n    LC_ALL: None\r\n    LANG: en_US.UTF-8\r\n    LOCALE: None.None\r\n\r\n    pandas: 0.20.1\r\n    pytest: 3.0.7\r\n    pip: 9.0.1\r\n    setuptools: 27.2.0\r\n    Cython: 0.25.2\r\n    numpy: 1.11.2\r\n    scipy: 0.18.1\r\n    xarray: None\r\n    IPython: 5.3.0\r\n    sphinx: 1.5.6\r\n    patsy: 0.4.1\r\n    dateutil: 2.6.0\r\n    pytz: 2017.2\r\n    blosc: None\r\n    bottleneck: 1.2.1\r\n    tables: 3.3.0\r\n    numexpr: 2.6.1\r\n    feather: None\r\n    matplotlib: 2.0.2\r\n    openpyxl: 2.4.7\r\n    xlrd: 1.0.0\r\n    xlwt: 1.2.0\r\n    xlsxwriter: 0.9.6\r\n    lxml: 3.7.3\r\n    bs4: 4.6.0\r\n    html5lib: 0.999\r\n    sqlalchemy: 1.1.9\r\n    pymysql: None\r\n    psycopg2: None\r\n    jinja2: 2.9.6\r\n    s3fs: None\r\n    pandas_gbq: None\r\n    pandas_datareader: None\r\n</details py27>\r\n"},"comment":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/comments/329978261","html_url":"https://github.com/pandas-dev/pandas/issues/17540#issuecomment-329978261","issue_url":"https://api.github.com/repos/pandas-dev/pandas/issues/17540","id":329978261,"user":{"login":"zoof","id":1737483,"avatar_url":"https://avatars0.githubusercontent.com/u/1737483?v=4","gravatar_id":"","url":"https://api.github.com/users/zoof","html_url":"https://github.com/zoof","followers_url":"https://api.github.com/users/zoof/followers","following_url":"https://api.github.com/users/zoof/following{/other_user}","gists_url":"https://api.github.com/users/zoof/gists{/gist_id}","starred_url":"https://api.github.com/users/zoof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zoof/subscriptions","organizations_url":"https://api.github.com/users/zoof/orgs","repos_url":"https://api.github.com/users/zoof/repos","events_url":"https://api.github.com/users/zoof/events{/privacy}","received_events_url":"https://api.github.com/users/zoof/received_events","type":"User","site_admin":false},"created_at":"2017-09-16T16:17:39Z","updated_at":"2017-09-16T16:17:39Z","author_association":"NONE","body":"Like this?\r\n\r\n```\r\nIn [3]: pd.DataFrame(['Executive Director of HR',\r\n 'Assistant Director of HR',\r\n 'Instructor Chair of Paramedics',\r\n 'Proctor Testing Center \\xe2\\x80\\x93 PT',\r\n 'Instructor \\xe2\\x80\\x93 Welding (Automotive)',\r\n 'Lab Tech \\xe2\\x80\\x93 Automotive \\xe2\\x80\\x93 PT',\r\n 'Lab Tech Technology \\xe2\\x80\\x93 PT',\r\n 'Maintenance Tech',\r\n 'Business Services Coordinator',\r\n 'Scheduler']).to_hdf('data.h5','data')\r\n\r\nIn [4]: pd.read_hdf('data.h5')\r\nOut[4]: \r\n                                     0\r\n0             Executive Director of HR\r\n1             Assistant Director of HR\r\n2       Instructor Chair of Paramedics\r\n3        Proctor Testing Center â PT\r\n4  Instructor â Welding (Automotive)\r\n5       Lab Tech â Automotive â PT\r\n6           Lab Tech Technology â PT\r\n7                     Maintenance Tech\r\n8        Business Services Coordinator\r\n9                            Scheduler\r\n\r\nIn [5]: pd.read_hdf('data16.h5')\r\n---------------------------------------------------------------------------\r\nUnicodeDecodeError                        Traceback (most recent call last)\r\n<ipython-input-5-2a895d140f15> in <module>()\r\n----> 1 pd.read_hdf('data16.h5')\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in read_hdf(path_or_buf, key, mode, **kwargs)\r\n    370                                      'contains multiple datasets.')\r\n    371             key = candidate_only_group._v_pathname\r\n--> 372         return store.select(key, auto_close=auto_close, **kwargs)\r\n    373     except:\r\n    374         # if there is an error, close the store\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in select(self, key, where, start, stop, columns, iterator, chunksize, auto_close, **kwargs)\r\n    740                            chunksize=chunksize, auto_close=auto_close)\r\n    741 \r\n--> 742         return it.get_result()\r\n    743 \r\n    744     def select_as_coordinates(\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in get_result(self, coordinates)\r\n   1447 \r\n   1448         # directly return the result\r\n-> 1449         results = self.func(self.start, self.stop, where)\r\n   1450         self.close()\r\n   1451         return results\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in func(_start, _stop, _where)\r\n    733             return s.read(start=_start, stop=_stop,\r\n    734                           where=_where,\r\n--> 735                           columns=columns, **kwargs)\r\n    736 \r\n    737         # create the iterator\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in read(self, **kwargs)\r\n   2751         kwargs = self.validate_read(kwargs)\r\n   2752         index = self.read_index('index', **kwargs)\r\n-> 2753         values = self.read_array('values', **kwargs)\r\n   2754         return Series(values, index=index, name=self.name)\r\n   2755 \r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py in read_array(self, key, start, stop)\r\n   2434         import tables\r\n   2435         node = getattr(self.group, key)\r\n-> 2436         data = node[start:stop]\r\n   2437         attrs = node._v_attrs\r\n   2438 \r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/vlarray.py in __getitem__(self, key)\r\n    673             start, stop, step = self._process_range(\r\n    674                 key.start, key.stop, key.step)\r\n--> 675             return self.read(start, stop, step)\r\n    676         # Try with a boolean or point selection\r\n    677         elif type(key) in (list, tuple) or isinstance(key, numpy.ndarray):\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/vlarray.py in read(self, start, stop, step)\r\n    813         atom = self.atom\r\n    814         if not hasattr(atom, 'size'):  # it is a pseudo-atom\r\n--> 815             outlistarr = [atom.fromarray(arr) for arr in listarr]\r\n    816         else:\r\n    817             # Convert the list to the right flavor\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/vlarray.py in <listcomp>(.0)\r\n    813         atom = self.atom\r\n    814         if not hasattr(atom, 'size'):  # it is a pseudo-atom\r\n--> 815             outlistarr = [atom.fromarray(arr) for arr in listarr]\r\n    816         else:\r\n    817             # Convert the list to the right flavor\r\n\r\n/home/tct/anaconda2/envs/py36/lib/python3.6/site-packages/tables/atom.py in fromarray(self, array)\r\n   1226         if array.size == 0:\r\n   1227             return None\r\n-> 1228         return six.moves.cPickle.loads(array.tostring())\r\n\r\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 23: ordinal not in range(128)\r\n```"}}
{"actor":{"display_login":"zoof"}}