{"action":"created","issue":{"url":"https://api.github.com/repos/HiSPARC/datastore/issues/21","repository_url":"https://api.github.com/repos/HiSPARC/datastore","labels_url":"https://api.github.com/repos/HiSPARC/datastore/issues/21/labels{/name}","comments_url":"https://api.github.com/repos/HiSPARC/datastore/issues/21/comments","events_url":"https://api.github.com/repos/HiSPARC/datastore/issues/21/events","html_url":"https://github.com/HiSPARC/datastore/pull/21","id":210624237,"number":21,"title":"Transistion to python 3","user":{"login":"tomkooij","id":6316468,"avatar_url":"https://avatars3.githubusercontent.com/u/6316468?v=3","gravatar_id":"","url":"https://api.github.com/users/tomkooij","html_url":"https://github.com/tomkooij","followers_url":"https://api.github.com/users/tomkooij/followers","following_url":"https://api.github.com/users/tomkooij/following{/other_user}","gists_url":"https://api.github.com/users/tomkooij/gists{/gist_id}","starred_url":"https://api.github.com/users/tomkooij/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomkooij/subscriptions","organizations_url":"https://api.github.com/users/tomkooij/orgs","repos_url":"https://api.github.com/users/tomkooij/repos","events_url":"https://api.github.com/users/tomkooij/events{/privacy}","received_events_url":"https://api.github.com/users/tomkooij/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-02-27T22:15:25Z","updated_at":"2017-03-13T10:47:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/HiSPARC/datastore/pulls/21","html_url":"https://github.com/HiSPARC/datastore/pull/21","diff_url":"https://github.com/HiSPARC/datastore/pull/21.diff","patch_url":"https://github.com/HiSPARC/datastore/pull/21.patch"},"body":"This is a port of the datastore to python 3.\r\n\r\n**tl;dr: It is not safe to merge this without (more) testing!**\r\n\r\nAlthough conversion is trivial, there are two major points to consider:\r\n - python 3 treats bytestring and strings very different from python 2. **Blobs are bytestrings** and stored as such. SAPPHiRE already deals with this.\r\n - **python 2 pickles are not readable in python 3**. This basically breaks the entire datastore. See note below.\r\n\r\nDue to the above caveats I have implemented some acceptance tests for both the writer and the WSGI app. These tests are certainly not unittests: I push something in and see what comes out. These tests were necessary, because things did break. The writer lacks coverage, but I feel critical py2<>py3 conversion parts are covered.\r\n\r\nToDo:\r\n - [x] 2to3 conversion\r\n - [x] Add tests for the WSGI app. (Using WebTest, so we don't need uWSGI)\r\n - [x] Add tests for the writer. Tests `process_data()`, not the writer main loop. Not all uploadcodes are tested. \r\n - [x] Add python 3 pickles to acceptance tests for the writer.\r\n - [x] Add tests to travis\r\n - [x] Add coverage [![Coverage Status](https://coveralls.io/repos/github/tomkooij/datastore/badge.svg?branch=datastore_py3)](https://coveralls.io/github/tomkooij/datastore?branch=datastore_py3)\r\n - [x] Test basic behavior in VM: the `send_fake_events.py` script works.\r\n - [ ] Test real live behavior in VM. Gather some (a lot?) real data and push into the VM. Check against py2 version.\r\n - [x] 2to3 `examples/datastore-config-server.py` (in provisioning as well)\r\n\r\n\r\nNotes about the unpickling mess:\r\n  - CPython 3 (py3) cannot read the CPython 2 (py2) protocol 0 pickles... Fortunately, I already dealt with this mess when doing the SAPPHiRE py2to3 conversion. (Fixed in PyTables).\r\n  - py3 can write protocol 0 (py2) pickles, but even those are unreadable in py3. At least that is consistent.\r\n  - py2 cannot read the default protocol used in py3.\r\n\r\nWhen python 3 (the datastore) reads a py2 pickle (from a station), `pickle` will throw `UnicodeDecodeError`, because the unpickled object is incompatible with `utf-8`. sigh. CPython 3.3 added `latin-1` encoding which unpickled most objects succesfully, however `datetime` still fails, throwing a `TypeError`. Did I say, sigh? CPython 3.4 added `bytes` encoding which basically leaves all strings as bytestring. Including dictonary keys (everything in `__dict__`). So that breaks everything.\r\n\r\nTo solve, we do this:\r\n```\r\ntry:\r\n   data = pickle.load(f)\r\nexcept UnicodeDecodeError:\r\n   data = decode_object(pickle.load(f, encoding='bytes'))\r\n```\r\n`decode_object()` recursively decodes all bytestring to strings. (Including things that actually are bytestrings, such as blobs. Because of base64 encoding of the blobs, we can safely do this. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/HiSPARC/datastore/issues/comments/286073518","html_url":"https://github.com/HiSPARC/datastore/pull/21#issuecomment-286073518","issue_url":"https://api.github.com/repos/HiSPARC/datastore/issues/21","id":286073518,"user":{"login":"davidfokkema","id":917137,"avatar_url":"https://avatars2.githubusercontent.com/u/917137?v=3","gravatar_id":"","url":"https://api.github.com/users/davidfokkema","html_url":"https://github.com/davidfokkema","followers_url":"https://api.github.com/users/davidfokkema/followers","following_url":"https://api.github.com/users/davidfokkema/following{/other_user}","gists_url":"https://api.github.com/users/davidfokkema/gists{/gist_id}","starred_url":"https://api.github.com/users/davidfokkema/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidfokkema/subscriptions","organizations_url":"https://api.github.com/users/davidfokkema/orgs","repos_url":"https://api.github.com/users/davidfokkema/repos","events_url":"https://api.github.com/users/davidfokkema/events{/privacy}","received_events_url":"https://api.github.com/users/davidfokkema/received_events","type":"User","site_admin":false},"created_at":"2017-03-13T10:47:34Z","updated_at":"2017-03-13T10:47:34Z","body":"Ah, ok! Just an example, I missed that."}}
{"actor":{"display_login":"davidfokkema"}}