{"action":"created","issue":{"url":"https://api.github.com/repos/MRtrix3/mrtrix3/issues/953","repository_url":"https://api.github.com/repos/MRtrix3/mrtrix3","labels_url":"https://api.github.com/repos/MRtrix3/mrtrix3/issues/953/labels{/name}","comments_url":"https://api.github.com/repos/MRtrix3/mrtrix3/issues/953/comments","events_url":"https://api.github.com/repos/MRtrix3/mrtrix3/issues/953/events","html_url":"https://github.com/MRtrix3/mrtrix3/issues/953","id":219267046,"number":953,"title":"decode('utf-8') falls over on alternative encodings","user":{"login":"bjeurissen","id":5675416,"avatar_url":"https://avatars2.githubusercontent.com/u/5675416?v=3","gravatar_id":"","url":"https://api.github.com/users/bjeurissen","html_url":"https://github.com/bjeurissen","followers_url":"https://api.github.com/users/bjeurissen/followers","following_url":"https://api.github.com/users/bjeurissen/following{/other_user}","gists_url":"https://api.github.com/users/bjeurissen/gists{/gist_id}","starred_url":"https://api.github.com/users/bjeurissen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bjeurissen/subscriptions","organizations_url":"https://api.github.com/users/bjeurissen/orgs","repos_url":"https://api.github.com/users/bjeurissen/repos","events_url":"https://api.github.com/users/bjeurissen/events{/privacy}","received_events_url":"https://api.github.com/users/bjeurissen/received_events","type":"User","site_admin":false},"labels":[{"id":96478020,"url":"https://api.github.com/repos/MRtrix3/mrtrix3/labels/bug","name":"bug","color":"fc2929","default":true},{"id":318389595,"url":"https://api.github.com/repos/MRtrix3/mrtrix3/labels/scripts","name":"scripts","color":"bfe5bf","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-04-04T14:20:41Z","updated_at":"2017-04-11T12:51:11Z","closed_at":null,"body":"I came across DICOM data sets that contain non utf-8 meta information, (e.g. accute accents stored as latin1), causing run.command() calls to fall over:\r\n\r\n```\r\n'utf-8' codec can't decode byte 0xc9 in position 85: invalid continuation byte\r\n```\r\nAs an ugly work-around, I have added ``errors='replace'`` to all decode('utf-8') calls, however, not sure what the best way of properly handling this would be... "},"comment":{"url":"https://api.github.com/repos/MRtrix3/mrtrix3/issues/comments/293250002","html_url":"https://github.com/MRtrix3/mrtrix3/issues/953#issuecomment-293250002","issue_url":"https://api.github.com/repos/MRtrix3/mrtrix3/issues/953","id":293250002,"user":{"login":"jdtournier","id":3221000,"avatar_url":"https://avatars0.githubusercontent.com/u/3221000?v=3","gravatar_id":"","url":"https://api.github.com/users/jdtournier","html_url":"https://github.com/jdtournier","followers_url":"https://api.github.com/users/jdtournier/followers","following_url":"https://api.github.com/users/jdtournier/following{/other_user}","gists_url":"https://api.github.com/users/jdtournier/gists{/gist_id}","starred_url":"https://api.github.com/users/jdtournier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdtournier/subscriptions","organizations_url":"https://api.github.com/users/jdtournier/orgs","repos_url":"https://api.github.com/users/jdtournier/repos","events_url":"https://api.github.com/users/jdtournier/events{/privacy}","received_events_url":"https://api.github.com/users/jdtournier/received_events","type":"User","site_admin":false},"created_at":"2017-04-11T12:51:11Z","updated_at":"2017-04-11T12:51:11Z","body":"Indeed, that `pydicom` link might come in handy if we're to support this properly. Ideally, I think we should aim to have all text converted to UTF8 directly within the DICOM import code, so that it is guaranteed to be valid UFT8 by the time any script makes use of it. The problem is that [character encoding support is actually pretty useless in C++11](http://stackoverflow.com/a/17106065) (contrary to my earlier post). As far as I can tell, the only library that seems to be widely available and suitable for the job is [libiconv](https://www.gnu.org/software/libiconv/). But to get this working would require quite a few modifications, from testing for the presence of `libiconv` in `configure` and checking that we can compile against it, and then implementing the actual conversion itself, first for `iso8859` itself, and then optionally for any encoding specified in the `(0008,0005)` tag, including figuring out the mapping between the encoding specified in the tag, and its representation in `libiconv`... Not something I'll have the time or inclination to look into any time soon, but as far as the C++ code itself is concerned, [this would be the place to do the conversion ](https://github.com/MRtrix3/mrtrix3/blob/master/lib/file/dicom/element.cpp#L323-L337) itself, if you're game?"}}
{"actor":{"display_login":"jdtournier"}}