{"action":"created","issue":{"url":"https://api.github.com/repos/Sp2000/colplus/issues/37","repository_url":"https://api.github.com/repos/Sp2000/colplus","labels_url":"https://api.github.com/repos/Sp2000/colplus/issues/37/labels{/name}","comments_url":"https://api.github.com/repos/Sp2000/colplus/issues/37/comments","events_url":"https://api.github.com/repos/Sp2000/colplus/issues/37/events","html_url":"https://github.com/Sp2000/colplus/issues/37","id":270483464,"number":37,"title":"CoL+: first, do no harm","user":{"login":"Mesibov","id":16182234,"avatar_url":"https://avatars0.githubusercontent.com/u/16182234?v=4","gravatar_id":"","url":"https://api.github.com/users/Mesibov","html_url":"https://github.com/Mesibov","followers_url":"https://api.github.com/users/Mesibov/followers","following_url":"https://api.github.com/users/Mesibov/following{/other_user}","gists_url":"https://api.github.com/users/Mesibov/gists{/gist_id}","starred_url":"https://api.github.com/users/Mesibov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mesibov/subscriptions","organizations_url":"https://api.github.com/users/Mesibov/orgs","repos_url":"https://api.github.com/users/Mesibov/repos","events_url":"https://api.github.com/users/Mesibov/events{/privacy}","received_events_url":"https://api.github.com/users/Mesibov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-11-01T23:17:08Z","updated_at":"2017-11-02T11:28:24Z","closed_at":null,"author_association":"NONE","body":"I haven't noticed yet any comments about poor data quality in CoL+. It's like the smudge on the kitchen window — you know it's there, you will get around to cleaning it someday but it isn't a priority job. So the window remains smudged.\r\n\r\nThe \"data quality\" issues I'm talking about have nothing to do with whether a name use is backed with evidence, or whether an author misspelled a name or whether the correct authority has been cited or whether a URL is correct. CoL, like GBIF and WoRMS and many other aggregations, is riddled with **low-level** errors, like invalid data items, character encoding failures, incorrect formatting, duplications and truncated data items. As I wrote in a recent email to sp2000, these errors render a very large number of records completely useless for digital processing and tediously difficult for human processing.\r\n\r\nLow-level errors first appeared in the aggregations mainly because incoming data were either not audited at all or were not audited carefully enough. They're still in the aggregations because existing data are either not audited at all or are not audited carefully enough.\r\n\r\nThey will persist in CoL+ unless this project uses data migration as an opportunity to clean data. The likelihood of this happening, to judge from the overall workplan and some correspondence I've had, is close to zero.\r\n\r\nWhat's worse is that CoL+ may reduce further the quality of existing data. The Hippocratic principle \"primum non nocere\", \"first, do no harm\", was ignored when backbone taxonomies first appeared, to the horror of taxonomists and collection specialists. CoL+ hopes to fix the damage with explicit linking of names.\r\n\r\nBut CoL (and other aggregators) are guilty of data-mangling at lower levels, and with no plan to check for further data-mangling with the migration to CoL+, it will happen. In addition to character encoding failures and inadequate checking for duplicates, a surprisingly common stuff-up is truncation. As an example, CoL inconsistently truncates authors. There's an absolute limit of 100 characters, but shorter strings have also been truncated. In at least two source databases I'm aware of, the truncated strings appear full-length.\r\n\r\nEvery time I point out low-level data quality issues to aggregators I get the same answers (http://iphylo.blogspot.com.au/2016/04/guest-post-10-explanations-for-messy.html), which can be summarised as \"We're aware of the problem, but...\"  Will CoL+ be any different?"},"comment":{"url":"https://api.github.com/repos/Sp2000/colplus/issues/comments/341393326","html_url":"https://github.com/Sp2000/colplus/issues/37#issuecomment-341393326","issue_url":"https://api.github.com/repos/Sp2000/colplus/issues/37","id":341393326,"user":{"login":"Mesibov","id":16182234,"avatar_url":"https://avatars0.githubusercontent.com/u/16182234?v=4","gravatar_id":"","url":"https://api.github.com/users/Mesibov","html_url":"https://github.com/Mesibov","followers_url":"https://api.github.com/users/Mesibov/followers","following_url":"https://api.github.com/users/Mesibov/following{/other_user}","gists_url":"https://api.github.com/users/Mesibov/gists{/gist_id}","starred_url":"https://api.github.com/users/Mesibov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mesibov/subscriptions","organizations_url":"https://api.github.com/users/Mesibov/orgs","repos_url":"https://api.github.com/users/Mesibov/repos","events_url":"https://api.github.com/users/Mesibov/events{/privacy}","received_events_url":"https://api.github.com/users/Mesibov/received_events","type":"User","site_admin":false},"created_at":"2017-11-02T11:28:23Z","updated_at":"2017-11-02T11:28:23Z","author_association":"NONE","body":"I'll respond to parts of both Markus' and Rod's posts by saying that the current CoL, as evidenced in the 2017 Annual Checklist (DwC-A) I've downloaded, is not so much bad science as bad informatics. No one's asked me to do a full audit of low-level errors, so I haven't. I looked at taxa.txt and references.txt and found:\r\n\r\n- truncated data items\r\n- invalid data items, like non-numeric entries and impossible years in a date (year) field\r\n- incorrectly formatted data items, like non-capitalised genus and subgenus names and incorrectly punctuated strings\r\n- incorrect use of Darwin Core fields\r\n- character encoding failures where characters were replaced by gibberish strings or HTML equivalents (potentially reversible failures)\r\n- character encoding failures where characters were replaced by question marks (not reversible, original character lost forever)\r\n- invisible gremlin characters ranging from ancient ANSI control characters to vertical tabs to non-breaking spaces, and not appended to data, but embedded in strings\r\n\r\nYou don't have to be a taxonomist to find and recognise these, any more than you need to be an author to find spelling and punctuation mistakes in a publication. Since the download is simply a set of text files, each of these error types can be found in a few seconds. It didn't happen, or if it did, someone asked \"does it matter?\".\r\n\r\nI've previously suggested to another aggregator that they audit incoming data and politely ask the data provider to fix provider-side errors before the data are accepted. It was suggested in reply that it was the data provider's job to make sure the data are OK, and the aggregator's job to flag (ineffectually) some of the problems before the data go live. This explains why the data published by that particular aggregator are the lowest in quality of any I've audited.\r\n\r\nWe're moving away from the original point of my first post, which is that aggregators can and do lower the quality of the data they aggregate. At a high level it's by tinkering with taxonomy (see my last post), at a low level it's by truncating data items and introducing further character encoding failures. It would be nice if CoL+ not only strenuously avoided doing harm, but also actively repaired the harm done by contributors.\r\n\r\nIf CoL+ doesn't do this, why will it bother to aggregate and publish data? Just to (1) refresh its database management and (2) help alleviate the taxonomic woes of an inadequately funded GBIF (leaving GBIF's many other data issues for an inadequately funded GBIF to fix), as Rod suggests?\r\n\r\nIf there isn't data quality management baked into CoL+ before it gets off the ground, then Rod's suggestions are perfectly acceptable. If CoL+ informatics people are content with tolerating errors that informatics people can quickly find and fix, then how can this CoL+ exercise be taken seriously by end users?"}}
{"actor":{"display_login":"Mesibov"}}