{"action":"opened","issue":{"url":"https://api.github.com/repos/webanno/webanno/issues/736","repository_url":"https://api.github.com/repos/webanno/webanno","labels_url":"https://api.github.com/repos/webanno/webanno/issues/736/labels{/name}","comments_url":"https://api.github.com/repos/webanno/webanno/issues/736/comments","events_url":"https://api.github.com/repos/webanno/webanno/issues/736/events","html_url":"https://github.com/webanno/webanno/issues/736","id":276765209,"number":736,"title":"Doc clarification: definition of token","user":{"login":"sanmai-NL","id":3374183,"avatar_url":"https://avatars3.githubusercontent.com/u/3374183?v=4","gravatar_id":"","url":"https://api.github.com/users/sanmai-NL","html_url":"https://github.com/sanmai-NL","followers_url":"https://api.github.com/users/sanmai-NL/followers","following_url":"https://api.github.com/users/sanmai-NL/following{/other_user}","gists_url":"https://api.github.com/users/sanmai-NL/gists{/gist_id}","starred_url":"https://api.github.com/users/sanmai-NL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanmai-NL/subscriptions","organizations_url":"https://api.github.com/users/sanmai-NL/orgs","repos_url":"https://api.github.com/users/sanmai-NL/repos","events_url":"https://api.github.com/users/sanmai-NL/events{/privacy}","received_events_url":"https://api.github.com/users/sanmai-NL/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-11-25T17:32:02Z","updated_at":"2017-11-25T17:32:02Z","closed_at":null,"author_association":"NONE","body":"In https://zoidberg.ukp.informatik.tu-darmstadt.de/jenkins/job/WebAnno%20(GitHub)%20(master)/de.tudarmstadt.ukp.clarin.webanno$webanno-doc/doclinks/1/#sect_webannotsv, AFAICT there is no definition of ‘token’. I suppose you mean the following:\r\n\r\n1. A **token** is the longest sequence of grapheme clusters in which no code point has the Unicode character property `WSpace=Y`. \r\n2. Each token has an **index** that is an integer ≥ 1.\r\n3. As the **sentence** is read in writing direction, the first token has index 1, whitespace is skipped, then the next token has index 2, and so forth.\r\n\r\nBetter still, can the grammar of TSV3 be based on a well-defined tokenization algorithm such as word boundaries according to [Unicode Standard Annex #29](http://unicode.org/reports/tr29/#Word_Boundaries)?"}}
{"actor":{"display_login":"sanmai-NL"}}