{"action":"created","issue":{"url":"https://api.github.com/repos/ropensci/tokenizers/issues/32","repository_url":"https://api.github.com/repos/ropensci/tokenizers","labels_url":"https://api.github.com/repos/ropensci/tokenizers/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/ropensci/tokenizers/issues/32/comments","events_url":"https://api.github.com/repos/ropensci/tokenizers/issues/32/events","html_url":"https://github.com/ropensci/tokenizers/issues/32","id":214358182,"number":32,"title":"`tokenize_ngram` deal with colon(:) inconsistently across different platforms","user":{"login":"everdark","id":3348233,"avatar_url":"https://avatars0.githubusercontent.com/u/3348233?v=3","gravatar_id":"","url":"https://api.github.com/users/everdark","html_url":"https://github.com/everdark","followers_url":"https://api.github.com/users/everdark/followers","following_url":"https://api.github.com/users/everdark/following{/other_user}","gists_url":"https://api.github.com/users/everdark/gists{/gist_id}","starred_url":"https://api.github.com/users/everdark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/everdark/subscriptions","organizations_url":"https://api.github.com/users/everdark/orgs","repos_url":"https://api.github.com/users/everdark/repos","events_url":"https://api.github.com/users/everdark/events{/privacy}","received_events_url":"https://api.github.com/users/everdark/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-03-15T11:17:56Z","updated_at":"2017-03-25T20:39:18Z","closed_at":null,"body":"Hi,\r\n\r\nRecently I just came across one issue that make me confused.\r\nOn my macbook I will have the results:\r\n```\r\n> library(tokenizers)\r\n> tokenize_ngrams(\"name:kyle\", n=1)\r\n[[1]]\r\n[1] \"name\" \"kyle\"\r\n```\r\n\r\nHowever on my Ubuntu machine it becomes\r\n```r\r\n> library(tokenizers)\r\n> tokenize_ngrams(\"name:kyle\", n=1)\r\n[[1]]\r\n[1] \"name:kyle\"\r\n```\r\n\r\nIf there is any space in between the colon and the other words then both case will be two tokens spearated. \r\n\r\nMy `sessionInfo` for two machines are as the followings:\r\n\r\n```r\r\nR version 3.3.2 (2016-10-31)\r\nPlatform: x86_64-apple-darwin13.4.0 (64-bit)\r\nRunning under: macOS Sierra 10.12.3\r\n\r\nlocale:\r\n[1] C/UTF-8/C/C/C/C\r\n\r\nattached base packages:\r\n[1] stats     graphics  grDevices utils     datasets  methods   base\r\n\r\nother attached packages:\r\n[1] nvimcom_0.9-26    colorout_1.1-2    magrittr_1.5      data.table_1.10.4\r\n\r\nloaded via a namespace (and not attached):\r\n[1] tools_3.3.2\r\n```\r\n\r\n```r\r\nR version 3.3.1 (2016-06-21)\r\nPlatform: x86_64-pc-linux-gnu (64-bit)\r\nRunning under: Ubuntu 16.04.1 LTS\r\n\r\nlocale:\r\n [1] LC_CTYPE=C                 LC_NUMERIC=C\r\n [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8\r\n [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8\r\n [7] LC_PAPER=en_US.UTF-8       LC_NAME=C\r\n [9] LC_ADDRESS=C               LC_TELEPHONE=C\r\n[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C\r\n\r\nattached base packages:\r\n[1] stats     graphics  grDevices utils     datasets  methods   base\r\n```\r\n\r\nAm I missing something about this difference? Is it a locale issue? Any help is appreciated.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ropensci/tokenizers/issues/comments/289238085","html_url":"https://github.com/ropensci/tokenizers/issues/32#issuecomment-289238085","issue_url":"https://api.github.com/repos/ropensci/tokenizers/issues/32","id":289238085,"user":{"login":"lmullen","id":183672,"avatar_url":"https://avatars3.githubusercontent.com/u/183672?v=3","gravatar_id":"","url":"https://api.github.com/users/lmullen","html_url":"https://github.com/lmullen","followers_url":"https://api.github.com/users/lmullen/followers","following_url":"https://api.github.com/users/lmullen/following{/other_user}","gists_url":"https://api.github.com/users/lmullen/gists{/gist_id}","starred_url":"https://api.github.com/users/lmullen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmullen/subscriptions","organizations_url":"https://api.github.com/users/lmullen/orgs","repos_url":"https://api.github.com/users/lmullen/repos","events_url":"https://api.github.com/users/lmullen/events{/privacy}","received_events_url":"https://api.github.com/users/lmullen/received_events","type":"User","site_admin":false},"created_at":"2017-03-25T20:39:18Z","updated_at":"2017-03-25T20:39:18Z","body":"Tokenization functions all call functions from the stringi package, which makes some determinations about word boundaries based on the locale. You might try using `stringi::stri_locale_set()` to set the locale on both machines to be the same. "}}
{"actor":{"display_login":"lmullen"}}