{"action":"created","issue":{"url":"https://api.github.com/repos/kbenoit/quanteda/issues/740","repository_url":"https://api.github.com/repos/kbenoit/quanteda","labels_url":"https://api.github.com/repos/kbenoit/quanteda/issues/740/labels{/name}","comments_url":"https://api.github.com/repos/kbenoit/quanteda/issues/740/comments","events_url":"https://api.github.com/repos/kbenoit/quanteda/issues/740/events","html_url":"https://github.com/kbenoit/quanteda/issues/740","id":229657721,"number":740,"title":"tokens_select not selecting ngrams","user":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars1.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"labels":[{"id":606938933,"url":"https://api.github.com/repos/kbenoit/quanteda/labels/design","name":"design","color":"006b75","default":false}],"state":"open","locked":false,"assignee":{"login":"koheiw","id":6572963,"avatar_url":"https://avatars1.githubusercontent.com/u/6572963?v=3","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false},"assignees":[{"login":"koheiw","id":6572963,"avatar_url":"https://avatars1.githubusercontent.com/u/6572963?v=3","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kbenoit/quanteda/milestones/5","html_url":"https://github.com/kbenoit/quanteda/milestone/5","labels_url":"https://api.github.com/repos/kbenoit/quanteda/milestones/5/labels","id":2440125,"number":5,"title":"CRAN v0.9.9.9000","description":"Last pre-1.0 release","creator":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars1.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":10,"state":"open","created_at":"2017-04-06T17:29:08Z","updated_at":"2017-06-19T15:08:07Z","due_on":"2017-05-30T07:00:00Z","closed_at":null},"comments":6,"created_at":"2017-05-18T12:41:29Z","updated_at":"2017-06-19T16:15:55Z","closed_at":null,"body":"I thought we had addressed this already, but maybe this is part of #719.\r\n\r\n### for tokens\r\n\r\nDefine two sets of tokens, simple unigrams and space-separated bigrams:\r\n```r\r\n(toks <- tokens(\"a b c d e f g\"))\r\n# tokens from 1 document.\r\n# Component 1 :\r\n# [1] \"a\" \"b\" \"c\" \"d\" \"e\" \"f\" \"g\"\r\n\r\n(toks2 <- tokens_ngrams(toks, n = 2, concatenator = \" \"))\r\n# tokens from 1 document.\r\n# Component 1 :\r\n# [1] \"a b\" \"b c\" \"c d\" \"d e\" \"e f\" \"f g\"\r\n```\r\n\r\nThen the first works, but the second fails. I'd like the second to work the same as the first.\r\n```r\r\ntokens_select(toks, c(\"a\", \"e\", \"g\"))\r\n# tokens from 1 document.\r\n# Component 1 :\r\n# [1] \"a\" \"e\" \"g\"\r\n\r\ntokens_select(toks2, c(\"a b\", \"d e\", \"f g\"))\r\n# tokens from 1 document.\r\n# Component 1 :\r\n# character(0)\r\n```\r\n\r\n### for dfms \r\n\r\nThe behaviour is correct for dfms already formed, but incorrect when forming, because I think `tokens_select()` is called before creating the dfm and then using `dfm_select()` inside `dfm()`.\r\n```r\r\ndfm(toks2, select = c(\"a b\", \"d e\", \"f g\"))\r\n# NULL\r\n\r\ndfm_select(dfm(toks2), c(\"a b\", \"d e\", \"f g\"))\r\n# Document-feature matrix of: 1 document, 3 features (0% sparse).\r\n# 1 x 3 sparse Matrix of class \"dfmSparse\"\r\n#        features\r\n# docs    a b d e f g\r\n#   text1   1   1   1\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kbenoit/quanteda/issues/comments/309490049","html_url":"https://github.com/kbenoit/quanteda/issues/740#issuecomment-309490049","issue_url":"https://api.github.com/repos/kbenoit/quanteda/issues/740","id":309490049,"user":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars1.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"created_at":"2017-06-19T16:15:55Z","updated_at":"2017-06-19T16:15:55Z","body":"will do."}}
{"actor":{"display_login":"kbenoit"}}