{"action":"created","issue":{"url":"https://api.github.com/repos/tidyverse/dplyr/issues/3052","repository_url":"https://api.github.com/repos/tidyverse/dplyr","labels_url":"https://api.github.com/repos/tidyverse/dplyr/issues/3052/labels{/name}","comments_url":"https://api.github.com/repos/tidyverse/dplyr/issues/3052/comments","events_url":"https://api.github.com/repos/tidyverse/dplyr/issues/3052/events","html_url":"https://github.com/tidyverse/dplyr/issues/3052","id":252649599,"number":3052,"title":"dplyr unnest_tokens not working","user":{"login":"AngryR11","id":31298353,"avatar_url":"https://avatars1.githubusercontent.com/u/31298353?v=4","gravatar_id":"","url":"https://api.github.com/users/AngryR11","html_url":"https://github.com/AngryR11","followers_url":"https://api.github.com/users/AngryR11/followers","following_url":"https://api.github.com/users/AngryR11/following{/other_user}","gists_url":"https://api.github.com/users/AngryR11/gists{/gist_id}","starred_url":"https://api.github.com/users/AngryR11/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AngryR11/subscriptions","organizations_url":"https://api.github.com/users/AngryR11/orgs","repos_url":"https://api.github.com/users/AngryR11/repos","events_url":"https://api.github.com/users/AngryR11/events{/privacy}","received_events_url":"https://api.github.com/users/AngryR11/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-08-24T15:58:38Z","updated_at":"2017-09-04T19:12:12Z","closed_at":null,"author_association":"NONE","body":"I am loading one of the 5-core datasets from\r\n\r\n[Reviews](http://jmcauley.ucsd.edu/data/amazon/)\r\n\r\nusing\r\n\r\n```\r\nlibrary(sparklyr)\r\nlibrary(dplyr)\r\n```\r\n\r\n```\r\nconfig <- spark_config()\r\nconfig$`sparklyr.shell.driver-memory` <- \"2G\"\r\nsc = spark_connect(master = \"local\",config = config)\r\ndf=spark_read_json(sc = sc, name = \"videos\", path = \"Path/to/reviews_Office_Products_5.json\")\r\n```\r\nwhere one of the variables is a column of text reviews, likewise:\r\n\r\n`select(df,reviewText)`\r\n\r\n```\r\n# Source: lazy query [?? x 1]\r\n\r\n# Database: spark_connection reviewText\r\n\r\n1 I bought my first HP12C in about 1984 or so, and it served me faithfully until 2002 wh\r\n\r\n2 \"WHY THIS BELATED REVIEW? I feel very obliged to share my views about this old workhor\r\n\r\n3 I have an HP 48GX that has been kicking for more than twenty years and an HP 11 that i\r\n\r\n4 I've started doing more finance stuff recently and went looking for a good time-value-\r\n\r\n5 For simple calculations and discounted cash flows, this one is still the best. I used\r\n\r\n6 While I don't have an MBA, it's hard to believe that a calculator I learned how to use\r\n\r\n7 I've had an HP 12C ever since they were first available, roughly twenty years ago. I'\r\n\r\n8 Bought this for my boss because he lost his. He loves this calculator & would not be\r\n\r\n9 This is a well-designed, simple calculator that handles typical four-function math. La\r\n\r\n10 I love this calculator, big numbers and calculate excellent so easy to use and make my\r\n\r\n# ... with more rows\r\n```\r\n\r\n\r\nI want to split the reviews into tokens, with each row containing a word, but that has proven to be difficult. When I try to use the function unnest_tokens, I get the following error message:\r\n\r\n```\r\nlibrary(stringr)\r\nlibrary(tidytext) \r\n\r\nWord_by_Word <- df %>% unnest_tokens(word\r\n\r\n, reviewText)\r\n```\r\n`Error in unnest_tokens_.default(., word, reviewText) : unnest_tokens expects all columns of input to be atomic vectors (not lists)`\r\nHow do I fix this without using the command \"pull\" (which stores the data in memory) and coercing the data into the requested format? I can not pull the data as suggested in [Extract a dplyr tbl column as a vector](https://stackoverflow.com/questions/21618423/extract-a-dplyr-tbl-column-as-a-vector)  or convert the data into a tibble format, because if the database is too big and I do any of those, the computer runs out of memory even after increasing the 2G limit and running the program on a computer with a lot of memory (that's the hole point of using dplyr instead)."},"comment":{"url":"https://api.github.com/repos/tidyverse/dplyr/issues/comments/327016834","html_url":"https://github.com/tidyverse/dplyr/issues/3052#issuecomment-327016834","issue_url":"https://api.github.com/repos/tidyverse/dplyr/issues/3052","id":327016834,"user":{"login":"AngryR11","id":31298353,"avatar_url":"https://avatars1.githubusercontent.com/u/31298353?v=4","gravatar_id":"","url":"https://api.github.com/users/AngryR11","html_url":"https://github.com/AngryR11","followers_url":"https://api.github.com/users/AngryR11/followers","following_url":"https://api.github.com/users/AngryR11/following{/other_user}","gists_url":"https://api.github.com/users/AngryR11/gists{/gist_id}","starred_url":"https://api.github.com/users/AngryR11/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AngryR11/subscriptions","organizations_url":"https://api.github.com/users/AngryR11/orgs","repos_url":"https://api.github.com/users/AngryR11/repos","events_url":"https://api.github.com/users/AngryR11/events{/privacy}","received_events_url":"https://api.github.com/users/AngryR11/received_events","type":"User","site_admin":false},"created_at":"2017-09-04T19:12:12Z","updated_at":"2017-09-04T19:12:12Z","author_association":"NONE","body":"Hi @edgararuiz, thank you for the reply. I hv tried using ft_tokenizer, but got the following error:\r\n```\r\n> tr %>% ft_tokenizer(\"reviewText\")\r\nError in ensure_scalar_character(output.col) : \r\n  'NULL' is not a length-one character vector\r\n```"}}
{"actor":{"display_login":"AngryR11"}}