{"action":"created","issue":{"url":"https://api.github.com/repos/Rdatatable/data.table/issues/1828","repository_url":"https://api.github.com/repos/Rdatatable/data.table","labels_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1828/labels{/name}","comments_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1828/comments","events_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1828/events","html_url":"https://github.com/Rdatatable/data.table/issues/1828","id":173510650,"number":1828,"title":"[Request] data.table spark interface","user":{"login":"ysgit","id":898918,"avatar_url":"https://avatars2.githubusercontent.com/u/898918?v=4","gravatar_id":"","url":"https://api.github.com/users/ysgit","html_url":"https://github.com/ysgit","followers_url":"https://api.github.com/users/ysgit/followers","following_url":"https://api.github.com/users/ysgit/following{/other_user}","gists_url":"https://api.github.com/users/ysgit/gists{/gist_id}","starred_url":"https://api.github.com/users/ysgit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ysgit/subscriptions","organizations_url":"https://api.github.com/users/ysgit/orgs","repos_url":"https://api.github.com/users/ysgit/repos","events_url":"https://api.github.com/users/ysgit/events{/privacy}","received_events_url":"https://api.github.com/users/ysgit/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-08-26T18:08:05Z","updated_at":"2017-11-09T18:41:16Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"data.table is awesome but most people don't have 100GB memory in order to handle really large data sets in memory.\n\nBig progress has been made making the Apache Spark framework available through R in the last couple of years. Two such projects are [Apache's sparkr](https://spark.apache.org/docs/latest/sparkr.html) and [Rstudio's sparklyr](http://spark.rstudio.com/). Both of these provide a dplyr style interface to spark's data processing engine.\n\nAs a heavy data.table user it would be amazing if there were to be a data.table interface for spark. That would make it incredibly easy for data scientists to migrate their projects from the smaller CSV style data sets to the huge data sets that can be processed by spark.\n\nA classic data pipeline for me is\n1. Bring the data into R by CSV\n2. Do some pre-processing (filters, joins, aggregation, feature extraction) of the data using data.table\n3. Build a model using one of R's many machine learning packages\n\nI want to be able to migrate this to\n1. Connect to data on hadoop cluster\n2. Do some pre-processing (filters, joins, aggregation, feature extraction) of the data using data.table's spark interface\n3. Build a model using one of spark's many machine learning algorithm.\n"},"comment":{"url":"https://api.github.com/repos/Rdatatable/data.table/issues/comments/343250622","html_url":"https://github.com/Rdatatable/data.table/issues/1828#issuecomment-343250622","issue_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1828","id":343250622,"user":{"login":"aadler","id":5596401,"avatar_url":"https://avatars2.githubusercontent.com/u/5596401?v=4","gravatar_id":"","url":"https://api.github.com/users/aadler","html_url":"https://github.com/aadler","followers_url":"https://api.github.com/users/aadler/followers","following_url":"https://api.github.com/users/aadler/following{/other_user}","gists_url":"https://api.github.com/users/aadler/gists{/gist_id}","starred_url":"https://api.github.com/users/aadler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aadler/subscriptions","organizations_url":"https://api.github.com/users/aadler/orgs","repos_url":"https://api.github.com/users/aadler/repos","events_url":"https://api.github.com/users/aadler/events{/privacy}","received_events_url":"https://api.github.com/users/aadler/received_events","type":"User","site_admin":false},"created_at":"2017-11-09T18:41:16Z","updated_at":"2017-11-09T18:41:16Z","author_association":"NONE","body":"Adding my voice to a desire to be able to use data.table instead of dplyr on distributed data in Spark/H20"}}
{"actor":{"display_login":"aadler"}}