{"action":"created","comment":{"url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments/109274556","pull_request_review_id":30371409,"id":109274556,"diff_hunk":"@@ -0,0 +1,496 @@\n+## Summary\n+ This example is to show how to load BigDL model as UDF to perform predictions in Spark SQL/Dataframes and StructuredStreaming (Spark 2.0+ only).\n+ \n+ First use a (pre-trained GloVe embedding) to convert word to vector,\n+ and uses it to train the text classification model on a 20 Newsgroup dataset\n+ with 20 different categories. This model can achieve around 90% accuracy after 2 epochs training.\n+(It was first described in: https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html)\n+\n+Then create a UDF to do the text classification with this model, and use this UDF to do the prediction in Spark SQL (spark 1.5, spark 1.6) and Structured Streaming (spark 2.0).\n+## Data\n+* Embedding: 100-dimensional pre-trained GloVe embeddings of 400k words which trained on a 2014 dump of English Wikipedia.\n+* Training data: \"20 Newsgroup dataset\" which containing 20 categories and with totally 19997 texts.\n+\n+## Get the JAR\n+Please build the source code with your specific version of spark referring the\n+                                                              [Build Page](https://github.com/intel-analytics/BigDL/wiki/Build-Page).\n+\n+\n+## Steps to run this example:\n+1.  Download [Pre-train GloVe word embeddings](http://nlp.stanford.edu/data/glove.6B.zip)\n+\n+    ```shell\n+    wget http://nlp.stanford.edu/data/glove.6B.zip\n+    unzip -q glove.6B.zip -d glove.6B\n+    ```\n+\n+2.  Download [20 Newsgroup dataset](http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/news20.html) as the training data\n+\n+    ```shell\n+    wget http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/news20.tar.gz\n+    tar zxf news20.tar.gz\n+    ```\n+\n+3.  Put those data under BASE_DIR, and the final structure would look like this:\n+\n+    ```\n+    [~/textclassification]$ tree . -L 1\n+    .\n+    ├── 20_newsgroup\n+    └── glove.6B\n+    ```\n+\n+4. Run this command to generate test data for prediction in Structured Streaming:\n+    ```bash\n+    usage: create_test_texts.py [-h] [-s SRC] [-t TEST]\n+    \n+    Create test text files from source\n+    \n+    optional arguments:\n+     -h, --help            show this help message and exit\n+     -s SRC, --src SRC     source directory\n+     -t TEST, --test TEST  test directory\n+     \n+    Example: python ./create_test_texts.py -s BASE_DIR/20_newsgroup -t BASE_DIR/test\n+       \n+    ```  \n+    \n+5. Run Spark SQL example\n+\n+* Training the model and make predictions.\n+   \n+   Run the commands:\n+   \n+   If you do not have the pre-trained model, you need to use this command to train the model and use this model to predict text classification of text records with UDF.\n+\n+   Example:\n+\n+   ```shell\n+   BASE_DIR=${PWD} # where is the data\n+   MASTER=loca[*] # the master url\n+   ./dist/bin/bigdl.sh -- spark-submit --master $MASTER --driver-memory 20g \\\n+       --class com.intel.analytics.bigdl.example.modeludf.BatchPredictor \\\n+         ./dist/lib/bigdl-0.1.0-SNAPSHOT-jar-with-dependencies.jar \\\n+       -c 4 \\\n+       -n 1 \\\n+       --batchSize 32 \\\n+       --baseDir $BASE_DIR \\\n+       --partitionNum 4 \\\n+       --checkpoint  $BASE_DIR/model/text \\\n+       --dataDir $BASE_DIR/test\n+   ```\n+        \n+   In the above commands, \n+   \n+   -c: How many cores of your machine will be used in the training. Note that the core number should be physical core number. If your machine turn on hyper threading, one physical core will map to two OS core.\n+   \n+   -n: Node number.\n+   \n+   --batchSize: how many text files to be trained at one time\n+   \n+   -baseDir: folder containing trainning text files and word2Vec embedding.\n+   \n+   --partitionNum: number to partition training data\n+   \n+   --checkpoint: location to save model\n+   \n+   --dataDir: Directory containing the test data\n+\n+   If you have saved model, you need to use this command to predict text classification of text records with UDF.\n+      \n+   Example:\n+     \n+   ```shell\n+   BASE_DIR=${PWD} # where is the data\n+   MASTER=loca[*] # the master url\n+   ./bigdl.sh -- spark-submit --master $MASTER --driver-memory 5g \\\n+              --class com.intel.analytics.bigdl.example.modeludf.BatchPredictor \\","path":"spark/dl/src/main/scala/com/intel/analytics/bigdl/example/udfpredictor/README.md","position":107,"original_position":107,"commit_id":"6e8d5c493373efb01b7752bffba8a4d522b82ac1","original_commit_id":"6e8d5c493373efb01b7752bffba8a4d522b82ac1","user":{"login":"jenniew","id":8434507,"avatar_url":"https://avatars3.githubusercontent.com/u/8434507?v=3","gravatar_id":"","url":"https://api.github.com/users/jenniew","html_url":"https://github.com/jenniew","followers_url":"https://api.github.com/users/jenniew/followers","following_url":"https://api.github.com/users/jenniew/following{/other_user}","gists_url":"https://api.github.com/users/jenniew/gists{/gist_id}","starred_url":"https://api.github.com/users/jenniew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jenniew/subscriptions","organizations_url":"https://api.github.com/users/jenniew/orgs","repos_url":"https://api.github.com/users/jenniew/repos","events_url":"https://api.github.com/users/jenniew/events{/privacy}","received_events_url":"https://api.github.com/users/jenniew/received_events","type":"User","site_admin":false},"body":"should be DataframePredictor","created_at":"2017-04-01T02:05:06Z","updated_at":"2017-04-01T02:36:48Z","html_url":"https://github.com/intel-analytics/BigDL/pull/660#discussion_r109274556","pull_request_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660","_links":{"self":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments/109274556"},"html":{"href":"https://github.com/intel-analytics/BigDL/pull/660#discussion_r109274556"},"pull_request":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660"}}},"pull_request":{"url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660","id":112409867,"html_url":"https://github.com/intel-analytics/BigDL/pull/660","diff_url":"https://github.com/intel-analytics/BigDL/pull/660.diff","patch_url":"https://github.com/intel-analytics/BigDL/pull/660.patch","issue_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/660","number":660,"state":"open","locked":false,"title":"Add UDF example for structured streaming (spark 2.x) and sql/dataframe (spark pre-2.0)","user":{"login":"yangw1234","id":10561966,"avatar_url":"https://avatars2.githubusercontent.com/u/10561966?v=3","gravatar_id":"","url":"https://api.github.com/users/yangw1234","html_url":"https://github.com/yangw1234","followers_url":"https://api.github.com/users/yangw1234/followers","following_url":"https://api.github.com/users/yangw1234/following{/other_user}","gists_url":"https://api.github.com/users/yangw1234/gists{/gist_id}","starred_url":"https://api.github.com/users/yangw1234/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yangw1234/subscriptions","organizations_url":"https://api.github.com/users/yangw1234/orgs","repos_url":"https://api.github.com/users/yangw1234/repos","events_url":"https://api.github.com/users/yangw1234/events{/privacy}","received_events_url":"https://api.github.com/users/yangw1234/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nAdd UDF example for structured streaming (spark 2.x) and sql/dataframe (spark pre-2.0).\r\n\r\nThis pr is based on #539. Thanks to @jenniew \r\n\r\n## How was this patch tested?\r\n\r\nRun this example according to README. Tested on spark 1.6.3, 2.0.2, 2.1.0\r\n\r\n","created_at":"2017-03-24T08:45:23Z","updated_at":"2017-04-01T02:36:48Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7357ca498bd4ac5c5b6387b55d7447b54f3432e4","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660/commits","review_comments_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660/comments","review_comment_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments{/number}","comments_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/660/comments","statuses_url":"https://api.github.com/repos/intel-analytics/BigDL/statuses/6e8d5c493373efb01b7752bffba8a4d522b82ac1","head":{"label":"yangw1234:udf","ref":"udf","sha":"6e8d5c493373efb01b7752bffba8a4d522b82ac1","user":{"login":"yangw1234","id":10561966,"avatar_url":"https://avatars2.githubusercontent.com/u/10561966?v=3","gravatar_id":"","url":"https://api.github.com/users/yangw1234","html_url":"https://github.com/yangw1234","followers_url":"https://api.github.com/users/yangw1234/followers","following_url":"https://api.github.com/users/yangw1234/following{/other_user}","gists_url":"https://api.github.com/users/yangw1234/gists{/gist_id}","starred_url":"https://api.github.com/users/yangw1234/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yangw1234/subscriptions","organizations_url":"https://api.github.com/users/yangw1234/orgs","repos_url":"https://api.github.com/users/yangw1234/repos","events_url":"https://api.github.com/users/yangw1234/events{/privacy}","received_events_url":"https://api.github.com/users/yangw1234/received_events","type":"User","site_admin":false},"repo":{"id":81434841,"name":"BigDL","full_name":"yangw1234/BigDL","owner":{"login":"yangw1234","id":10561966,"avatar_url":"https://avatars2.githubusercontent.com/u/10561966?v=3","gravatar_id":"","url":"https://api.github.com/users/yangw1234","html_url":"https://github.com/yangw1234","followers_url":"https://api.github.com/users/yangw1234/followers","following_url":"https://api.github.com/users/yangw1234/following{/other_user}","gists_url":"https://api.github.com/users/yangw1234/gists{/gist_id}","starred_url":"https://api.github.com/users/yangw1234/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yangw1234/subscriptions","organizations_url":"https://api.github.com/users/yangw1234/orgs","repos_url":"https://api.github.com/users/yangw1234/repos","events_url":"https://api.github.com/users/yangw1234/events{/privacy}","received_events_url":"https://api.github.com/users/yangw1234/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/yangw1234/BigDL","description":"BigDL: Distributed Deep Learning Library for Apache Spark","fork":true,"url":"https://api.github.com/repos/yangw1234/BigDL","forks_url":"https://api.github.com/repos/yangw1234/BigDL/forks","keys_url":"https://api.github.com/repos/yangw1234/BigDL/keys{/key_id}","collaborators_url":"https://api.github.com/repos/yangw1234/BigDL/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/yangw1234/BigDL/teams","hooks_url":"https://api.github.com/repos/yangw1234/BigDL/hooks","issue_events_url":"https://api.github.com/repos/yangw1234/BigDL/issues/events{/number}","events_url":"https://api.github.com/repos/yangw1234/BigDL/events","assignees_url":"https://api.github.com/repos/yangw1234/BigDL/assignees{/user}","branches_url":"https://api.github.com/repos/yangw1234/BigDL/branches{/branch}","tags_url":"https://api.github.com/repos/yangw1234/BigDL/tags","blobs_url":"https://api.github.com/repos/yangw1234/BigDL/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/yangw1234/BigDL/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/yangw1234/BigDL/git/refs{/sha}","trees_url":"https://api.github.com/repos/yangw1234/BigDL/git/trees{/sha}","statuses_url":"https://api.github.com/repos/yangw1234/BigDL/statuses/{sha}","languages_url":"https://api.github.com/repos/yangw1234/BigDL/languages","stargazers_url":"https://api.github.com/repos/yangw1234/BigDL/stargazers","contributors_url":"https://api.github.com/repos/yangw1234/BigDL/contributors","subscribers_url":"https://api.github.com/repos/yangw1234/BigDL/subscribers","subscription_url":"https://api.github.com/repos/yangw1234/BigDL/subscription","commits_url":"https://api.github.com/repos/yangw1234/BigDL/commits{/sha}","git_commits_url":"https://api.github.com/repos/yangw1234/BigDL/git/commits{/sha}","comments_url":"https://api.github.com/repos/yangw1234/BigDL/comments{/number}","issue_comment_url":"https://api.github.com/repos/yangw1234/BigDL/issues/comments{/number}","contents_url":"https://api.github.com/repos/yangw1234/BigDL/contents/{+path}","compare_url":"https://api.github.com/repos/yangw1234/BigDL/compare/{base}...{head}","merges_url":"https://api.github.com/repos/yangw1234/BigDL/merges","archive_url":"https://api.github.com/repos/yangw1234/BigDL/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/yangw1234/BigDL/downloads","issues_url":"https://api.github.com/repos/yangw1234/BigDL/issues{/number}","pulls_url":"https://api.github.com/repos/yangw1234/BigDL/pulls{/number}","milestones_url":"https://api.github.com/repos/yangw1234/BigDL/milestones{/number}","notifications_url":"https://api.github.com/repos/yangw1234/BigDL/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/yangw1234/BigDL/labels{/name}","releases_url":"https://api.github.com/repos/yangw1234/BigDL/releases{/id}","deployments_url":"https://api.github.com/repos/yangw1234/BigDL/deployments","created_at":"2017-02-09T09:47:20Z","updated_at":"2017-02-09T09:47:23Z","pushed_at":"2017-03-31T11:17:36Z","git_url":"git://github.com/yangw1234/BigDL.git","ssh_url":"git@github.com:yangw1234/BigDL.git","clone_url":"https://github.com/yangw1234/BigDL.git","svn_url":"https://github.com/yangw1234/BigDL","homepage":"","size":11926,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"intel-analytics:master","ref":"master","sha":"e78dcf0a1da041827889f25b9cea9e1b6502e1f9","user":{"login":"intel-analytics","id":10941215,"avatar_url":"https://avatars0.githubusercontent.com/u/10941215?v=3","gravatar_id":"","url":"https://api.github.com/users/intel-analytics","html_url":"https://github.com/intel-analytics","followers_url":"https://api.github.com/users/intel-analytics/followers","following_url":"https://api.github.com/users/intel-analytics/following{/other_user}","gists_url":"https://api.github.com/users/intel-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/intel-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/intel-analytics/subscriptions","organizations_url":"https://api.github.com/users/intel-analytics/orgs","repos_url":"https://api.github.com/users/intel-analytics/repos","events_url":"https://api.github.com/users/intel-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/intel-analytics/received_events","type":"Organization","site_admin":false},"repo":{"id":66823715,"name":"BigDL","full_name":"intel-analytics/BigDL","owner":{"login":"intel-analytics","id":10941215,"avatar_url":"https://avatars0.githubusercontent.com/u/10941215?v=3","gravatar_id":"","url":"https://api.github.com/users/intel-analytics","html_url":"https://github.com/intel-analytics","followers_url":"https://api.github.com/users/intel-analytics/followers","following_url":"https://api.github.com/users/intel-analytics/following{/other_user}","gists_url":"https://api.github.com/users/intel-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/intel-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/intel-analytics/subscriptions","organizations_url":"https://api.github.com/users/intel-analytics/orgs","repos_url":"https://api.github.com/users/intel-analytics/repos","events_url":"https://api.github.com/users/intel-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/intel-analytics/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/intel-analytics/BigDL","description":"BigDL: Distributed Deep Learning Library for Apache Spark","fork":false,"url":"https://api.github.com/repos/intel-analytics/BigDL","forks_url":"https://api.github.com/repos/intel-analytics/BigDL/forks","keys_url":"https://api.github.com/repos/intel-analytics/BigDL/keys{/key_id}","collaborators_url":"https://api.github.com/repos/intel-analytics/BigDL/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/intel-analytics/BigDL/teams","hooks_url":"https://api.github.com/repos/intel-analytics/BigDL/hooks","issue_events_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/events{/number}","events_url":"https://api.github.com/repos/intel-analytics/BigDL/events","assignees_url":"https://api.github.com/repos/intel-analytics/BigDL/assignees{/user}","branches_url":"https://api.github.com/repos/intel-analytics/BigDL/branches{/branch}","tags_url":"https://api.github.com/repos/intel-analytics/BigDL/tags","blobs_url":"https://api.github.com/repos/intel-analytics/BigDL/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/intel-analytics/BigDL/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/intel-analytics/BigDL/git/refs{/sha}","trees_url":"https://api.github.com/repos/intel-analytics/BigDL/git/trees{/sha}","statuses_url":"https://api.github.com/repos/intel-analytics/BigDL/statuses/{sha}","languages_url":"https://api.github.com/repos/intel-analytics/BigDL/languages","stargazers_url":"https://api.github.com/repos/intel-analytics/BigDL/stargazers","contributors_url":"https://api.github.com/repos/intel-analytics/BigDL/contributors","subscribers_url":"https://api.github.com/repos/intel-analytics/BigDL/subscribers","subscription_url":"https://api.github.com/repos/intel-analytics/BigDL/subscription","commits_url":"https://api.github.com/repos/intel-analytics/BigDL/commits{/sha}","git_commits_url":"https://api.github.com/repos/intel-analytics/BigDL/git/commits{/sha}","comments_url":"https://api.github.com/repos/intel-analytics/BigDL/comments{/number}","issue_comment_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/comments{/number}","contents_url":"https://api.github.com/repos/intel-analytics/BigDL/contents/{+path}","compare_url":"https://api.github.com/repos/intel-analytics/BigDL/compare/{base}...{head}","merges_url":"https://api.github.com/repos/intel-analytics/BigDL/merges","archive_url":"https://api.github.com/repos/intel-analytics/BigDL/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/intel-analytics/BigDL/downloads","issues_url":"https://api.github.com/repos/intel-analytics/BigDL/issues{/number}","pulls_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls{/number}","milestones_url":"https://api.github.com/repos/intel-analytics/BigDL/milestones{/number}","notifications_url":"https://api.github.com/repos/intel-analytics/BigDL/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/intel-analytics/BigDL/labels{/name}","releases_url":"https://api.github.com/repos/intel-analytics/BigDL/releases{/id}","deployments_url":"https://api.github.com/repos/intel-analytics/BigDL/deployments","created_at":"2016-08-29T07:59:50Z","updated_at":"2017-03-31T20:16:37Z","pushed_at":"2017-04-01T02:24:43Z","git_url":"git://github.com/intel-analytics/BigDL.git","ssh_url":"git@github.com:intel-analytics/BigDL.git","clone_url":"https://github.com/intel-analytics/BigDL.git","svn_url":"https://github.com/intel-analytics/BigDL","homepage":"","size":10486,"stargazers_count":1618,"watchers_count":1618,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":292,"mirror_url":null,"open_issues_count":88,"forks":292,"open_issues":88,"watchers":1618,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660"},"html":{"href":"https://github.com/intel-analytics/BigDL/pull/660"},"issue":{"href":"https://api.github.com/repos/intel-analytics/BigDL/issues/660"},"comments":{"href":"https://api.github.com/repos/intel-analytics/BigDL/issues/660/comments"},"review_comments":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660/comments"},"review_comment":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/660/commits"},"statuses":{"href":"https://api.github.com/repos/intel-analytics/BigDL/statuses/6e8d5c493373efb01b7752bffba8a4d522b82ac1"}}}}
{"actor":{"display_login":"jenniew"}}