{"action":"created","comment":{"url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments/128716803","pull_request_review_id":51421823,"id":128716803,"diff_hunk":"@@ -339,6 +343,128 @@ val bcastFeatures = sc.loadFeatures(\"my/features.adam\").broadcast()\n val readsByFeature = reads.broadcastRegionJoinAgainst(bcastFeatures)\n ```\n \n+To demonstrate how the RegionJoin APIs can be used to answer scientific\n+questions, we will walk through three common queries that can be written using\n+the RegionJoin API. First, we will perform a simple filter on genotypes based\n+on a file of features. We will then demonstrate a join and group by on variants\n+and features, providing variant data grouped by the feature they overlap.\n+Finally, we will separate reads into those that overlap and those that do not\n+overlap features from a feature file.\n+\n+Each of these demonstrations illustrates the difference between calling the\n+ShuffleRegionJoin and BroadcastRegionJoin and provides example code that can\n+be expanded from.\n+\n+###### Filter Genotypes by Features\n+\n+This query joins an RDD of Genotypes against an RDD of Features using an inner\n+join. Because this is an inner join, records from either dataset that don't\n+pair to the other are automatically dropped, providing the filter we are\n+interested in. This query is useful for trying to identify genotypes that\n+overlap features of interest. For example, if our feature file contains all the\n+exonic regions of the genome, this query would extract all genotypes that fall\n+in exonic regions.\n+\n+```scala\n+// Inner join will filter out genotypes not covered by a feature\n+val genotypes = sc.loadGenotypes(\"my/genotypes.adam\")\n+val features = sc.loadFeatures(\"my/features.adam\")\n+\n+// We can use ShuffleRegionJoin…\n+val joinedGenotypesShuffle = genotypes.shuffleRegionJoin(features)\n+\n+// …or BroadcastRegionJoin\n+val joinedGenotypesBcast = features.broadcastRegionJoin(genotypes)\n+\n+// In the case that we only want Genotypes, we can use a simple projection\n+val filteredGenotypesShuffle = joinedGenotypesShuffle.rdd.map(_._1)\n+\n+val filteredGenotypesBcast = joinedGenotypesBcast.rdd.map(_._2)\n+```\n+\n+After the join, we can perform a transform function on the resulting RDD to\n+manipulate it into providing the answer to our question. Since we were\n+interested in the `Genotype`s that overlap a `Feature`, we map over the tuples\n+and select just the `Genotype`.\n+\n+Since a broadcast join sends the left dataset to all executors, we chose to\n+send the `features` dataset because feature data is usually smaller in size\n+than genotypic data.\n+\n+###### Group overlapping variant data by the gene they overlap\n+\n+This query joins an RDD of Variants against an RDD of Features, and immediately\n+performs a group-by on the Feature. This produces an RDD whose elements are a\n+tuple containing a Feature, and all of the Variants overlapping the Feature.\n+This query is useful for trying to identify annotated variants that may","path":"docs/source/55_api.md","position":173,"original_position":173,"commit_id":"a49cd0409ffda9da9bb369ea4ec6972e43841587","original_commit_id":"a49cd0409ffda9da9bb369ea4ec6972e43841587","user":{"login":"heuermh","id":35293,"avatar_url":"https://avatars2.githubusercontent.com/u/35293?v=4","gravatar_id":"","url":"https://api.github.com/users/heuermh","html_url":"https://github.com/heuermh","followers_url":"https://api.github.com/users/heuermh/followers","following_url":"https://api.github.com/users/heuermh/following{/other_user}","gists_url":"https://api.github.com/users/heuermh/gists{/gist_id}","starred_url":"https://api.github.com/users/heuermh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/heuermh/subscriptions","organizations_url":"https://api.github.com/users/heuermh/orgs","repos_url":"https://api.github.com/users/heuermh/repos","events_url":"https://api.github.com/users/heuermh/events{/privacy}","received_events_url":"https://api.github.com/users/heuermh/received_events","type":"User","site_admin":false},"body":"This produces an RDD whose elements are tuples containing a Feature and all of the Variants overlapping the Feature.","created_at":"2017-07-21T09:07:06Z","updated_at":"2017-07-21T09:11:06Z","html_url":"https://github.com/bigdatagenomics/adam/pull/1605#discussion_r128716803","pull_request_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605","_links":{"self":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments/128716803"},"html":{"href":"https://github.com/bigdatagenomics/adam/pull/1605#discussion_r128716803"},"pull_request":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605"}}},"pull_request":{"url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605","id":130290657,"html_url":"https://github.com/bigdatagenomics/adam/pull/1605","diff_url":"https://github.com/bigdatagenomics/adam/pull/1605.diff","patch_url":"https://github.com/bigdatagenomics/adam/pull/1605.patch","issue_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/1605","number":1605,"state":"open","locked":false,"title":"Adding examples of how to use joins in the real world","user":{"login":"devin-petersohn","id":10732128,"avatar_url":"https://avatars0.githubusercontent.com/u/10732128?v=4","gravatar_id":"","url":"https://api.github.com/users/devin-petersohn","html_url":"https://github.com/devin-petersohn","followers_url":"https://api.github.com/users/devin-petersohn/followers","following_url":"https://api.github.com/users/devin-petersohn/following{/other_user}","gists_url":"https://api.github.com/users/devin-petersohn/gists{/gist_id}","starred_url":"https://api.github.com/users/devin-petersohn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devin-petersohn/subscriptions","organizations_url":"https://api.github.com/users/devin-petersohn/orgs","repos_url":"https://api.github.com/users/devin-petersohn/repos","events_url":"https://api.github.com/users/devin-petersohn/events{/privacy}","received_events_url":"https://api.github.com/users/devin-petersohn/received_events","type":"User","site_admin":false},"body":"Resolves #890.","created_at":"2017-07-13T05:44:56Z","updated_at":"2017-07-21T09:11:06Z","closed_at":null,"merged_at":null,"merge_commit_sha":"793261c8b709e63d4f9ab401a338c6d73af684b2","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":{"url":"https://api.github.com/repos/bigdatagenomics/adam/milestones/16","html_url":"https://github.com/bigdatagenomics/adam/milestone/16","labels_url":"https://api.github.com/repos/bigdatagenomics/adam/milestones/16/labels","id":2068939,"number":16,"title":"0.23.0","description":"","creator":{"login":"heuermh","id":35293,"avatar_url":"https://avatars2.githubusercontent.com/u/35293?v=4","gravatar_id":"","url":"https://api.github.com/users/heuermh","html_url":"https://github.com/heuermh","followers_url":"https://api.github.com/users/heuermh/followers","following_url":"https://api.github.com/users/heuermh/following{/other_user}","gists_url":"https://api.github.com/users/heuermh/gists{/gist_id}","starred_url":"https://api.github.com/users/heuermh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/heuermh/subscriptions","organizations_url":"https://api.github.com/users/heuermh/orgs","repos_url":"https://api.github.com/users/heuermh/repos","events_url":"https://api.github.com/users/heuermh/events{/privacy}","received_events_url":"https://api.github.com/users/heuermh/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":98,"state":"open","created_at":"2016-10-13T22:02:14Z","updated_at":"2017-07-21T08:18:15Z","due_on":"2017-07-08T07:00:00Z","closed_at":null},"commits_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605/commits","review_comments_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605/comments","review_comment_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments{/number}","comments_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/1605/comments","statuses_url":"https://api.github.com/repos/bigdatagenomics/adam/statuses/a49cd0409ffda9da9bb369ea4ec6972e43841587","head":{"label":"devin-petersohn:issue#890joinDocExamples","ref":"issue#890joinDocExamples","sha":"a49cd0409ffda9da9bb369ea4ec6972e43841587","user":{"login":"devin-petersohn","id":10732128,"avatar_url":"https://avatars0.githubusercontent.com/u/10732128?v=4","gravatar_id":"","url":"https://api.github.com/users/devin-petersohn","html_url":"https://github.com/devin-petersohn","followers_url":"https://api.github.com/users/devin-petersohn/followers","following_url":"https://api.github.com/users/devin-petersohn/following{/other_user}","gists_url":"https://api.github.com/users/devin-petersohn/gists{/gist_id}","starred_url":"https://api.github.com/users/devin-petersohn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devin-petersohn/subscriptions","organizations_url":"https://api.github.com/users/devin-petersohn/orgs","repos_url":"https://api.github.com/users/devin-petersohn/repos","events_url":"https://api.github.com/users/devin-petersohn/events{/privacy}","received_events_url":"https://api.github.com/users/devin-petersohn/received_events","type":"User","site_admin":false},"repo":{"id":67724277,"name":"adam","full_name":"devin-petersohn/adam","owner":{"login":"devin-petersohn","id":10732128,"avatar_url":"https://avatars0.githubusercontent.com/u/10732128?v=4","gravatar_id":"","url":"https://api.github.com/users/devin-petersohn","html_url":"https://github.com/devin-petersohn","followers_url":"https://api.github.com/users/devin-petersohn/followers","following_url":"https://api.github.com/users/devin-petersohn/following{/other_user}","gists_url":"https://api.github.com/users/devin-petersohn/gists{/gist_id}","starred_url":"https://api.github.com/users/devin-petersohn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devin-petersohn/subscriptions","organizations_url":"https://api.github.com/users/devin-petersohn/orgs","repos_url":"https://api.github.com/users/devin-petersohn/repos","events_url":"https://api.github.com/users/devin-petersohn/events{/privacy}","received_events_url":"https://api.github.com/users/devin-petersohn/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/devin-petersohn/adam","description":"ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark and Parquet. Apache 2 licensed.","fork":true,"url":"https://api.github.com/repos/devin-petersohn/adam","forks_url":"https://api.github.com/repos/devin-petersohn/adam/forks","keys_url":"https://api.github.com/repos/devin-petersohn/adam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/devin-petersohn/adam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/devin-petersohn/adam/teams","hooks_url":"https://api.github.com/repos/devin-petersohn/adam/hooks","issue_events_url":"https://api.github.com/repos/devin-petersohn/adam/issues/events{/number}","events_url":"https://api.github.com/repos/devin-petersohn/adam/events","assignees_url":"https://api.github.com/repos/devin-petersohn/adam/assignees{/user}","branches_url":"https://api.github.com/repos/devin-petersohn/adam/branches{/branch}","tags_url":"https://api.github.com/repos/devin-petersohn/adam/tags","blobs_url":"https://api.github.com/repos/devin-petersohn/adam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/devin-petersohn/adam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/devin-petersohn/adam/git/refs{/sha}","trees_url":"https://api.github.com/repos/devin-petersohn/adam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/devin-petersohn/adam/statuses/{sha}","languages_url":"https://api.github.com/repos/devin-petersohn/adam/languages","stargazers_url":"https://api.github.com/repos/devin-petersohn/adam/stargazers","contributors_url":"https://api.github.com/repos/devin-petersohn/adam/contributors","subscribers_url":"https://api.github.com/repos/devin-petersohn/adam/subscribers","subscription_url":"https://api.github.com/repos/devin-petersohn/adam/subscription","commits_url":"https://api.github.com/repos/devin-petersohn/adam/commits{/sha}","git_commits_url":"https://api.github.com/repos/devin-petersohn/adam/git/commits{/sha}","comments_url":"https://api.github.com/repos/devin-petersohn/adam/comments{/number}","issue_comment_url":"https://api.github.com/repos/devin-petersohn/adam/issues/comments{/number}","contents_url":"https://api.github.com/repos/devin-petersohn/adam/contents/{+path}","compare_url":"https://api.github.com/repos/devin-petersohn/adam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/devin-petersohn/adam/merges","archive_url":"https://api.github.com/repos/devin-petersohn/adam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/devin-petersohn/adam/downloads","issues_url":"https://api.github.com/repos/devin-petersohn/adam/issues{/number}","pulls_url":"https://api.github.com/repos/devin-petersohn/adam/pulls{/number}","milestones_url":"https://api.github.com/repos/devin-petersohn/adam/milestones{/number}","notifications_url":"https://api.github.com/repos/devin-petersohn/adam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/devin-petersohn/adam/labels{/name}","releases_url":"https://api.github.com/repos/devin-petersohn/adam/releases{/id}","deployments_url":"https://api.github.com/repos/devin-petersohn/adam/deployments","created_at":"2016-09-08T17:16:12Z","updated_at":"2016-09-08T17:16:15Z","pushed_at":"2017-07-19T16:45:31Z","git_url":"git://github.com/devin-petersohn/adam.git","ssh_url":"git@github.com:devin-petersohn/adam.git","clone_url":"https://github.com/devin-petersohn/adam.git","svn_url":"https://github.com/devin-petersohn/adam","homepage":"http://bdgenomics.org/","size":7745,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"base":{"label":"bigdatagenomics:master","ref":"master","sha":"607cd50be5d5806fd3c79352d3a0bbc866d469b4","user":{"login":"bigdatagenomics","id":5839438,"avatar_url":"https://avatars3.githubusercontent.com/u/5839438?v=4","gravatar_id":"","url":"https://api.github.com/users/bigdatagenomics","html_url":"https://github.com/bigdatagenomics","followers_url":"https://api.github.com/users/bigdatagenomics/followers","following_url":"https://api.github.com/users/bigdatagenomics/following{/other_user}","gists_url":"https://api.github.com/users/bigdatagenomics/gists{/gist_id}","starred_url":"https://api.github.com/users/bigdatagenomics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bigdatagenomics/subscriptions","organizations_url":"https://api.github.com/users/bigdatagenomics/orgs","repos_url":"https://api.github.com/users/bigdatagenomics/repos","events_url":"https://api.github.com/users/bigdatagenomics/events{/privacy}","received_events_url":"https://api.github.com/users/bigdatagenomics/received_events","type":"Organization","site_admin":false},"repo":{"id":14541530,"name":"adam","full_name":"bigdatagenomics/adam","owner":{"login":"bigdatagenomics","id":5839438,"avatar_url":"https://avatars3.githubusercontent.com/u/5839438?v=4","gravatar_id":"","url":"https://api.github.com/users/bigdatagenomics","html_url":"https://github.com/bigdatagenomics","followers_url":"https://api.github.com/users/bigdatagenomics/followers","following_url":"https://api.github.com/users/bigdatagenomics/following{/other_user}","gists_url":"https://api.github.com/users/bigdatagenomics/gists{/gist_id}","starred_url":"https://api.github.com/users/bigdatagenomics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bigdatagenomics/subscriptions","organizations_url":"https://api.github.com/users/bigdatagenomics/orgs","repos_url":"https://api.github.com/users/bigdatagenomics/repos","events_url":"https://api.github.com/users/bigdatagenomics/events{/privacy}","received_events_url":"https://api.github.com/users/bigdatagenomics/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/bigdatagenomics/adam","description":"ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark and Parquet. Apache 2 licensed.","fork":false,"url":"https://api.github.com/repos/bigdatagenomics/adam","forks_url":"https://api.github.com/repos/bigdatagenomics/adam/forks","keys_url":"https://api.github.com/repos/bigdatagenomics/adam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bigdatagenomics/adam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bigdatagenomics/adam/teams","hooks_url":"https://api.github.com/repos/bigdatagenomics/adam/hooks","issue_events_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/events{/number}","events_url":"https://api.github.com/repos/bigdatagenomics/adam/events","assignees_url":"https://api.github.com/repos/bigdatagenomics/adam/assignees{/user}","branches_url":"https://api.github.com/repos/bigdatagenomics/adam/branches{/branch}","tags_url":"https://api.github.com/repos/bigdatagenomics/adam/tags","blobs_url":"https://api.github.com/repos/bigdatagenomics/adam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bigdatagenomics/adam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bigdatagenomics/adam/git/refs{/sha}","trees_url":"https://api.github.com/repos/bigdatagenomics/adam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bigdatagenomics/adam/statuses/{sha}","languages_url":"https://api.github.com/repos/bigdatagenomics/adam/languages","stargazers_url":"https://api.github.com/repos/bigdatagenomics/adam/stargazers","contributors_url":"https://api.github.com/repos/bigdatagenomics/adam/contributors","subscribers_url":"https://api.github.com/repos/bigdatagenomics/adam/subscribers","subscription_url":"https://api.github.com/repos/bigdatagenomics/adam/subscription","commits_url":"https://api.github.com/repos/bigdatagenomics/adam/commits{/sha}","git_commits_url":"https://api.github.com/repos/bigdatagenomics/adam/git/commits{/sha}","comments_url":"https://api.github.com/repos/bigdatagenomics/adam/comments{/number}","issue_comment_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/comments{/number}","contents_url":"https://api.github.com/repos/bigdatagenomics/adam/contents/{+path}","compare_url":"https://api.github.com/repos/bigdatagenomics/adam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bigdatagenomics/adam/merges","archive_url":"https://api.github.com/repos/bigdatagenomics/adam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bigdatagenomics/adam/downloads","issues_url":"https://api.github.com/repos/bigdatagenomics/adam/issues{/number}","pulls_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls{/number}","milestones_url":"https://api.github.com/repos/bigdatagenomics/adam/milestones{/number}","notifications_url":"https://api.github.com/repos/bigdatagenomics/adam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bigdatagenomics/adam/labels{/name}","releases_url":"https://api.github.com/repos/bigdatagenomics/adam/releases{/id}","deployments_url":"https://api.github.com/repos/bigdatagenomics/adam/deployments","created_at":"2013-11-19T23:47:57Z","updated_at":"2017-07-21T02:27:05Z","pushed_at":"2017-07-21T08:21:03Z","git_url":"git://github.com/bigdatagenomics/adam.git","ssh_url":"git@github.com:bigdatagenomics/adam.git","clone_url":"https://github.com/bigdatagenomics/adam.git","svn_url":"https://github.com/bigdatagenomics/adam","homepage":"http://bdgenomics.org/","size":7886,"stargazers_count":639,"watchers_count":639,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":234,"mirror_url":null,"open_issues_count":70,"forks":234,"open_issues":70,"watchers":639,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605"},"html":{"href":"https://github.com/bigdatagenomics/adam/pull/1605"},"issue":{"href":"https://api.github.com/repos/bigdatagenomics/adam/issues/1605"},"comments":{"href":"https://api.github.com/repos/bigdatagenomics/adam/issues/1605/comments"},"review_comments":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605/comments"},"review_comment":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/1605/commits"},"statuses":{"href":"https://api.github.com/repos/bigdatagenomics/adam/statuses/a49cd0409ffda9da9bb369ea4ec6972e43841587"}}}}
{"actor":{"display_login":"heuermh"}}