{"action":"created","issue":{"url":"https://api.github.com/repos/apache/beam/issues/1868","repository_url":"https://api.github.com/repos/apache/beam","labels_url":"https://api.github.com/repos/apache/beam/issues/1868/labels{/name}","comments_url":"https://api.github.com/repos/apache/beam/issues/1868/comments","events_url":"https://api.github.com/repos/apache/beam/issues/1868/events","html_url":"https://github.com/apache/beam/pull/1868","id":203888492,"number":1868,"title":"[BEAM-797] A PipelineVisitor that creates a Spark-native pipeline.","user":{"login":"aviemzur","id":5708714,"avatar_url":"https://avatars.githubusercontent.com/u/5708714?v=3","gravatar_id":"","url":"https://api.github.com/users/aviemzur","html_url":"https://github.com/aviemzur","followers_url":"https://api.github.com/users/aviemzur/followers","following_url":"https://api.github.com/users/aviemzur/following{/other_user}","gists_url":"https://api.github.com/users/aviemzur/gists{/gist_id}","starred_url":"https://api.github.com/users/aviemzur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aviemzur/subscriptions","organizations_url":"https://api.github.com/users/aviemzur/orgs","repos_url":"https://api.github.com/users/aviemzur/repos","events_url":"https://api.github.com/users/aviemzur/events{/privacy}","received_events_url":"https://api.github.com/users/aviemzur/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2017-01-29T19:07:13Z","updated_at":"2017-02-10T13:22:44Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/beam/pulls/1868","html_url":"https://github.com/apache/beam/pull/1868","diff_url":"https://github.com/apache/beam/pull/1868.diff","patch_url":"https://github.com/apache/beam/pull/1868.patch"},"body":"Be sure to do all of the following to help us incorporate your contribution\r\nquickly and easily:\r\n\r\n - [x] Make sure the PR title is formatted like:\r\n   `[BEAM-<Jira issue #>] Description of pull request`\r\n - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable\r\n       Travis-CI on your fork and ensure the whole test matrix passes).\r\n - [x] Replace `<Jira issue #>` in the title with the actual Jira issue\r\n       number, if there is one.\r\n - [ ] If this contribution is large, please file an Apache\r\n       [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).\r\n\r\n---\r\n"},"comment":{"url":"https://api.github.com/repos/apache/beam/issues/comments/278940703","html_url":"https://github.com/apache/beam/pull/1868#issuecomment-278940703","issue_url":"https://api.github.com/repos/apache/beam/issues/1868","id":278940703,"user":{"login":"aviemzur","id":5708714,"avatar_url":"https://avatars.githubusercontent.com/u/5708714?v=3","gravatar_id":"","url":"https://api.github.com/users/aviemzur","html_url":"https://github.com/aviemzur","followers_url":"https://api.github.com/users/aviemzur/followers","following_url":"https://api.github.com/users/aviemzur/following{/other_user}","gists_url":"https://api.github.com/users/aviemzur/gists{/gist_id}","starred_url":"https://api.github.com/users/aviemzur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aviemzur/subscriptions","organizations_url":"https://api.github.com/users/aviemzur/orgs","repos_url":"https://api.github.com/users/aviemzur/repos","events_url":"https://api.github.com/users/aviemzur/events{/privacy}","received_events_url":"https://api.github.com/users/aviemzur/received_events","type":"User","site_admin":false},"created_at":"2017-02-10T13:22:44Z","updated_at":"2017-02-10T13:22:44Z","body":"Added support for unbounded pipelines and tests and fixed what we talked about regarding composites (`TextIO.Write` in the previous comments).\r\n\r\nWhat the output looks like right now:\r\n\r\nBounded pipeline:\r\n```java\r\nsparkContext.parallelize(Arrays.asList(...))\r\n.mapPartitions(new org.apache.beam.runners.spark.examples.WordCount$ExtractWordsFn())\r\n.mapPartitions(new org.apache.beam.sdk.transforms.Count$PerElement$1())\r\n.<combinePerKey>\r\n.mapPartitions(new org.apache.beam.runners.spark.examples.WordCount$FormatAsTextFn())\r\n.<org.apache.beam.sdk.io.TextIO$Write$Bound>\r\n```\r\n\r\nUnbounded pipeline:\r\n```java\r\nKafkaUtils.createDirectStream(...)\r\n.<window>\r\n.mapPartitions(new org.apache.beam.runners.spark.SparkNativePipelineVisitorTest$FormatKVFn())\r\n.mapPartitions(new org.apache.beam.sdk.transforms.Distinct$2())\r\n.<combinePerKey>\r\n.mapPartitions(new org.apache.beam.sdk.transforms.Keys$1())\r\n.mapPartitions(new org.apache.beam.sdk.transforms.WithKeys$2())\r\n.<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write>\r\n```\r\n\r\nStill left to do:\r\n* Return native translations for every translation still with a `// TODO: return meaningful Spark native operation` in `TransformTranslator` and `StreamingTransformTranslator`.\r\n* Decide if the native operation strings should be returned from `TransformEvaluator`'s `toString()` method (current implementation) or add a more binding method `toNativeString()` to its interface to make sure that nothing is forgotten in future implementations of `TransformEvaluator` resulting in parts of the debug pipeline output looking like `.TransformTranslator$123@abc12`"}}
{"actor":{"display_login":"aviemzur"}}