{"action":"created","issue":{"url":"https://api.github.com/repos/InterestingLab/waterdrop/issues/30","repository_url":"https://api.github.com/repos/InterestingLab/waterdrop","labels_url":"https://api.github.com/repos/InterestingLab/waterdrop/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/InterestingLab/waterdrop/issues/30/comments","events_url":"https://api.github.com/repos/InterestingLab/waterdrop/issues/30/events","html_url":"https://github.com/InterestingLab/waterdrop/issues/30","id":253729302,"number":30,"title":"从一个udf中生成多个新的column","user":{"login":"garyelephant","id":2291859,"avatar_url":"https://avatars3.githubusercontent.com/u/2291859?v=4","gravatar_id":"","url":"https://api.github.com/users/garyelephant","html_url":"https://github.com/garyelephant","followers_url":"https://api.github.com/users/garyelephant/followers","following_url":"https://api.github.com/users/garyelephant/following{/other_user}","gists_url":"https://api.github.com/users/garyelephant/gists{/gist_id}","starred_url":"https://api.github.com/users/garyelephant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garyelephant/subscriptions","organizations_url":"https://api.github.com/users/garyelephant/orgs","repos_url":"https://api.github.com/users/garyelephant/repos","events_url":"https://api.github.com/users/garyelephant/events{/privacy}","received_events_url":"https://api.github.com/users/garyelephant/received_events","type":"User","site_admin":false},"labels":[{"id":662393851,"url":"https://api.github.com/repos/InterestingLab/waterdrop/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":{"login":"garyelephant","id":2291859,"avatar_url":"https://avatars3.githubusercontent.com/u/2291859?v=4","gravatar_id":"","url":"https://api.github.com/users/garyelephant","html_url":"https://github.com/garyelephant","followers_url":"https://api.github.com/users/garyelephant/followers","following_url":"https://api.github.com/users/garyelephant/following{/other_user}","gists_url":"https://api.github.com/users/garyelephant/gists{/gist_id}","starred_url":"https://api.github.com/users/garyelephant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garyelephant/subscriptions","organizations_url":"https://api.github.com/users/garyelephant/orgs","repos_url":"https://api.github.com/users/garyelephant/repos","events_url":"https://api.github.com/users/garyelephant/events{/privacy}","received_events_url":"https://api.github.com/users/garyelephant/received_events","type":"User","site_admin":false},"assignees":[{"login":"garyelephant","id":2291859,"avatar_url":"https://avatars3.githubusercontent.com/u/2291859?v=4","gravatar_id":"","url":"https://api.github.com/users/garyelephant","html_url":"https://github.com/garyelephant","followers_url":"https://api.github.com/users/garyelephant/followers","following_url":"https://api.github.com/users/garyelephant/following{/other_user}","gists_url":"https://api.github.com/users/garyelephant/gists{/gist_id}","starred_url":"https://api.github.com/users/garyelephant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garyelephant/subscriptions","organizations_url":"https://api.github.com/users/garyelephant/orgs","repos_url":"https://api.github.com/users/garyelephant/repos","events_url":"https://api.github.com/users/garyelephant/events{/privacy}","received_events_url":"https://api.github.com/users/garyelephant/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/InterestingLab/waterdrop/milestones/1","html_url":"https://github.com/InterestingLab/waterdrop/milestone/1","labels_url":"https://api.github.com/repos/InterestingLab/waterdrop/milestones/1/labels","id":2725165,"number":1,"title":"M1","description":"","creator":{"login":"garyelephant","id":2291859,"avatar_url":"https://avatars3.githubusercontent.com/u/2291859?v=4","gravatar_id":"","url":"https://api.github.com/users/garyelephant","html_url":"https://github.com/garyelephant","followers_url":"https://api.github.com/users/garyelephant/followers","following_url":"https://api.github.com/users/garyelephant/following{/other_user}","gists_url":"https://api.github.com/users/garyelephant/gists{/gist_id}","starred_url":"https://api.github.com/users/garyelephant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garyelephant/subscriptions","organizations_url":"https://api.github.com/users/garyelephant/orgs","repos_url":"https://api.github.com/users/garyelephant/repos","events_url":"https://api.github.com/users/garyelephant/events{/privacy}","received_events_url":"https://api.github.com/users/garyelephant/received_events","type":"User","site_admin":false},"open_issues":12,"closed_issues":3,"state":"open","created_at":"2017-08-27T11:14:18Z","updated_at":"2017-08-31T04:50:22Z","due_on":"2017-09-17T07:00:00Z","closed_at":null},"comments":3,"created_at":"2017-08-29T16:36:47Z","updated_at":"2017-09-02T06:30:27Z","closed_at":null,"author_association":"OWNER","body":"应用场景grok, split等filter"},"comment":{"url":"https://api.github.com/repos/InterestingLab/waterdrop/issues/comments/326725844","html_url":"https://github.com/InterestingLab/waterdrop/issues/30#issuecomment-326725844","issue_url":"https://api.github.com/repos/InterestingLab/waterdrop/issues/30","id":326725844,"user":{"login":"RickyHuo","id":18550072,"avatar_url":"https://avatars1.githubusercontent.com/u/18550072?v=4","gravatar_id":"","url":"https://api.github.com/users/RickyHuo","html_url":"https://github.com/RickyHuo","followers_url":"https://api.github.com/users/RickyHuo/followers","following_url":"https://api.github.com/users/RickyHuo/following{/other_user}","gists_url":"https://api.github.com/users/RickyHuo/gists{/gist_id}","starred_url":"https://api.github.com/users/RickyHuo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RickyHuo/subscriptions","organizations_url":"https://api.github.com/users/RickyHuo/orgs","repos_url":"https://api.github.com/users/RickyHuo/repos","events_url":"https://api.github.com/users/RickyHuo/events{/privacy}","received_events_url":"https://api.github.com/users/RickyHuo/received_events","type":"User","site_admin":false},"created_at":"2017-09-02T06:30:27Z","updated_at":"2017-09-02T06:30:27Z","author_association":"OWNER","body":"```\r\ndef process(df : DataFrame) : DataFrame = {\r\n\r\n  import org.apache.spark.sql.types._\r\n\r\n  val sqlContext = this.sqlContext\r\n\r\n  import sqlContext.implicits._\r\n\r\n  val rows = df.rdd.map { r =>\r\n    Row.fromSeq(r.toSeq ++ udfFunc(r.getAs[String](\"raw_message\")))\r\n  }\r\n\r\n  val schema = StructType(df.schema.fields ++ structField())\r\n  sqlContext.createDataFrame(rows, schema)\r\n}\r\n\r\ndef udfFunc(str : String) : Seq[Any] = {\r\n\r\n  Seq(str.trim, str.length)\r\n}\r\n\r\ndef structField() : Array[StructField] = {\r\n  import org.apache.spark.sql.types._\r\n\r\n  Array(StructField(\"prefix\", StringType),\r\n    StructField(\"prelen\", IntegerType))\r\n}\r\n```\r\n\r\n\r\n以上可以实现dataframe新增多个字段，但处理性能需要检测"}}
{"actor":{"display_login":"RickyHuo"}}