{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16605","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16605/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16605/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16605/events","html_url":"https://github.com/apache/spark/pull/16605","id":201033023,"number":16605,"title":"[SPARK-18884][SQL] Support Array[_] in ScalaUDF","user":{"login":"maropu","id":692303,"avatar_url":"https://avatars.githubusercontent.com/u/692303?v=3","gravatar_id":"","url":"https://api.github.com/users/maropu","html_url":"https://github.com/maropu","followers_url":"https://api.github.com/users/maropu/followers","following_url":"https://api.github.com/users/maropu/following{/other_user}","gists_url":"https://api.github.com/users/maropu/gists{/gist_id}","starred_url":"https://api.github.com/users/maropu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maropu/subscriptions","organizations_url":"https://api.github.com/users/maropu/orgs","repos_url":"https://api.github.com/users/maropu/repos","events_url":"https://api.github.com/users/maropu/events{/privacy}","received_events_url":"https://api.github.com/users/maropu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-01-16T14:30:10Z","updated_at":"2017-01-16T14:49:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16605","html_url":"https://github.com/apache/spark/pull/16605","diff_url":"https://github.com/apache/spark/pull/16605.diff","patch_url":"https://github.com/apache/spark/pull/16605.patch"},"body":"## What changes were proposed in this pull request?\r\nThis pr is to support `Array[_]` in `ScalaUDF`.\r\nCurrently, a query below throws an exception if we use the type in `ScalaUDF`;\r\n\r\n```\r\nscala> import org.apache.spark.sql.execution.debug._\r\nscala> Seq((0, 1)).toDF(\"a\", \"b\").select(array($\"a\", $\"b\").as(\"ar\")).write.mode(\"overwrite\").parquet(\"/Users/maropu/Desktop/data/\")\r\nscala> val df = spark.read.load(\"/Users/maropu/Desktop/data/\")\r\nscala> val df = Seq((0, 1)).toDF(\"a\", \"b\").select(array($\"a\", $\"b\").as(\"ar\"))\r\nscala> val testArrayUdf = udf { (ar: Array[Int]) => ar.sum }\r\nscala> df.select(testArrayUdf($\"ar\")).show\r\n\r\nCaused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [I\r\n  at $anonfun$1.apply(<console>:23)\r\n  at org.apache.spark.sql.catalyst.expressions.ScalaUDF$$anonfun$2.apply(ScalaUDF.scala:89)\r\n  at org.apache.spark.sql.catalyst.expressions.ScalaUDF$$anonfun$2.apply(ScalaUDF.scala:88)\r\n  at org.apache.spark.sql.catalyst.expressions.ScalaUDF.eval(ScalaUDF.scala:1069)\r\n  ... 99 more\r\n```\r\nOn the other hand, another query below is passed;\r\n```\r\nscala> val testSeqUdf = udf { (ar: Seq[Int]) => ar.sum }\r\nscala> df.select(testSeqUdf($\"ar\")).show\r\n+-------+\r\n|UDF(ar)|\r\n+-------+\r\n|      1|\r\n+-------+\r\n```\r\n\r\nThe existing implementation checks argument types (`DataType`)　by reflection (`ScalaReflection.schemaFor`) in `sql.functions.udf`, and then creates type converters (`CatalystTypeConverters`) from the types.　`Seq[_]` and `Array[_]` are represented as `ArrayType` in `DataType` and both types are handled by using `ArrayConverter`.　So, since it cannot tell a difference between both types in `DataType`. This pr modified code to create type converters directly from `TypeTag` in `sql.functions.udf` (and also in `UDFRegistration.register`).\r\n\r\n## How was this patch tested?\r\nAdded tests in `DataFrameSuite`."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/272881766","html_url":"https://github.com/apache/spark/pull/16605#issuecomment-272881766","issue_url":"https://api.github.com/repos/apache/spark/issues/16605","id":272881766,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2017-01-16T14:49:19Z","updated_at":"2017-01-16T14:49:19Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}