{"action":"created","issue":{"url":"https://api.github.com/repos/hortonworks-spark/shc/issues/132","repository_url":"https://api.github.com/repos/hortonworks-spark/shc","labels_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/132/labels{/name}","comments_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/132/comments","events_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/132/events","html_url":"https://github.com/hortonworks-spark/shc/issues/132","id":223692648,"number":132,"title":"Caused by: java.lang.Exception: unsupported data type ARRAY","user":{"login":"mavencode01","id":542912,"avatar_url":"https://avatars3.githubusercontent.com/u/542912?v=3","gravatar_id":"","url":"https://api.github.com/users/mavencode01","html_url":"https://github.com/mavencode01","followers_url":"https://api.github.com/users/mavencode01/followers","following_url":"https://api.github.com/users/mavencode01/following{/other_user}","gists_url":"https://api.github.com/users/mavencode01/gists{/gist_id}","starred_url":"https://api.github.com/users/mavencode01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mavencode01/subscriptions","organizations_url":"https://api.github.com/users/mavencode01/orgs","repos_url":"https://api.github.com/users/mavencode01/repos","events_url":"https://api.github.com/users/mavencode01/events{/privacy}","received_events_url":"https://api.github.com/users/mavencode01/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-04-24T02:58:35Z","updated_at":"2017-04-24T20:43:17Z","closed_at":null,"body":"I have been stuck trying to figure if am doing something wrong but basically, I'm trying to use avro to writes data into hbase using your library but it's given me the error below:\r\n\r\nHere is what my `AVRO` schema looks like:\r\n```\r\nplayer_schema = \"\"\"\r\n                    {\r\n                        \"type\":\"array\",\r\n                        \"items\": {\r\n                            \"name\":\"activities\",\r\n                            \"type\":\"record\",\r\n                            \"fields\":[\r\n                                    {\"name\":\"start\", \"type\":\"double\"},\r\n                                    {\"name\":\"y\", \"type\":\"double\"},\r\n                                    {\"name\":\"width\", \"type\":\"double\"}\r\n                                ]\r\n                            }\r\n                    }\r\n                 \"\"\"     \r\n\r\nopponent_schema = \"\"\"\r\n                {\r\n                    \"type\":\"array\",\r\n                    \"items\": {\r\n                        \"name\":\"opponent\",\r\n                        \"type\":\"record\",\r\n                        \"fields\":[\r\n                                {\"name\":\"x\", \"type\":\"double\"},\r\n                                {\"name\":\"y\", \"type\":\"double\"}\r\n                            ]\r\n                        }\r\n                }                    \r\n         \"\"\"       \r\n\r\n```\r\nFinally,\r\n\r\n```\r\n\r\ncatalog = \"\"\"{\r\n                \"table\":{\"namespace\":\"default\", \"name\":\"gm_drake\"},\r\n                \"rowkey\":\"id\",\r\n                \"columns\":{                \r\n                    \"id\":{\"cf\":\"rowkey\", \"col\":\"id\", \"type\":\"string\"},\r\n                    \"feature\":{\"cf\":\"data\", \"col\":\"feature\", \"type\":\"string\"},\r\n                    \"player\":{\"cf\":\"data\", \"col\":\"player\", \"avro\":\"player_schema\"},\r\n                    \"opponent\":{\"cf\":\"data\", \"col\":\"opponent\", \"avro\":\"opponent_schema\"}\r\n                }\r\n            }\"\"\"\r\n\r\ndf_drake_json.write.options(player_schema =player_schema, opponent_schema = opponent_schema, catalog = catalog).format(\"org.apache.spark.sql.execution.datasources.hbase\").save()\r\n\r\n```\r\n\r\nAnd here is my sample dataset\r\n```\r\n{\"feature\":\"ROLLOVER_ATTACK\",\"player\":[{\"start\":-7.281526191298157,\"y\":1.0,\"width\":0.15025269761092996}],\"opponent\":[{\"y\":1.7573350289423742E-4,\"x\":-7.281526191298157},{\"y\":1.7580108506663706E-4,\"x\":-7.281375788197446}]}\r\n```\r\n\r\nGetting this error\r\n*java.lang.Exception: unsupported data type ARRAY* \r\n```\r\nCaused by: java.lang.Exception: unsupported data type ARRAY\r\n        at org.apache.spark.sql.execution.datasources.hbase.AvroSedes$.serialize(SchemaConverters.scala:400)\r\n        at org.apache.spark.sql.execution.datasources.hbase.Utils$.toBytes(Utils.scala:74)\r\n        at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$org$apache$spark$sql$execution$datasources$hbase$HBaseRelation$$convertToPut$1$2.apply(HBaseRelation.scala:155)\r\n        at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$org$apache$spark$sql$execution$datasources$hbase$HBaseRelation$$convertToPut$1$2.apply(HBaseRelation.scala:154)\r\n        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\r\n        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)\r\n        at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.org$apache$spark$sql$execution$datasources$hbase$HBaseRelation$$convertToPut$1(HBaseRelation.scala:154)\r\n        at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$insert$1.apply(HBaseRelation.scala:161)\r\n        at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$insert$1.apply(HBaseRelation.scala:161)\r\n        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)\r\n        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1112)\r\n        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1111)\r\n        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1111)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277)\r\n        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119)\r\n        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1091)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:89)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        ... 1 more\r\n```\r\n\r\nHow can I fix this exception ?"},"comment":{"url":"https://api.github.com/repos/hortonworks-spark/shc/issues/comments/296816389","html_url":"https://github.com/hortonworks-spark/shc/issues/132#issuecomment-296816389","issue_url":"https://api.github.com/repos/hortonworks-spark/shc/issues/132","id":296816389,"user":{"login":"mavencode01","id":542912,"avatar_url":"https://avatars3.githubusercontent.com/u/542912?v=3","gravatar_id":"","url":"https://api.github.com/users/mavencode01","html_url":"https://github.com/mavencode01","followers_url":"https://api.github.com/users/mavencode01/followers","following_url":"https://api.github.com/users/mavencode01/following{/other_user}","gists_url":"https://api.github.com/users/mavencode01/gists{/gist_id}","starred_url":"https://api.github.com/users/mavencode01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mavencode01/subscriptions","organizations_url":"https://api.github.com/users/mavencode01/orgs","repos_url":"https://api.github.com/users/mavencode01/repos","events_url":"https://api.github.com/users/mavencode01/events{/privacy}","received_events_url":"https://api.github.com/users/mavencode01/received_events","type":"User","site_admin":false},"created_at":"2017-04-24T20:43:16Z","updated_at":"2017-04-24T20:43:16Z","body":"Cool...good to know - thank you once again @weiqingy "}}
{"actor":{"display_login":"mavencode01"}}