{"action":"created","issue":{"url":"https://api.github.com/repos/apache/mahout/issues/271","repository_url":"https://api.github.com/repos/apache/mahout","labels_url":"https://api.github.com/repos/apache/mahout/issues/271/labels{/name}","comments_url":"https://api.github.com/repos/apache/mahout/issues/271/comments","events_url":"https://api.github.com/repos/apache/mahout/issues/271/events","html_url":"https://github.com/apache/mahout/pull/271","id":204793453,"number":271,"title":"[MAHOUT-1894] Add Support for Spark 2.x","user":{"login":"rawkintrevo","id":5852441,"avatar_url":"https://avatars.githubusercontent.com/u/5852441?v=3","gravatar_id":"","url":"https://api.github.com/users/rawkintrevo","html_url":"https://github.com/rawkintrevo","followers_url":"https://api.github.com/users/rawkintrevo/followers","following_url":"https://api.github.com/users/rawkintrevo/following{/other_user}","gists_url":"https://api.github.com/users/rawkintrevo/gists{/gist_id}","starred_url":"https://api.github.com/users/rawkintrevo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rawkintrevo/subscriptions","organizations_url":"https://api.github.com/users/rawkintrevo/orgs","repos_url":"https://api.github.com/users/rawkintrevo/repos","events_url":"https://api.github.com/users/rawkintrevo/events{/privacy}","received_events_url":"https://api.github.com/users/rawkintrevo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-02-02T06:33:25Z","updated_at":"2017-02-05T00:15:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/mahout/pulls/271","html_url":"https://github.com/apache/mahout/pull/271","diff_url":"https://github.com/apache/mahout/pull/271.diff","patch_url":"https://github.com/apache/mahout/pull/271.patch"},"body":"As long as we're sticking to Scala 2.10, running mahout on spark 2.x is simply a matter of \r\n\r\n`mvn clean package -Dspark.version=2.0.2`\r\nor \r\n`mvn clean package -Dspark.version=2.1.0`\r\n\r\nThe trouble comes with the shell...\r\n\r\nI checked Apache Zeppelin to see how they handle multiple spark/scala versions...\r\n[a brief preview of the descent into hell that is having a shell that handles multiple spark/scala versions](https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java)\r\n\r\nSo I took an alternate root.  I dropped the Mahout shell all together, changed the mahout bin file to load the spark shell directly, and pass a scala script that takes care of our imports.  \r\n\r\nWhen building there is a single deprecation warning regarding the sqlContext and how it is created in the spark-bindings.  \r\n\r\nI think we should add binaries for Spark 2.0 and Spark 2.1 as a matter of convenience and the Zeppelin integration.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/mahout/issues/comments/277487809","html_url":"https://github.com/apache/mahout/pull/271#issuecomment-277487809","issue_url":"https://api.github.com/repos/apache/mahout/issues/271","id":277487809,"user":{"login":"andrewpalumbo","id":7681565,"avatar_url":"https://avatars.githubusercontent.com/u/7681565?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewpalumbo","html_url":"https://github.com/andrewpalumbo","followers_url":"https://api.github.com/users/andrewpalumbo/followers","following_url":"https://api.github.com/users/andrewpalumbo/following{/other_user}","gists_url":"https://api.github.com/users/andrewpalumbo/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewpalumbo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewpalumbo/subscriptions","organizations_url":"https://api.github.com/users/andrewpalumbo/orgs","repos_url":"https://api.github.com/users/andrewpalumbo/repos","events_url":"https://api.github.com/users/andrewpalumbo/events{/privacy}","received_events_url":"https://api.github.com/users/andrewpalumbo/received_events","type":"User","site_admin":false},"created_at":"2017-02-05T00:15:28Z","updated_at":"2017-02-05T00:15:28Z","body":"hmm.. just tried to launch into `local[4]` and blew it up:\r\n\r\n```\r\nAP-RE-X16743C45L:mahout apalumbo$ MASTER=local[4] mahout spark-shell\r\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).\r\nlog4j:WARN Please initialize the log4j system properly.\r\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\r\nUsing Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties\r\nTo adjust logging level use sc.setLogLevel(\"INFO\")\r\nWelcome to\r\n      ____              __\r\n     / __/__  ___ _____/ /__\r\n    _\\ \\/ _ \\/ _ `/ __/  '_/\r\n   /___/ .__/\\_,_/_/ /_/\\_\\   version 1.6.2\r\n      /_/\r\n\r\nUsing Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_102)\r\nType in expressions to have them evaluated.\r\nType :help for more information.\r\nSpark context available as sc.\r\nSQL context available as sqlContext.\r\nLoading /Users/apalumbo/sandbox/mahout/bin/load-shell.scala...\r\nimport org.apache.mahout.math._\r\nimport org.apache.mahout.math.scalabindings._\r\nimport org.apache.mahout.math.drm._\r\nimport org.apache.mahout.math.scalabindings.RLikeOps._\r\nimport org.apache.mahout.math.drm.RLikeDrmOps._\r\nimport org.apache.mahout.sparkbindings._\r\nsdc: org.apache.mahout.sparkbindings.SparkDistributedContext = org.apache.mahout.sparkbindings.SparkDistributedContext@73e0c775\r\n\r\n                _                 _\r\n_ __ ___   __ _| |__   ___  _   _| |_\r\n '_ ` _ \\ / _` | '_ \\ / _ \\| | | | __|\r\n | | | | (_| | | | | (_) | |_| | |_\r\n_| |_| |_|\\__,_|_| |_|\\___/ \\__,_|\\__|  version 0.13.0\r\n\r\n\r\n\r\nException in thread \"main\" java.io.FileNotFoundException: spark-shell (Is a directory)\r\n\tat java.io.FileInputStream.open0(Native Method)\r\n\tat java.io.FileInputStream.open(FileInputStream.java:195)\r\n\tat java.io.FileInputStream.<init>(FileInputStream.java:138)\r\n\tat scala.reflect.io.File.inputStream(File.scala:97)\r\n\tat scala.reflect.io.File.inputStream(File.scala:82)\r\n\tat scala.reflect.io.Streamable$Chars$class.reader(Streamable.scala:93)\r\n\tat scala.reflect.io.File.reader(File.scala:82)\r\n\tat scala.reflect.io.Streamable$Chars$class.bufferedReader(Streamable.scala:98)\r\n\tat scala.reflect.io.File.bufferedReader(File.scala:82)\r\n\tat scala.reflect.io.Streamable$Chars$class.bufferedReader(Streamable.scala:97)\r\n\tat scala.reflect.io.File.bufferedReader(File.scala:82)\r\n\tat scala.reflect.io.Streamable$Chars$class.applyReader(Streamable.scala:103)\r\n\tat scala.reflect.io.File.applyReader(File.scala:82)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$interpretAllFrom$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SparkILoop.scala:677)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$interpretAllFrom$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:677)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$interpretAllFrom$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:677)\r\n\tat org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$savingReplayStack(SparkILoop.scala:162)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$interpretAllFrom$1.apply$mcV$sp(SparkILoop.scala:676)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$interpretAllFrom$1.apply(SparkILoop.scala:676)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$interpretAllFrom$1.apply(SparkILoop.scala:676)\r\n\tat org.apache.spark.repl.SparkILoop.savingReader(SparkILoop.scala:167)\r\n\tat org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$interpretAllFrom(SparkILoop.scala:675)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$loadCommand$1.apply(SparkILoop.scala:740)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$loadCommand$1.apply(SparkILoop.scala:739)\r\n\tat org.apache.spark.repl.SparkILoop.withFile(SparkILoop.scala:733)\r\n\tat org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loadCommand(SparkILoop.scala:739)\r\n\r\n{...}\r\n\r\n```"}}
{"actor":{"display_login":"andrewpalumbo"}}