{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/2115","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/2115/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/2115/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/2115/events","html_url":"https://github.com/dmlc/xgboost/issues/2115","id":214885274,"number":2115,"title":"[jvm-packages] Persist CrossValidator model with xgboost4j-spark  error","user":{"login":"Frank111","id":9346661,"avatar_url":"https://avatars2.githubusercontent.com/u/9346661?v=3","gravatar_id":"","url":"https://api.github.com/users/Frank111","html_url":"https://github.com/Frank111","followers_url":"https://api.github.com/users/Frank111/followers","following_url":"https://api.github.com/users/Frank111/following{/other_user}","gists_url":"https://api.github.com/users/Frank111/gists{/gist_id}","starred_url":"https://api.github.com/users/Frank111/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Frank111/subscriptions","organizations_url":"https://api.github.com/users/Frank111/orgs","repos_url":"https://api.github.com/users/Frank111/repos","events_url":"https://api.github.com/users/Frank111/events{/privacy}","received_events_url":"https://api.github.com/users/Frank111/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":31,"created_at":"2017-03-17T01:39:01Z","updated_at":"2017-05-08T00:08:19Z","closed_at":null,"body":"## Environment info\r\nOperating System:\r\nredhat 6.5（with `spark-2.1.0）`\r\nCompiler:\r\n\r\nPackage: jvm, xgboost4j-spark\r\nlastest xgboost version used\r\n\r\nI want to save CrossValidator model ，but i got a error \r\n\r\n```\r\njava.lang.UnsupportedOperationException: Pipeline write will fail on this Pipeline because it contains a stage which does not implement Writable. Non-Writable stage: XGBoostEstimator_88624dc1e519 of type class ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator\r\n  at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply(Pipeline.scala:231)\r\n  at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply(Pipeline.scala:228)\r\n  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\r\n  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)\r\n  at org.apache.spark.ml.Pipeline$SharedReadWrite$.validateStages(Pipeline.scala:228)\r\n  at org.apache.spark.ml.Pipeline$PipelineWriter.<init>(Pipeline.scala:202)\r\n  at org.apache.spark.ml.Pipeline.write(Pipeline.scala:188)\r\n  at org.apache.spark.ml.util.MLWritable$class.save(ReadWrite.scala:154)\r\n  at org.apache.spark.ml.Pipeline.save(Pipeline.scala:96)\r\n  at org.apache.spark.ml.tuning.ValidatorParams$.saveImpl(ValidatorParams.scala:148)\r\n  at org.apache.spark.ml.tuning.CrossValidatorModel$CrossValidatorModelWriter.saveImpl(CrossValidator.scala:256)\r\n  at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:111)\r\n  ... 50 elided\r\n\r\n```\r\n\r\nmy code :\r\n\r\n```\r\nimport org.apache.spark.ml.Pipeline\r\nimport ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator\r\nimport org.apache.spark.ml.evaluation._\r\nimport org.apache.spark.ml.feature.{HashingTF, Tokenizer}\r\nimport org.apache.spark.ml.linalg.Vector\r\nimport org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}\r\nimport org.apache.spark.ml.feature.VectorAssembler\r\nimport org.apache.spark.ml.evaluation.BinaryClassificationEvaluator\r\nimport org.apache.spark.sql.Row\r\nimport scala.collection.mutable\r\nimport scala.util.Random\r\n\r\n// xgboost parameters\r\ndef get_param(): mutable.HashMap[String, Any] = {\r\n  val params = new mutable.HashMap[String, Any]()\r\n  params += \"eta\" -> 0.1\r\n  params += \"max_depth\" -> 4\r\n  params += \"min_child_weight\" -> 4\r\n  params += \"num_rounds\" -> 20\r\n  params += \"silent\" -> 1\r\n  params += \"objective\" -> \"binary:logistic\"\r\n  params += \"booster\" -> \"gbtree\"\r\n  params += \"gamma\" -> 0.0\r\n  params += \"colsample_bylevel\" -> 1\r\n\r\n  return params\r\n}\r\n\r\nval r = new Random(0)\r\n\r\nval training = spark.createDataFrame(\r\n  Seq.fill(10000)(r.nextInt(2)).map(i => (i, i))\r\n).toDF(\"feature\", \"label\")\r\n\r\nval test = spark.createDataFrame(\r\n  Seq.fill(10000)(r.nextInt(2)).map(i => (i, i))\r\n).toDF(\"feature\", \"label\")\r\n\r\n\r\n// create pipeline\r\nval assembler = new VectorAssembler()\r\n  .setInputCols(Array(\"feature\"))\r\n  .setOutputCol(\"features\")\r\n\r\nval xgb = new XGBoostEstimator(get_param().toMap)\r\n  .setFeaturesCol(\"features\")\r\n\r\nval pipeline = new Pipeline()\r\n  .setStages(Array(assembler, xgb))\r\n\r\n\r\n// grid\r\nval paramGrid = new ParamGridBuilder()\r\n//  .addGrid(hashingTF.numFeatures, Array(10, 100, 1000))\r\n  .addGrid(xgb.round, Array(10, 30, 50,100))\r\n .addGrid(xgb.maxDepth,  Array(5, 6, 8,10))\r\n .addGrid(xgb.minChildWeight, Array(0.5, 0.7, 1.0))\r\n  .build()\r\n// cv\r\n// val evaluator = new BinaryClassificationEvaluator().setRawPredictionCol(\"probabilities\")\r\nval evaluator = new RegressionEvaluator().setLabelCol(\"label\")\r\nval cv = new CrossValidator()\r\n  .setEstimator(pipeline)\r\n   .setEvaluator(new RegressionEvaluator()\r\n      .setLabelCol(\"label\")\r\n      .setPredictionCol(\"prediction\")\r\n      .setMetricName(\"rmse\")\r\n      )\r\n  .setEstimatorParamMaps(paramGrid)\r\n  .setNumFolds(5)\r\n\r\n\r\n// Run cross-validation, and choose the best set of parameters.\r\nval cvModel = cv.fit(training)\r\ncvModel.write.overwrite.save(\"/tmp/xgbModel\")\r\n\r\n```\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/comments/299745744","html_url":"https://github.com/dmlc/xgboost/issues/2115#issuecomment-299745744","issue_url":"https://api.github.com/repos/dmlc/xgboost/issues/2115","id":299745744,"user":{"login":"Frank111","id":9346661,"avatar_url":"https://avatars2.githubusercontent.com/u/9346661?v=3","gravatar_id":"","url":"https://api.github.com/users/Frank111","html_url":"https://github.com/Frank111","followers_url":"https://api.github.com/users/Frank111/followers","following_url":"https://api.github.com/users/Frank111/following{/other_user}","gists_url":"https://api.github.com/users/Frank111/gists{/gist_id}","starred_url":"https://api.github.com/users/Frank111/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Frank111/subscriptions","organizations_url":"https://api.github.com/users/Frank111/orgs","repos_url":"https://api.github.com/users/Frank111/repos","events_url":"https://api.github.com/users/Frank111/events{/privacy}","received_events_url":"https://api.github.com/users/Frank111/received_events","type":"User","site_admin":false},"created_at":"2017-05-08T00:08:19Z","updated_at":"2017-05-08T00:08:19Z","body":"yeah, i will test it later. thanks your replay"}}
{"actor":{"display_login":"Frank111"}}