{"action":"closed","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/2311","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/2311/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/2311/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/2311/events","html_url":"https://github.com/dmlc/xgboost/issues/2311","id":229591991,"number":2311,"title":"[jvm-package]  jvm4spark: Missing an output location for shuffle 15","user":{"login":"cnglen","id":4279208,"avatar_url":"https://avatars2.githubusercontent.com/u/4279208?v=4","gravatar_id":"","url":"https://api.github.com/users/cnglen","html_url":"https://github.com/cnglen","followers_url":"https://api.github.com/users/cnglen/followers","following_url":"https://api.github.com/users/cnglen/following{/other_user}","gists_url":"https://api.github.com/users/cnglen/gists{/gist_id}","starred_url":"https://api.github.com/users/cnglen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cnglen/subscriptions","organizations_url":"https://api.github.com/users/cnglen/orgs","repos_url":"https://api.github.com/users/cnglen/repos","events_url":"https://api.github.com/users/cnglen/events{/privacy}","received_events_url":"https://api.github.com/users/cnglen/received_events","type":"User","site_admin":false},"labels":[{"id":617466546,"url":"https://api.github.com/repos/dmlc/xgboost/labels/Java-Scala","name":"Java-Scala","color":"c2e0c6","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-05-18T08:26:40Z","updated_at":"2017-11-18T20:02:00Z","closed_at":"2017-11-18T20:02:00Z","author_association":"NONE","body":"Any suggestions?  \r\n\r\nFetchFailed(null, shuffleId=15, mapId=-1, reduceId=147, message=\r\norg.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 15\r\n\tat org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)\r\n\tat org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)\r\n\tat scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)\r\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\r\n\tat scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)\r\n\tat scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)\r\n\tat org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)\r\n\tat org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)\r\n\tat org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)\r\n\tat org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:169)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\n)\r\n17/05/18 15:30:05 INFO TaskSetManager: Task 147.1 in stage 96.0 (TID 17692) failed, but another instance of the task has already succeeded, so not re-queuing the task to be re-executed.\r\n17/05/18 15:30:05 INFO DAGScheduler: Marking ShuffleMapStage 96 (count at UserSkuPrediction.scala:433) as failed due to a fetch failure from ShuffleMapStage 95 (rdd at XGBoostModel.scala:222)\r\n17/05/18 15:30:05 INFO DAGScheduler: ShuffleMapStage 96 (count at UserSkuPrediction.scala:433) failed in 15.649 s due to org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 15\r\n\tat org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)\r\n\tat org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)\r\n\tat scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)\r\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\r\n\tat scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)\r\n\tat scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)\r\n\tat org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)\r\n\tat org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)\r\n\tat org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)\r\n\tat org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:169)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\n17/05/18 15:30:05 INFO DAGScheduler: Job 24 failed: count at UserSkuPrediction.scala:433, took 15.721147 s\r\nException in thread \"main\" org.apache.spark.SparkException: Job aborted due to stage failure: Fetch failure will not retry stage due to testing config\r\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)\r\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1259)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\r\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1944)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1958)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:935)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\r\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:362)\r\n\tat org.apache.spark.rdd.RDD.collect(RDD.scala:934)\r\n\tat org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:275)\r\n\tat org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2371)\r\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)\r\n\tat org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2765)\r\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2370)\r\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2377)\r\n\tat org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2405)\r\n\tat org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2404)\r\n\tat org.apache.spark.sql.Dataset.withCallback(Dataset.scala:2778)\r\n\tat org.apache.spark.sql.Dataset.count(Dataset.scala:2404)\r\n\tat com.jd.travel.UserSkuPrediction$.main(UserSkuPrediction.scala:433)\r\n\tat com.jd.travel.UserSkuPrediction.main(UserSkuPrediction.scala)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:740)\r\n\tat org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)\r\n\tat org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)\r\n\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)\r\n\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n17/05/18 15:30:05 INFO SparkContext: Invoking stop() from shutdown hook\r\n17/05/18 15:30:05 INFO SparkUI: Stopped Spark web UI at http://172.19.165.48:8965\r\n17/05/18 15:30:06 INFO YarnClientSchedulerBackend: Interrupting monitor thread\r\n17/05/18 15:30:06 INFO YarnClientSchedulerBackend: Shutting down all executors\r\n17/05/18 15:30:06 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down\r\n17/05/18 15:30:06 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices\r\n\r\n\r\n"}}
{"actor":{"display_login":"superbobry"}}