title: Why is my Spark driver program so slow?
body: <p>My problem: I have a model engine that takes a list of parameter configuration, and evaluates a double value that corresponds to the metric associated to that configuration. I have six parameters, and each of them can vary according to a list. I want to find by brute force the best parameter configuration considering the combination that will produce the higher value for the output metric. Since I'm learning Spark, I realized that with the cartesian product operation I can easily generate the combinations, and split the RDD to be processed in parallel. So, I came up with this driver program:</p>
  
  <pre><code>public static void main(String[] args) {
  
      String scriptName = "model.mry";
  
      String scriptStr = null;
  
      try {
          scriptStr = new String(Files.readAllBytes(Paths.get(scriptName)));
      } catch (IOException ex) {
          Logger.getLogger(BruteForceDriver.class.getName()).log(Level.SEVERE, null, ex);
          System.exit(1);
      }
  
      final String script = scriptStr;
  
      SparkConf conf = new SparkConf()
              .setAppName("wordCount")
              .setSparkHome("/home/danilo/bin/spark-2.2.0-bin-hadoop2.7")
              .setJars(new String[]{"/home/danilo/NetBeansProjects/SparkHello1/target/SparkHello1-1.0.jar",
          "/home/danilo/.m2/repository/org/modcs/mercury/4.7/mercury-4.7.jar"})
              .setMaster("spark://danilo-desktop:7077");
  
      String baseDir = "/home/danilo/NetBeansProjects/SimulationOptimization/workspace/";
  
      JavaSparkContext sc = new JavaSparkContext(conf);
  
      final int NUM_SERVICES = 6;
      final int QTD = 3;
  
      JavaRDD&lt;Service&gt;[] providers = new JavaRDD[NUM_SERVICES];
  
      for (int i = 1; i &lt;= NUM_SERVICES; i++) {
          providers[i - 1] = sc.textFile(baseDir + "provider"
                  + i
                  + ".mat")
                  .filter((t1) -&gt; !t1.contains("#") &amp;&amp; !t1.trim().isEmpty())
                  .map(Service.createParser("" + i))
                  .zipWithIndex().filter((t1) -&gt; {
                      return t1._2 &lt; QTD;
                  }).keys();
      }
  
      JavaPairRDD c = null;
      JavaRDD&lt;Service&gt; p = providers[0];
  
      for (int i = 1; i &lt; NUM_SERVICES; i++) {
          if (c == null) {
              c = p.cartesian(providers[i]);
          } else {
              c = c.cartesian(providers[i]);
          }
      }
  
      JavaRDD&lt;List&lt;Service&gt;&gt; cartesian = c.map(new FlattenTuple&lt;&gt;());
  
      final Broadcast&lt;ModelEvaluator&gt; model = sc.broadcast(new ModelEvaluator(script));
  
      JavaPairRDD&lt;Double, List&lt;Service&gt;&gt; results = cartesian.mapToPair(
              (t) -&gt; {
                  try {
                      double val = model.value().evaluateModel(t);
                      System.out.println(val);
                      return new Tuple2&lt;&gt;(val, t);
                  } catch (Exception ex) {
                      return null;
                  }
              }
      );
  
      results.sortByKey().collect().forEach((t) -&gt; {
          System.out.println(t._1 + ", " + t._2);
      });
  
      sc.close();
  }
  </code></pre>
  
  <p>The "QTD" variable allows me to control the size the interval which each parameter will vary. For QTD = 3, I'll have 3^6 = 729 combinations. The problem is that it is taking so long to compute all those combinations. I wrote a implementations using only normal Java threads, and the runtime is about 40 seconds. Using my Spark driver program, the runtime more than 6 minutes. Why is my Spark program so slow compared to the plain Java multi-thread program?</p>
  
  <p><strong>Edit</strong>: </p>
  
  <p>I put:</p>
  
  <p>results = results.cache();</p>
  
  <p>before sorting the results and now the runtime is 2.5 minutes.</p>
  
  <p><strong>Edit 2</strong>: </p>
  
  <p>I created a RDD with the cartesian product of the parameters by hand instead of using the operation provided by the framework. Now my runtime is 1'25''. It does make sense now, since there is some overhead to start the driver and move the jars to the workers.</p>