title: My Scala Spark code doesn't work, although pyspark it works fine
body: <p>I am running Linear regression in Pyspark with multiple parameters and it works fine, But if run the same code with same configuration in scala it gives me below error on fitting the model.</p>
  
  <p>Please ask question and I will put all the details, I am stuck here since last one month.</p>
  
  <p>19/12/06 23:59:35 ERROR cluster.YarnScheduler: Lost executor 6 on bvpr-bdaws09.vq.internal.vodafone.com: Container killed by YARN for exceeding memory limits.  1.5 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.</p>
  
  <p>one thing more in scala if I just directly fit the linear regression model without multiple parameters then it also works.</p>
  
  <pre><code>val spark = (SparkSession.builder
           .appName("MLib_scala")
           .master("local")
           .config("spark.driver.memory", "3g") 
           .config("spark.executor.memory", "20g") 
           .config("spark.dynamicAllocation.maxExecutors","5")
           .config("spark.executor.cores", "3")
           .config("spark.yarn.executor.memoryOverhead","2g")
           .enableHiveSupport()
           .getOrCreate())
  
  val df1 = spark.sql("select * from Table_4kCols_10k_rows") //some 3GB hive table
  val list_cols =df1.columns
  val featureCols = list_cols.filter(_!= "dormant_flag")
  val df2=df1.withColumnRenamed("dormant_flag","label")
  val assembler = new VectorAssembler().setInputCols(featureCols).setOutputCol("features")
  val df = assembler.transform(df2)
  val seed = 5043
  
  
  val Array(train, test) = df.randomSplit(Array(0.7, 0.3))
  val lr = new LogisticRegression().setLabelCol("label").setFeaturesCol("features")
  val fittedLR = lr.fit(train) // This works fine
  val stages = Array(lr)
  val pipeline = new Pipeline().setStages(stages)
  val params = new ParamGridBuilder().addGrid(lr.elasticNetParam, Array(0.0, 0.5, 1.0)).addGrid(lr.regParam, Array(0.1, 2.0)).build()
  val evaluator = new BinaryClassificationEvaluator().setMetricName("areaUnderROC").setRawPredictionCol("prediction").setLabelCol("label")
  val tvs = new TrainValidationSplit().setTrainRatio(0.75).setEstimatorParamMaps(params).setEstimator(pipeline).setEvaluator(evaluator)
  val tvsFitted = tvs.fit(train) //This gives an error //pyspark works fine
  </code></pre>