title: Initializing logistic regression coefficients when using the Spark dataset-based ML APIs?
body: <p>By default, logistic regression training initializes the coefficients to be all-zero. However, I would like to initialize the coefficients myself. This would be useful, for example, if a previous training run crashed after several iterations -- I could simply restart training with the last known set of coefficients.</p>
  
  <p>Is this possible with any of the dataset/dataframe-based APIs, preferably Scala?</p>
  
  <p>Looking at the Spark source code, it seems that there <em>is</em> a method <code>setInitialModel</code> to initialize the model and its coefficients, <a href="https://github.com/apache/spark/blob/267aca5bd5042303a718d10635bc0d1a1596853f/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala#L315" rel="nofollow noreferrer">but it's unfortunately marked as private</a>.</p>
  
  <p>The RDD-based API seems to allow initializing coefficients: one of the overloads of <code>LogisticRegressionWithSGD.run(...)</code> accepts an <code>initialWeights</code> vector. However, I would like to use the dataset-based API instead of the RDD-based API because (1) the former supports elastic net regularization (I couldn't figure out how to do elastic net with the RDD-based logistic regression) and (2) because <a href="https://spark.apache.org/docs/2.1.1/ml-guide.html" rel="nofollow noreferrer">the RDD-based API is in maintenance mode</a>.</p>
  
  <p>I could always try using reflection to call that private <code>setInitialModel</code> method, but I would like to avoid this if possible (and maybe that wouldn't even work... I also can't tell if <code>setInitialModel</code> is marked private for a good reason).</p>