title: Cannot load a saved Spark model in pyspark: "java.lang.NoSuchMethodException"
body: <p>When I run the following Python program</p>
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.ml.classification import LinearSVC
  from pyspark.sql import SparkSession
  
  spark = SparkSession.builder.appName("Sparkmodel").getOrCreate()
  data = spark.read.format("libsvm").load("/usr/local/spark/data/mllib/sample_libsvm_data.txt")
  model = LinearSVC().fit(data)
  model.save("mymodel")
  LinearSVC.load("mymodel")
  </code></pre>
  
  <p>the <code>load</code> fails with a "java.lang.NoSuchMethodException".</p>
  
  <pre class="lang-python prettyprint-override"><code>/anaconda3/envs/scratch/bin/python /Users/billmcn/src/toy/sparkmodel/sparkmodel/little.py
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  Setting default log level to "WARN".
  To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
  17/11/12 13:23:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  17/11/12 13:23:06 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
  17/11/12 13:23:06 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
  17/11/12 13:23:17 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
  17/11/12 13:23:17 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
  Traceback (most recent call last):
    File "/Users/billmcn/src/toy/sparkmodel/sparkmodel/little.py", line 9, in &lt;module&gt;
      LinearSVC.load("mymodel")
    File "/anaconda3/envs/scratch/lib/python3.6/site-packages/pyspark/ml/util.py", line 257, in load
      return cls.read().load(path)
    File "/anaconda3/envs/scratch/lib/python3.6/site-packages/pyspark/ml/util.py", line 197, in load
      java_obj = self._jread.load(path)
    File "/anaconda3/envs/scratch/lib/python3.6/site-packages/py4j/java_gateway.py", line 1160, in __call__
      answer, self.gateway_client, self.target_id, self.name)
    File "/anaconda3/envs/scratch/lib/python3.6/site-packages/pyspark/sql/utils.py", line 63, in deco
      return f(*a, **kw)
    File "/anaconda3/envs/scratch/lib/python3.6/site-packages/py4j/protocol.py", line 320, in get_return_value
      format(target_id, ".", name), value)
  py4j.protocol.Py4JJavaError: An error occurred while calling o64.load.
  : java.lang.NoSuchMethodException: org.apache.spark.ml.classification.LinearSVCModel.&lt;init&gt;(java.lang.String)
      at java.lang.Class.getConstructor0(Class.java:3082)
      at java.lang.Class.getConstructor(Class.java:1825)
      at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:328)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:280)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:748)
  
  
  Process finished with exit code 1
  </code></pre>
  
  <p>The "mymodel" directory is created and its contents appear to be valid.</p>
  
  <p>I am running Spark 2.2.0 and pyspark 2.2.0. I have the following mllib jars in my installation.</p>
  
  <pre><code>&gt; ll /usr/local/spark.versions/spark-2.2.0-bin-hadoop2.7/jars/spark-mllib*
  -rw-r--r--@ 1 billmcn  admin  6501535 Jun 30 18:09 /usr/local/spark.versions/spark-2.2.0-bin-hadoop2.7/jars/spark-mllib_2.11-2.2.0.jar
  -rw-r--r--@ 1 billmcn  admin   182887 Jun 30 18:09 /usr/local/spark.versions/spark-2.2.0-bin-hadoop2.7/jars/spark-mllib-local_2.11-2.2.0.jar
  </code></pre>
  
  <p>And the latter contains the class I want.</p>
  
  <pre><code>jar tf /usr/local/spark.versions/spark-2.2.0-bin-hadoop2.7/jars/spark-mllib_2.11-2.2.0.jar | grep LinearSVCModel
  org/apache/spark/ml/classification/LinearSVCModel$LinearSVCWriter$Data.class
  org/apache/spark/ml/classification/LinearSVCModel$.class
  org/apache/spark/ml/classification/LinearSVCModel$LinearSVCWriter.class
  org/apache/spark/ml/classification/LinearSVCModel$LinearSVCReader.class
  org/apache/spark/ml/classification/LinearSVCModel$$anonfun$11.class
  org/apache/spark/ml/classification/LinearSVCModel$LinearSVCWriter$$typecreator1$1.class
  org/apache/spark/ml/classification/LinearSVCModel$LinearSVCWriter$Data$.class
  org/apache/spark/ml/classification/LinearSVCModel.class
  </code></pre>
  
  <p>The same problem happens on two different machines.</p>
  
  <p>What am I doing wrong?</p>