title: Getting Spark, Java, and MongoDB to work together
body: <p>Similar to <a href="https://stackoverflow.com/questions/33391840/getting-spark-python-and-mongodb-to-work-together">my question here</a> but this time it's Java, not Python, causing me problems.</p>
  <p>I have followed the steps advised (to the best of my knowledge) <a href="https://github.com/mongodb/mongo-hadoop/wiki/Spark-Usage" rel="noreferrer">here</a> but since I'm using hadoop-2.6.1 I think I should be using the old API, rather than the new API referred to in the example.</p>
  <p>I'm working on Ubuntu and the various component versions I have are</p>
  <ul>
  <li><strong>Spark</strong> spark-1.5.1-bin-hadoop2.6</li>
  <li><strong>Hadoop</strong> hadoop-2.6.1</li>
  <li><strong>Mongo</strong> 3.0.8</li>
  <li><strong>Mongo-Hadoop connector</strong> jars included via Maven</li>
  <li><strong>Java</strong> 1.8.0_66</li>
  <li><strong>Maven</strong> 3.0.5</li>
  </ul>
  <p>My Java program is basic</p>
  <pre class="lang-java prettyprint-override"><code>import org.apache.spark.api.java.*;
  import org.apache.spark.SparkConf;
  import org.apache.spark.api.java.function.Function;
  import com.mongodb.hadoop.MongoInputFormat;
  import org.apache.hadoop.conf.Configuration;
  import org.bson.BSONObject;
  
  public class SimpleApp {
    public static void main(String[] args) {
      Configuration mongodbConfig = new Configuration();
      mongodbConfig.set(&quot;mongo.job.input.format&quot;, &quot;com.mongodb.hadoop.MongoInputFormat&quot;);
      mongodbConfig.set(&quot;mongo.input.uri&quot;, &quot;mongodb://localhost:27017/db.collection&quot;);
      SparkConf conf = new SparkConf().setAppName(&quot;Simple Application&quot;);
      JavaSparkContext sc = new JavaSparkContext(conf);
      JavaPairRDD&lt;Object, BSONObject&gt; documents = sc.newAPIHadoopRDD(
          mongodbConfig,            // Configuration
          MongoInputFormat.class,   // InputFormat: read from a live cluster.
          Object.class,             // Key class
          BSONObject.class          // Value class
      );
    }
  }
  </code></pre>
  <p>It is building fine using Maven (<code>mvn package</code>) with the following pom file</p>
  <pre class="lang-xml prettyprint-override"><code>&lt;project&gt;
  &lt;groupId&gt;edu.berkeley&lt;/groupId&gt;
    &lt;artifactId&gt;simple-project&lt;/artifactId&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
    &lt;name&gt;Simple Project&lt;/name&gt;
    &lt;packaging&gt;jar&lt;/packaging&gt;
    &lt;version&gt;1.0&lt;/version&gt;
    &lt;dependencies&gt;
      &lt;dependency&gt; &lt;!-- Spark dependency --&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-core_2.10&lt;/artifactId&gt;
        &lt;version&gt;1.5.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.mongodb&lt;/groupId&gt;
          &lt;artifactId&gt;mongo-java-driver&lt;/artifactId&gt;
          &lt;version&gt;3.2.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.mongodb.mongo-hadoop&lt;/groupId&gt;
        &lt;artifactId&gt;mongo-hadoop-core&lt;/artifactId&gt;
        &lt;version&gt;1.4.2&lt;/version&gt;
      &lt;/dependency&gt;
    &lt;/dependencies&gt;
    &lt;build&gt;
      &lt;plugins&gt;
          &lt;plugin&gt;
              &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
              &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
              &lt;configuration&gt;
                  &lt;source&gt;1.8&lt;/source&gt;
                  &lt;target&gt;1.8&lt;/target&gt;
              &lt;/configuration&gt;
          &lt;/plugin&gt;
      &lt;/plugins&gt;
  &lt;/build&gt;
  &lt;/project&gt;
  </code></pre>
  <p>I then submit the jar</p>
  <pre><code>/usr/local/share/spark-1.5.1-bin-hadoop2.6/bin/spark-submit --class &quot;SimpleApp&quot; --master local[4] target/simple-project-1.0.jar
  </code></pre>
  <p>and get the following error</p>
  <pre><code>Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: com/mongodb/hadoop/MongoInputFormat
      at SimpleApp.main(SimpleApp.java:18)
  </code></pre>
  <hr />
  <h1>NOTICE</h1>
  <p>I edited this question on the 18th December as it had grown far too confusing and verbose. Previous comments might look irrelevant. The context of the question, however, is the same.</p>