title: Writing using Pyspark and MongoDB Spark Connector gets stuck on Databricks
body: <p>I'm using the <a href="https://www.mongodb.com/docs/spark-connector/master/" rel="nofollow noreferrer">MongoDB-Spark-Connector</a> (2.12:3.0.1) to write data when running a Databricks (runtime 9.1 LTS ML Spark 3.1.2, Scala 2.12) job from notebook using PySpark. I'm able to run the job successfully when sampling smaller amount of rows, but when I run full scale (180 M rows) the job seems to get stuck after roughly 1.5 hours without throwing any error.</p>
  <p>To clarify - The spark process keeps on staying alive but nothing seems to happen on both the Spark nodes side, as well as on the MongoDB side. On the MongoDB side I see a drop of writes to 0 writes while at the same time on Ganglia I see the nodes utilization drops to near 0. The job on Spark UI is at a running state with just few last running tasks still represented as running though nothing is progressing.</p>
  <p>My initial code:</p>
  <pre class="lang-py prettyprint-override"><code>df.write.format(&quot;com.mongodb.spark.sql.DefaultSource&quot;).mode(&quot;overwrite&quot;) \
    .option(&quot;database&quot;, database) \
    .option(&quot;collection&quot;, destination_collection).save()
  </code></pre>
  <p>After investigating a bit it seemed like the root cause could be some timeouts occurring on the MongoDB side which might be related to the <a href="https://www.mongodb.com/docs/spark-connector/master/configuration/#output-configuration" rel="nofollow noreferrer">writeConcern</a> (w/wTimeoutMS) so I've added the following options to test if I get an exception on a very small allowed timeout but I didn't so I guess it's not being applied correctly.</p>
  <p>My refactored code:</p>
  <pre class="lang-py prettyprint-override"><code>df.write.format(&quot;com.mongodb.spark.sql.DefaultSource&quot;).mode(&quot;overwrite&quot;) \
    .option(&quot;database&quot;, database) \
    .option(&quot;collection&quot;, destination_collection) \
    .option(&quot;writeConcern.w&quot;, 2) \
    .option(&quot;writeConcern.wTimeoutMS&quot;, 1).save()
  </code></pre>
  <p>Have anyone else encountered this issue and have a proper solution?</p>