title: Writing to elasticsearch indexes dynamically via pyspark
body: <p>I have a pyspark DataFrame like this:</p>
  
  <pre><code>my_df = spark.read.load("some-parquet-path")
  </code></pre>
  
  <p>I'd like to be able to write it out to some elasticsearch indexes dynamically based on the contents of the "id" column in my DataFrame. I tried doing this:</p>
  
  <pre><code>my_df.write.format(
      "org.elasticsearch.spark.sql"
  ).mode('overwrite').options(**conf).save("my_index_{id}/my_type")
  </code></pre>
  
  <p>But I get:</p>
  
  <pre><code>org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no such index
  </code></pre>
  
  <p>How can I do this?</p>
  
  <p><strong>Update</strong></p>
  
  <p>This seems to work when I change the mode from 'overwrite' to 'append'. It would be great to have an explanation of why that is the case...</p>