title: How to use Spark DataFrame column value to update the elasticsearch
body: <p>Problem description:  I am trying to update documents in elastic search. I have data in a dataframe. Dataframe has <code>esIndex</code>, <code>id</code> and <code>body</code> attributes. I need to use the value in <code>index</code> attribute to update the document in the right index. The following code below does not work. What is possible fix?</p>
  <p>Command 1</p>
  <p><code>val esUrl = &quot;test.elasticsearch.com&quot;</code></p>
  <p>Command 2</p>
  <pre><code>import spark.implicits._
  val df = Seq(
    (&quot;test_datbricks001&quot;,&quot;1221866122238136328&quot;, &quot;TEST DATA 001&quot;),
      (&quot;test_datbricks001&quot;,&quot;1221866122238136329&quot;, &quot;TEST DATA 002&quot;),
        (&quot;test_datbrick&quot;,&quot;1221866122238136329&quot;, &quot;TEST DATA 002&quot;)
  
  ).toDF(&quot;esIndex&quot;, &quot;id&quot;, &quot;body&quot;)
  </code></pre>
  <p>Command 3</p>
  <pre><code>display(df)
  </code></pre>
  <p>Command 4</p>
  <pre><code>import org.apache.spark.sql.functions.col
  val retval = df.toDF.write
          .format(&quot;org.elasticsearch.spark.sql&quot;)
          .option(&quot;es.nodes.wan.only&quot;,&quot;true&quot;)
          .option(&quot;es.mapping.id&quot;,&quot;id&quot;)
          .option(&quot;es.mapping.exclude&quot;, &quot;id, esIndex&quot;)
          .option(&quot;es.port&quot;,&quot;80&quot;)
          .option(&quot;es.nodes&quot;, esUrl)
          .option(&quot;es.write.operation&quot;, &quot;update&quot;) 
          .option(&quot;es.index.auto.create&quot;, &quot;no&quot;)
          .mode(&quot;Append&quot;)
          .save(&quot;{esIndex}&quot;)  // &lt;&lt;== This index is not taking value from &quot;index&quot; column of df
  </code></pre>
  <p>I am getting the following error <code> EsHadoopIllegalArgumentException: Target index [{esIndex}] does not exist and auto-creation is disabled [setting 'es.index.auto.create' is 'false']</code></p>
  <p>Question: How do I get code in Command 4 to use the value in <code>esIndex</code> column of <code>df</code></p>