title: Spark JDBC - Read -> update -> write huge table without primary key
body: <p>I am trying to update the few fields of each row of a big mysql table (having close to <code>500</code> million rows). The table doesn't have any primary key (or having string primary key like UUID). I don't have enough executor memory to read and hold the entire data in once. Can anyone please let me know what are my options to process such tables.</p>
  <p>Below is the schema</p>
  <pre><code>CREATE TABLE Persons ( Personid varchar(255) NOT NULL, LastName varchar(255) NOT NULL, FirstName varchar(255) DEFAULT NULL, Email varchar(255) DEFAULT NULL, Age int(11) DEFAULT NULL) ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
  </code></pre>
  <p>Spark code is like</p>
  <pre><code> SparkSession spark = SparkSession.builder().master(&quot;spark://localhost:7077&quot;).appName(&quot;KMASK&quot;).getOrCreate();
  DataFrame rawDataFrame = spark.read().format(&quot;jdbc&quot;).load();
  rawDataFrame.createOrReplaceTempView(&quot;data&quot;);
  //encrypt is UDF
  String sql = &quot;select Personid, LastName, FirstName, encrypt(Email), Age from data&quot;;
  Dataset newData = spark.sql(sql);
  newData.write().mode(SaveMode.Overwrite).format(&quot;jdbc&quot;).options(options).save();
  </code></pre>
  <p>This table has around <code>150</code> million records, size of data is around <code>6GB</code>. My executor memory is just <code>2 gb</code>. Can I process this table using Spark - jdbc.</p>