title: Pyspark dataframe or parquet file to DynamoDB
body: <p>I want to put a pyspark dataframe or a parquet file into a DynamoDB table</p>
  <p>The pyspark dataframe that I have has 30MM rows and 20 columns</p>
  <p><strong>Solution 1: using boto3, pandas and Batch writing (<a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/dynamodb.html" rel="nofollow noreferrer">Amazon DynamoDB</a>)</strong></p>
  <p>With this I read the parquet file and pass it to pandas, then I put row by row into the DynamoDB table, but this is taking too long, is very very slow</p>
  <pre><code>import boto3
  
  dynamodb = boto3.resource('dynamodb', region_name='name')
  
  table = dynamodb.Table('DynamoDB_table_name')
  with table.batch_writer() as batch:
      for index, row in pandas_dataframe.iterrows():
        batch.put_item(
            Item = {
                'column_name_DynamoDB_table': int(row['column_name_in_pandas_dataframe']),
                ...
            }
        )
  </code></pre>
  <p><strong>Solution 2: using boto3, pyspark and SQL (<a href="https://stackoverflow.com/questions/53044026/how-to-write-pyspark-dataframe-to-dynamodb-table">how-to-write-pyspark-dataframe-to-dynamodb-table</a>)</strong></p>
  <p>Here I keep getting an error in the step 3 describe in the solution, a <strong>ParseException</strong> error, I check in the amazon documentation and I see the code is correct (<a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/EMR_Hive_Commands.html" rel="nofollow noreferrer">EMR_Hive_Commands.html</a>), maybe it is not SQL code and that is my error, but if it's not, I don't know which language is</p>
  <pre><code>-- Step 1
  DROP TABLE IF EXISTS TEMP;
  CREATE TABLE TEMP(
          column_name_DynamoDB_table type,
          ... )
   STORED AS ORC; 
  
  --step 2.1
  pyspark_dataframe.createOrReplaceTempView(&quot;df&quot;)
  
  
  --step 2.2
  INSERT INTO temp
      SELECT *
      FROM df
  
  
  --step 3
  CREATE TABLE TEMPTODYNAMO(
          column_name_DynamoDB_table type,
          ... )
  STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
  TBLPROPERTIES ( &quot;dynamodb.table.name&quot; =&quot;temp-to-dynamo&quot; , 
                  &quot;dynamodb.column.mapping&quot; = &quot;col1:column_name_DynamoDB_table,...&quot;);
  </code></pre>
  <p>the error I keep getting:</p>
  <pre><code>Error in SQL statement: ParseException: 
  Operation not allowed: STORED BY(line 22, pos 0)
  
  == SQL ==
  CREATE TABLE TEMPTODYNAMO(
          column_name_DynamoDB_table type,
          ...  )
  STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
  ^^^
  TBLPROPERTIES ( &quot;dynamodb.table.name&quot; =&quot;temp-to-dynamo&quot; , 
                  &quot;dynamodb.column.mapping&quot; = &quot;col1:column_name_DynamoDB_table,...&quot;)
  </code></pre>
  <p><strong>Solution 3: using boto3, pyspark and com.audienceproject (<a href="https://github.com/audienceproject/spark-dynamodb" rel="nofollow noreferrer">Spark+DynamoDB</a>)</strong></p>
  <p>I did not understand what to put in the code, the python code show in the page is:</p>
  <pre><code># Python
  # Load a DataFrame from a Dynamo table. Only incurs the cost of a single scan for schema inference.
  dynamoDf = spark.read.option(&quot;tableName&quot;, &quot;SomeTableName&quot;) \
                       .format(&quot;dynamodb&quot;) \
                       .load() # &lt;-- DataFrame of Row objects with inferred schema.
  
  # Scan the table for the first 100 items (the order is arbitrary) and print them.
  dynamoDf.show(100)
  
  # write to some other table overwriting existing item with same keys
  dynamoDf.write.option(&quot;tableName&quot;, &quot;SomeOtherTable&quot;) \
                .format(&quot;dynamodb&quot;) \
                .save()
  </code></pre>
  <p>But I really did not get where to put the name of my DynamoDB table and my pyspark dataframe</p>
  <p>Update: I tried</p>
  <pre><code>pysaprk_dataframe.write.option(&quot;tableName&quot;, &quot;name_DynamoDB_table&quot;) \
                  .format(&quot;dynamodb&quot;) \
                  .save()
  </code></pre>
  <p>And got this error:</p>
  <pre><code>AnalysisException: TableProvider implementation dynamodb cannot be written with ErrorIfExists mode, please use Append or Overwrite modes instead
  </code></pre>
  <p>Regards</p>