title: Apache Beam -> BigQuery - insertId for deduplication not working
body: <p>I am streaming data from kafka to BigQuery using apache beam with google dataflow runner.
  I wanted to make use of insertId for deduplication, that I found described in google docs. But even tho inserts are happening within few seconds from each other I still see a lot of rows with the same insertId.
  Now I'm wondering that perhaps I am not using the API correctly to take advantage of deduplication mechanism for streaming inserts offered by BQ.</p>
  
  <p>My code in beam for writing looks as follows:</p>
  
  <pre><code>payments.apply("Write Fx Payments to BQ", BigQueryIO.&lt;FxPayment&gt;write()
              .withFormatFunction(ps -&gt; FxTableRowConverter.convertFxPaymentToTableRow(ps))
              .to(bqTradePaymentTable)
              .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
              .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND));
  </code></pre>
  
  <p>Besides all other fields I am setting insertId directly on TableRow in FxTableRowConverter.convertFxPaymentToTableRow method passed to BigQueryIO as format function:</p>
  
  <pre><code>row.set("insertId", insertId);
  </code></pre>
  
  <p>I also added that field as a column to BQ. Without it, it was failing on inserts (obviously).
  I couldn't find any other way to set insertId directly on BigQueryIO other than adding it to TableRow object.</p>
  
  <p>Is this the correct way of using this? Because it does not work for me, I am seeing many duplicates even tho I shouldn't, since like I already mentioned inserts are happening within seconds. BigQuery doc states that streaming buffer is keeping insertId for at least one minute.</p>