title: How to handle batch SQL inserts from Spring JDBC when table column too small
body: <p>I'm trying to batch insert a large volume of data into a MYSQL database and need advice about how to handle the situation where a table column is not large enough to cope with the size of the incoming data.</p>
  
  <p>Since the data set is very large (over 1Gb) I'm performing a streaming read of the input data into memory. At certain points, I "flush" the data and try to insert it into the table using jdbcTemplate.batchUpdate . All of the columns of the destination table are VARCHARs and are assumed to be of size 50. This is perfectly adequate for the vast majority of the data. However since occasionally data arrives which is larger than size 15, the INSERT will fail (Data too long for column).</p>
  
  <p>What is the best approach to tackling this issue? I don't want to blindly make all of the table columns larger since this seems like a strategy which won't scale very well. I'd also prefer not to pre-parse the data due to its size. So should I wait for the fail, then perform an appropriate "ALTER TABLE" and re-submit the data? Is it possible to cache away the failed items, determine the required column size and re-submit failed items only?</p>
  
  <p>I'm looking for pointers and general advice about the optimal way to perform this.</p>
  
  <p>Thanks.</p>