title: Processing millions of database records in Java
body: <p>I have a requirement to write a batch job that fetches rows from a database table and based on a certain conditions, write to other tables or update this row with a certain value. We are using spring and jdbc to fetch the result set and iterate through and process the records using a standalone java program that is scheduled to run weekly. I know this is not the right way to do it, but we had to do it as a temporary solution. As the records grow into millions, we will end up with out of memory exceptions, so I know this is not the best approach.</p>
  
  <p>Can any of you recommend what is the best way to deal with such a situation?</p>
  
  <p>Use Threads and fetch 1000 records per thread and process them in parallel?</p>
  
  <p>(OR)</p>
  
  <p>Use any other batch mechanism to do this (i know there is spring-batch but have never used this)</p>
  
  <p>(OR)</p>
  
  <p>Any other ideas?</p>