title: convert cvs to database and graphing data: advice on optimisation
body: <p>Was hoping for some advice on best practice here. Working in objective C and Xcode.</p>
  
  <p>I made a "FileConverter" class which has a method to reads a cvs file with 7 columns of float values into a SQLite database (after verifying the data and parsing it).</p>
  
  <p>The way I have done this is to load the whole file into an NSString, then split into row components, then split each row into column components (saving the result as a 2x2 NSArray.</p>
  
  <p>I then open database and copy the array into the sqlite database.  I'm using the TEXT datatype for storage at the moment.  Once there, I plan to graph the data.</p>
  
  <p>It seems to check and convert the cvs ok.  However if the csv is quite long ( say 10,000 rows), I get the spinning wheel for several seconds while it does its work.  For shorter files it converts almost instantly.</p>
  
  <p>Ultimately, at the point the user clicks "Convert CSV", I will also be running another method which will graph the data and I expect this will result in huge delay while it reads the sql database, assembles the data into CGPoints and then draws into the graph view.</p>
  
  <p>My question is about how best to optimise the process, so it can handle the larger files without spinning wheels appearing.  Is this possible? </p>
  
  <p>a) Using NSStrings and NSArrays certainly makes the job of reading and splitting up the data super simple and makes verifying the data easy.  Is this the best way?  Should I  malloc a float array instead?</p>
  
  <p>b) I'm working on the basis that by saving the data as TEXT values in the database, converting them to CGFloat values will be straightforward, but realise this will add processing time.</p>
  
  <p>c)  I'm imagining that a sqlite3 database would be a faster way for getting the data when I come to graphing it, but I could also simply copy the cvs file and parse it at the point of graphing the data.</p>
  
  <p>Really appreciate advice on this </p>