title: Fast sqlite data retrieval with pandas
body: <p>For one batch process I query my database ~30_000x. Every query returns between 0 and 20_000 rows. Profiling the code shows that the most time is spend getting this data out.</p>
  <p>I tested 2 different methods with similar results. Assuming that the database schema is not the bottleneck, what can I do to speed up the data retrieval (besides going parallel)?</p>
  <ul>
  <li>I was thinking on using another library/wrapper for sqlite</li>
  <li>getting rid of pandas, but a lot of computation is done later on with the df, dask eventual faster?</li>
  <li>wrapping it in cpython/julia?</li>
  </ul>
  <pre><code>db = sqlite.connect('D:\data.db')
  cur = db.cursor()
  
  # 1.just pandas, 231 ms
  t = pd.read_sql(&quot;SELECT * FROM daily d WHERE d.id == 1 &quot;, db)
  
  # 2. db-api &amp; pandas, 242ms
  # query is actual half the time of 1., but creating that df cost time
  cur.execute(&quot;SELECT * FROM daily d WHERE d.id ==1&quot;)
  rows = cur.fetchall()
  t = pd.DataFrame(rows, columns=col)
  </code></pre>