title: 'Streaming' data into Sql server
body: <p>I'm working on a project where we're receiving data from multiple sources, that needs to be saved into various tables in our database.</p>
  
  <p>Fast.</p>
  
  <p>I've played with various methods, and the fastest I've found so far is using a collection of TableValue parameters, filling them up and periodically sending them to the database via a corresponding collection of stored procedures.</p>
  
  <p>The results are quite satisfying. However, looking at disk usage (% Idle Time in Perfmon), I can see that the disk is getting periodically 'thrashed' (a 'spike' down to 0% every 13-18 seconds), whilst in between the %Idle time is around 90%. I've tried varying the 'batch' size, but it doesn't have an enormous influence.</p>
  
  <ol>
  <li>Should I be able to get better throughput by (somehow) avoiding the spikes while decreasing the overall idle time?</li>
  <li>What are some things I should be looking out to work out where the spiking is happening? (The database is in Simple recovery mode, and pre-sized to 'big', so it's not the log file growing)</li>
  <li>Bonus: I've seen other questions referring to 'streaming' data into the database, but this seems to involve having a <code>Stream</code> from another database (last section <a href="http://msdn.microsoft.com/en-us/library/bb675163.aspx" rel="nofollow noreferrer">here</a>). Is there any way I could shoe-horn 'pushed' data into that?</li>
  </ol>
  
  <p><img src="https://i.stack.imgur.com/1xoah.png" alt="enter image description here"></p>