title: Async rewrite of sync code performs 20 times slower in loops
body: <p>I'm trying not to panic here, so please, bear with me! :S</p>
  
  <p>I've spent a considerable amount of time rewriting a big chunk of code from sync (i.e. thread-blocking) to async (i.e. C# 6+). The code in question runs inside an ASP.NET application and spans everything from low-level ADO.NET DB access to higher-level unit-of-work pattern, and finally a custom async HTTP handler for public API access - the full server-side stack, so to speak. The primary purpose of the rewrite wasn't optimization, but untangling, general clean-up, and bringing the code up to something that resembles a modern and deliberate design. Naturally, optimization gain was implicitly assumed.</p>
  
  <p>Everything in general is great and I'm very satisfied with the overall quality of the new code, as well as the improved scalability it's shown so far in the last couple of weeks of real-world tests. The CPU and memory loads on the server have fallen drastically!</p>
  
  <p>So what's the problem, you might ask?</p>
  
  <p>Well, I've recently been tasked with optimizing a simple data import that is still utilizing the old sync code. Naturally, it didn't take me long before I tried changing it to the new async code-base to see what would happen.</p>
  
  <p>Given everything, the import code is quite simple. It's basically a loop that reads items from a list that's been previously read into memory, adds each of them individually to a unit of work, and saves it to a SQL database by means of an INSERT statement. After the loop is done, the unit of work is committed, which makes the changes permanent (i.e. the DB transaction is committed as well).</p>
  
  <p>The problem is that the new code takes about <em>20 times as long</em> as the old one, when the expectation was quite the opposite! I've checked and double-checked and there is no obvious overhead in the new code that would warrant such sluggishness.</p>
  
  <p>To be specific: the old code is able to import 1100 items/sec steadily, while the new one manages 40 items/sec AT BEST (on average, it's even less, because the rate is falling slightly over time)! If I run the same test over a VPN, so that the network cost outweighs everything else, the throughputs is somewhere along 25 items/sec for sync and 20 for async.</p>
  
  <p>I've read about multiple cases here on SO which report a 10-20% slowdown when switching from sync to async in similar situations and I was prepared to deal with that for tight loops such as mine. But a 20-fold penalty in a non-networked scenario?! That's completely unacceptable!</p>
  
  <p>What is my best course of action here? How do I tackle this unexpected problem?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I've run the import under a profiler, as suggested.</p>
  
  <p>I'm not sure what to make of the results, though. It would seem that the process spends more than 80% of its time just... waiting. See for yourselves:</p>
  
  <p><a href="https://i.stack.imgur.com/Ysnmc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Ysnmc.png" alt="Profiler report"></a></p>
  
  <p>The 14% spent inside the custom HTTP handler corresponds to the <code>IDataReader.Read</code> which is a consequence of a tiny remainder of the old sync API. This is still subject to optimization and is likely to be reduced in the near future. Regardless, it's dwarfed by the <code>WaitAny</code> cost, which definitely isn't there in the all-sync version!</p>
  
  <p>What's curious is that the report isn't showing any direct calls from my code to <code>WaitAny</code>, which makes me think this is probably part of the async/await infrastructure. Am I wrong in this conclusion? I kind of hope I am!</p>
  
  <p>What worries me is that I might be reading this all wrong. I know that async costs are much harder to reason about than single-threaded costs. In the end, the <code>WaitAny</code> might be nothing more than the equivalent of the "Sytem Idle Process" on Windows - an artificial representation of the CPU infrastructure that reflects a free percentage of the CPU resource.</p>
  
  <p>Can anyone shed some light here for me, please?</p>