title: WinDbg takes extremely long time to loading symbols; is searching every directory in large network UNC symbol store
body: <p>I've spent several days trying to speed up loading of symbols when debugging crash dumps using <a href="https://msdn.microsoft.com/en-US/windows/hardware/gg454513" rel="noreferrer">WinDbg</a>, and I'm unable to get past a particular problem. </p>
  
  <p>The issue is that when symbols for a module in the dump doesn't exist in any accessible symbol store or symbol server location (e.g. it's a third-party modules without available symbols), WinDbg will spend literally hours looking for them. </p>
  
  <p>I've set up my symbol path correctly to properly set the search order and the cache directories:</p>
  
  <pre><code>.sympath cache*C:\SymbolCache1;\\our.corp\SymbolStore;SRV*C:\SymbolCache2*http://msdl.microsoft.com/download/symbols
  </code></pre>
  
  <p>Running with <code>!sym noisy</code> and <code>.reload /f</code> I can see:</p>
  
  <pre><code>SYMSRV:  Notifies the client application that a proxy has been detected. 
  SYMSRV:  Connecting to the Server: http://msdl.microsoft.com/download/symbols. 
  SYMSRV:  Successfully connected to the Server. 
  SYMSRV:  Sending the information request to the server. 
  SYMSRV:  Successfully sent the information request to the server. 
  SYMSRV:  Waiting for the server to respond to a request. 
  SYMSRV:  Successfully received a response from the server. 
  SYMSRV:  Closing the connection to the Server. 
  SYMSRV:  Successfully closed the connection to the Server. 
  SYMSRV:  c:\SymbolCache1\Some3rdParty.dll\0060D200cd1000\Some3rdParty.dll not found 
  SYMSRV:  c:\SymbolCache2\Some3rdParty.dll\0060D200cd1000\Some3rdParty.dll not found 
  SYMSRV:  http://msdl.microsoft.com/download/symbols/Some3rdParty.dll/0060D200cd1000/Some3rdParty.dll not found 
  &lt;---- !!!! hanging here with *BUSY* showing in WinDbg
  </code></pre>
  
  <p>By running <a href="https://technet.microsoft.com/en-us/sysinternals/bb795533.aspx" rel="noreferrer">Process Monitor</a> at the point in which it's hanging, I can see that WinDbg is searching what appears to be <strong><em>every directory</em></strong> in our giant network symbol store (\our.corp\SymbolStore) looking for symbols, even in directories for modules that are clearly unrelated. </p>
  
  <p>What's weird is that in WinDbg you can see that it's extracted the timestamp of the module (0060D200cd1000) and is using that to look in the expected location in local directories and the MS symbol server. I can't figure out why it's doing a full scan of our (massive) network symbol store. Perhaps there's something unique about how it treats UNC paths?</p>
  
  <p>This search can take 15 minutes or more per symbol, and if the dump has many missing symbols this can cause a <code>!analyze -v</code> to take hours (if you're using the Visual Studio integration of WinDbg, it causes the hang to occur as soon as you load a crash dump, since for some reason that integration tries to load all symbols immediately despite <code>.symopt</code> settings).</p>
  
  <p>This problem is also easily reproducible if you try to load symbols for a non-existent made-up module name, e.g. <code>.reload /f bogus.dll</code>. </p>
  
  <p>Here's my WinDbg .symopt settings:</p>
  
  <pre><code>0:000&gt; .symopt
  Symbol options are 0x30337:
    0x00000001 - SYMOPT_CASE_INSENSITIVE
    0x00000002 - SYMOPT_UNDNAME
    0x00000004 - SYMOPT_DEFERRED_LOADS
    0x00000010 - SYMOPT_LOAD_LINES
    0x00000020 - SYMOPT_OMAP_FIND_NEAREST
    0x00000100 - SYMOPT_NO_UNQUALIFIED_LOADS
    0x00000200 - SYMOPT_FAIL_CRITICAL_ERRORS
    0x00010000 - SYMOPT_AUTO_PUBLICS
    0x00020000 - SYMOPT_NO_IMAGE_SEARCH
  </code></pre>
  
  <p>I've looked all over thinking that there must be some flag to control this, but I can't seem to find it. </p>
  
  <p>A couple things:</p>
  
  <ul>
  <li>This is not an issue with network speed or lack of local symbol cache. The issue only occurs with symbols that cannot be found, and only with the UNC symbol store (e.g. not with the Microsoft symbol server)</li>
  <li>I've already tried SYMOPT_NO_PUBLICS instead of SYMOPT_AUTO_PUBLICS</li>
  <li>I've verified that my symbol path is what I expect it to be, using the <code>sympath</code> command. I've also tried using the <code>_NT_SYMBOL_PATH</code> environment variable instead.</li>
  <li>I know that I can exclude certain symbols via a configuration file, but this isn't a workable solution because sometimes the missing symbol name is not known in advance</li>
  <li>I've seen someone else on the internet had this same issue and mentioned it on a Microsoft forum in a post titled "<a href="https://social.msdn.microsoft.com/Forums/en-US/93fa6283-46e9-444b-b4ac-e359c114bf2d/poor-windbg-6111404-performance-when-it-is-pointed-to-a-large-symbol-cache?forum=windbg&amp;prof=required" rel="noreferrer">Poor WinDbg 6.11.1.404 performance when it is pointed to a large symbol cache</a>" but received no help.</li>
  </ul>