title: How to debug a traceless crash
body: <p>During development of out application we have in special encountered a really nasty bug. The symptom is quite simply that the process disappears. The logs just end abruptly, no crash dumps or anything can be found, no zombie processes exist. Dr.Watson haven't noticed anything leaving us without any trace.</p>
  
  <p>The error is not simple to reproduce, it takes on average 3-4 hours to reproduce this error, doing the same actions repeatedly. So somewhere there is some kind of race condition. We have special functions handling both SEH and normal exceptions so none of these should go unnoticed.</p>
  
  <p>The debugging must be done on a special computer, because it is running on very specialized hardware. So only remote debugging is available. And when remote debugging is connected C++ builder doesn't noticed that the application is missing, and crash and burns when we try to do any debugging on the non existent process.</p>
  
  <p>We are using a great variety of technologies with this software:</p>
  
  <ul>
  <li>OpenGL</li>
  <li>Directshow + some COTS filters</li>
  <li>COM/DCOM</li>
  <li>Serial COM ports talking to specialized hardware</li>
  <li>C++ Builder (which uses different stackframes than VC++)</li>
  </ul>
  
  <p>So, as you understand, I do not have much to work with here. What I am doing now is that I am trying to narrow it down by logging in different places in the code to find if there is some particular point in the code the error occurs. I am also trying to remove as many aspects of the action I am performing to get the case as simple as possible. But this is a really complex operation and this process is taking a lot of time, and time is (as usual) a scarce resource. </p>
  
  <p>I am wondering if anyone out there have good tips for me, either to the cause (in general what causes a process just to stop without any notification) or to techniques for debugging such an elusive failure?</p>