title: How do I debug a process that is in uninterruptible sleep?
body: <p>This is on a Linux system (Red Hat distro).</p>
  
  <p>I have a daemon implemented in Perl. It monitors a table of jobs and, when it finds a new one, it forks off a child to execute the job.</p>
  
  <p>I'm finding that the children are hanging around in the process table in a uninterruptable state:</p>
  
  <pre><code> 682 16658  100 dumpdeco dumpdeco ?          0:27.62  0.0  1.8 D perl /root/runObjectBrowserJobs.pl
   1453 16658  100 dumpdeco dumpdeco ?          0:13.34  0.0  1.5 D perl /root/runObjectBrowserJobs.pl
   1458 16658  100 dumpdeco dumpdeco ?          0:12.42  0.0  1.5 D perl /root/runObjectBrowserJobs.pl
   1467 16658  100 dumpdeco dumpdeco ?          0:13.35  2.9  1.8 D perl /root/runObjectBrowserJobs.pl
   1474 16658  100 dumpdeco dumpdeco ?          0:14.31  0.0  2.1 D perl /root/runObjectBrowserJobs.pl
   1479 16658  100 dumpdeco dumpdeco ?          0:13.73  0.0  2.1 D perl /root/runObjectBrowserJobs.pl
   1496 16658  100 dumpdeco dumpdeco ?          0:13.97  0.0  2.0 D perl /root/runObjectBrowserJobs.pl
   1499 16658  100 dumpdeco dumpdeco ?          0:13.93  0.0  2.2 D perl /root/runObjectBrowserJobs.pl
   1501 16658  100 dumpdeco dumpdeco ?          0:14.47  0.0  1.4 D perl /root/runObjectBrowserJobs.pl
   1503 16658  100 dumpdeco dumpdeco ?          0:14.86  2.0  2.2 D perl /root/runObjectBrowserJobs.pl
   1505 16658  100 dumpdeco dumpdeco ?          0:13.88  1.0  2.0 D perl /root/runObjectBrowserJobs.pl
   1520 16658  100 dumpdeco dumpdeco ?          0:14.39  0.0  1.5 D perl /root/runObjectBrowserJobs.pl
   1522 16658  100 dumpdeco dumpdeco ?          0:14.12  1.0  1.8 D perl /root/runObjectBrowserJobs.pl
   1531 16658  100 dumpdeco dumpdeco ?          0:16.45  0.0  1.7 D perl /root/runObjectBrowserJobs.pl
   3619 16658  100 dumpdeco dumpdeco ?          2:03.59  0.0  4.5 D perl /root/runObjectBrowserJobs.pl
  </code></pre>
  
  <p>I know from my logging that the children have finished the job that was assigned to them. They have logged this and the job has been removed from the pending queue. So, from code inspection, they don't have anything left to do but exit.</p>
  
  <p>This appears to be a new phenomenon. This is an old piece of code and I've done the dance of zombie processes, so I'm pretty sure that this isn't an issue with incorrect reaping.</p>
  
  <p>What I'm trying to figure out is how to find out what these processes are hanging on. I've tried <code>strace</code>, but it doesn't return anything. To be honest, that's not a surprise, because the process (by my guess) isn't <em>actually</em> doing anything other than sleeping, waiting for some syscall to return.</p>
  
  <p>So, I'm thinking:</p>
  
  <ul>
  <li>Is there a way to print the current call stack of a running process?</li>
  <li>Is there a way to dump a process and view its call stack at the time of the dump?</li>
  <li>Any suggestions as to what I should be looking for? I'm thinking a broken storage driver (we've had other issues with our storage lately), but blaming drivers always seems like a lazy option to me.</li>
  </ul>