title: Help required with pthreads debugging
body: <p>I have a server-client program in which there are multiple threads in both the server and client. There are variable number of clients and servers (like 3 servers (replicas), 10 clients). I am debugging a source file in this program. I think there is some kind of deadlock, possibly the following:</p>
  
  <p>A mutex lock is already held by a server method and a request from the client invokes a server method which wants to acquire the mutex again.</p>
  
  <p>The program is launched by a test script which spawns the servers and clients and makes the client send specific requests to the servers. I have used the following code in the suspicious area of code to see if there is a deadlock, but it doesnt seem to work, ie the code enters neither block:</p>
  
  <pre><code>if (pthread_mutex_lock(&amp;a_mutex) == EDEADLK) {
      cout&lt;&lt;"couldnt acquire lock."&lt;&lt;endl;
  }
  else cout&lt;&lt;"acquired lock"&lt;&lt;endl;
  </code></pre>
  
  <p>I tried to debug (by attaching one running server process) with gdb. I added "display" and "watch" (in different runs of gdb) for a_mutex. I get a result of the following form:</p>
  
  <pre><code>1: a_mutex = {__data = {__lock = 2, __count = 0, __owner = 4193, __kind = 0, __nusers = 2, 
  {__spins = 0, __list = {__next = 0x0}}}, 
    __size = "\002\000\000\000\000\000\000\000a\020\000\000\000\000\000\000\002\000\000  \000\000\000\000", __align = 2}
  </code></pre>
  
  <p>I dont know the meaning of all the things in the above output, but I could see that a thread (4193) is holding the mutex. I saw the backtrace of that thread (snipped):</p>
  
  <pre><code>#0  0xb8082430 in __kernel_vsyscall ()
  #1  0xb7e347a6 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
  #2  0xb7e345be in sleep () from /lib/tls/i686/cmov/libc.so.6
  #3  0x0804cb59 in class1::method1 (this=0xbfa9fe6c, clt=1, id=
      {static npos = 4294967295, _M_dataplus = {&lt;std::allocator&lt;char&gt;&gt; = {&lt;__gnu_cxx::new_allocator&lt;char&gt;&gt; = {&lt;No data fields&gt;}, &lt;No data fields&gt;}, _M_p = 0xb7c9c11c "l/%\b"}})
  at file1.cc:33
  </code></pre>
  
  <p>I dont know how and where the bug is.</p>
  
  <p>I would highly appreciate any help with the following questions:</p>
  
  <ol>
  <li>What is a good method of debugging such conditions/programs?</li>
  <li>How do I detect the deadlock condition (ie where a lock is being held and not released)?</li>
  <li>In such a multi-process program, is there a better way of using gdb? (ie inspecting states in all processes? configuring gdb to watch/display a variable before the start of the process?) </li>
  <li>Because, when I attach gdb with the server after it has started (by the tester script), the server might have already advanced ahead of the code which I want to inspect. I tried adding a sleep(20) before the suspicious area to help me with gdb, but I think this is not a good way. I also think that opening multiple terminals, starting servers and client manually and checking states of each of them is also not a very good idea (please correct me if i am wrong).</li>
  </ol>
  
  <p>PS: I have read this <a href="https://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads">question</a> already.</p>
  
  <p>Thank you very much.</p>