title: Cause of `fork: retry: No child processes`
body: <p>When I run a certain commercial build flow in OpenSuse, I encounter the following error:</p>
  
  <pre><code>fork: retry: No child processes
  </code></pre>
  
  <p>I encounter this error at various locations in the flow that corresponds with different executables, so I believe that the tool itself is not the problem.   However, I run several instances (up to 16) of this build flow at the same time.  The build is dispatched to different servers according to load, so sometimes a few builds run on the same machine.  I suspect that the server load has somehow to do with this, but I cannot figure out exactly how.</p>
  
  <p>I suspect that the error message corresponds with the <code>EAGAIN</code> error of <code>fork</code>. The <code>fork</code> manual gives several potential causes for this error:</p>
  
  <ol>
  <li>the <code>RLIMIT_NPROC</code> soft resource limit (set via <code>setrlimit(2)</code>), which limits the number of processes and threads for a real user ID, was reached</li>
  <li>the kernel's system-wide limit on the number of processes and threads, <code>/proc/sys/kernel/threads-max</code>, was reached (see <code>proc(5)</code>)</li>
  <li>the maximum number of PIDs, <code>/proc/sys/kernel/pid_max</code>, was reached</li>
  </ol>
  
  <p>I believe none of these are satisfied, because</p>
  
  <ol>
  <li><code>ulimit -a</code> reports unlimited as the maximum number of user processes.</li>
  <li><code>/proc/sys/kernel/threads-max</code> is set to 1031129. I monitor the number of lines in <code>ps auxH</code>, which should be the number of threads (plus the header line that ps outputs I guess), at a 1-second granularity, but it never gets higher than 917.</li>
  <li><code>/proc/sys/kernel/pid_max</code> is set to 32768. I monitor the number of processes using <code>ps aux</code>, but it never gets higher than 726.</li>
  </ol>
  
  <p>I believe I read somewhere that there is also a possibility that the kernel ran out of other (memory?) resources.  How can I check that?  Or is something else wrong in my line of reasoning?</p>
  
  <p>The output form <code>ulimit -a</code> is:</p>
  
  <pre><code>core file size          (blocks, -c) 0
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 515564
  max locked memory       (kbytes, -l) 64
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 5120
  pipe size            (512 bytes, -p) 8
  POSIX message queues     (bytes, -q) 819200
  real-time priority              (-r) 0
  stack size              (kbytes, -s) 8192
  cpu time               (seconds, -t) unlimited
  max user processes              (-u) unlimited
  virtual memory          (kbytes, -v) unlimited
  file locks                      (-x) unlimited
  </code></pre>