title: VFS: file-max limit 1231582 reached
body: <p>I'm running a Linux 2.6.36 kernel, and I'm seeing some random errors.  Things like</p>
  <pre><code>ls: error while loading shared libraries: libpthread.so.0: cannot open shared object file: Error 23
  </code></pre>
  <p>Yes, my system can't consistently run an 'ls' command.  :(</p>
  <p>I note several errors in my dmesg output:</p>
  <pre><code># dmesg | tail
  [2808967.543203] EXT4-fs (sda3): re-mounted. Opts: (null)
  [2837776.220605] xv[14450] general protection ip:7f20c20c6ac6 sp:7fff3641b368 error:0 in libpng14.so.14.4.0[7f20c20a9000+29000]
  [4931344.685302] EXT4-fs (md16): re-mounted. Opts: (null)
  [4982666.631444] VFS: file-max limit 1231582 reached
  [4982666.764240] VFS: file-max limit 1231582 reached
  [4982767.360574] VFS: file-max limit 1231582 reached
  [4982901.904628] VFS: file-max limit 1231582 reached
  [4982964.930556] VFS: file-max limit 1231582 reached
  [4982966.352170] VFS: file-max limit 1231582 reached
  [4982966.649195] top[31095]: segfault at 14 ip 00007fd6ace42700 sp 00007fff20746530 error 6 in libproc-3.2.8.so[7fd6ace3b000+e000]
  </code></pre>
  <p>Obviously, the file-max errors look suspicious, being clustered together and recent.</p>
  <pre><code># cat /proc/sys/fs/file-max
  1231582
  # cat /proc/sys/fs/file-nr
  1231712 0       1231582
  </code></pre>
  <p>That also looks a bit odd to me, but the thing is, there's no way I have 1.2 million files open on this system.  I'm the only one using it, and it's not visible to anyone outside the local network.</p>
  <pre><code># lsof | wc
    16046  148253 1882901
  # ps -ef | wc 
      574    6104   44260
  </code></pre>
  <p>I saw some documentation saying:</p>
  <blockquote>
  <p>file-max &amp; file-nr:</p>
  <p>The kernel allocates file handles dynamically, but as yet it doesn't free them again.</p>
  <p>The value in file-max denotes the maximum number of file- handles that the Linux kernel will allocate. When you get lots of error messages about running out of file handles, you might want to increase this limit.</p>
  <p>Historically, the three values in file-nr denoted the number of allocated file handles, the number of allocated but unused file handles, and the maximum number of file handles. Linux 2.6 always reports 0 as the number of free file handles -- this is not an error, it just means that the number of allocated file handles exactly matches the number of used file handles.</p>
  <p>Attempts to allocate more file descriptors than file-max are reported with printk, look for &quot;VFS: file-max limit reached&quot;.</p>
  </blockquote>
  <p>My first reading of this is that the kernel basically has a built-in file descriptor leak, but I find that very hard to believe.  It would imply that any system in active use needs to be rebooted every so often to free up the file descriptors.  As I said, I can't believe this would be true, since it's normal to me to have Linux systems stay up for months (even years) at a time.  On the other hand, I also can't believe that my nearly-idle system is holding over a million files open.</p>
  <p>Does anyone have any ideas, either for fixes or further diagnosis?  I could, of course, just reboot the system, but I don't want this to be a recurring problem every few weeks.  As a stopgap measure, I've quit Firefox, which was accounting for almost 2000 lines of lsof output (!) even though I only had one window open, and now I can run 'ls' again, but I doubt that will fix the problem for long.  (edit: Oops, spoke too soon.  By the time I finished typing out this question, the symptom was/is back)</p>
  <p>Thanks in advance for any help.</p>