{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/7401","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/7401/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/7401/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/7401/events","html_url":"https://github.com/elastic/logstash/issues/7401","id":234844519,"number":7401,"title":"Too many file descriptors used by UDP and TCP plugins (IO.select())","user":{"login":"micoq","id":1741616,"avatar_url":"https://avatars0.githubusercontent.com/u/1741616?v=3","gravatar_id":"","url":"https://api.github.com/users/micoq","html_url":"https://github.com/micoq","followers_url":"https://api.github.com/users/micoq/followers","following_url":"https://api.github.com/users/micoq/following{/other_user}","gists_url":"https://api.github.com/users/micoq/gists{/gist_id}","starred_url":"https://api.github.com/users/micoq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/micoq/subscriptions","organizations_url":"https://api.github.com/users/micoq/orgs","repos_url":"https://api.github.com/users/micoq/repos","events_url":"https://api.github.com/users/micoq/events{/privacy}","received_events_url":"https://api.github.com/users/micoq/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-06-09T14:29:30Z","updated_at":"2017-07-02T20:27:10Z","closed_at":null,"body":"Currently, the udp and tcp input plugins (logstash-input-udp and logstash-input-tcp) cannot accept more than ~1024 clients because the use of `IO::Select` in these plugins (it's not an \"ulimit\" problem, it's a hard coded limitation of the `select()` syscall).\r\n\r\nIt can be an issue if you want to use Logstash as a central collector for Syslog events (e.g. to replace Rsyslog).\r\n\r\nIf the number of clients reach the limit, the plugin's thread crash (not Logstash) and cannot recover (despite of the message `A plugin had an unrecoverable error. Will restart this plugin.`) with the following message (for TCP):\r\n```\r\n[ERROR][logstash.inputs.tcp      ] An error occurred. Closing connection {:client=>\"10.0.0.1:50748\", :exception=>#<IOError: Too many open files>, :backtrace=>[\"org/jruby/RubyIO.java:3019:in `sysread'\", \"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.1.0/lib/logstash/inputs/tcp.rb:240:in `read'\", \"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.1.0/lib/logstash/inputs/tcp.rb:164:in `handle_socket'\", \"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.1.0/lib/logstash/inputs/tcp.rb:153:in `server_connection_thread'\", \"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.1.0/lib/logstash/inputs/tcp.rb:151:in `server_connection_thread'\"]}\r\n```\r\nor this message (for UDP):\r\n```\r\n[WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>[\"org/jruby/RubyIO.java:3705:in `select'\", \"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:93:in `udp_listener'\", \"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:56:in `run'\", \"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:425:in `inputworker'\", \"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `start_input'\"]}\r\n```\r\nTherefore, the listening socket is closed.\r\n\r\nIs there a better solution to avoid the `IO.select()` call (like `poll()` or maybe `nio` in the java layer) to handle more clients ?"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/312515023","html_url":"https://github.com/elastic/logstash/issues/7401#issuecomment-312515023","issue_url":"https://api.github.com/repos/elastic/logstash/issues/7401","id":312515023,"user":{"login":"micoq","id":1741616,"avatar_url":"https://avatars0.githubusercontent.com/u/1741616?v=3","gravatar_id":"","url":"https://api.github.com/users/micoq","html_url":"https://github.com/micoq","followers_url":"https://api.github.com/users/micoq/followers","following_url":"https://api.github.com/users/micoq/following{/other_user}","gists_url":"https://api.github.com/users/micoq/gists{/gist_id}","starred_url":"https://api.github.com/users/micoq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/micoq/subscriptions","organizations_url":"https://api.github.com/users/micoq/orgs","repos_url":"https://api.github.com/users/micoq/repos","events_url":"https://api.github.com/users/micoq/events{/privacy}","received_events_url":"https://api.github.com/users/micoq/received_events","type":"User","site_admin":false},"created_at":"2017-07-02T20:27:10Z","updated_at":"2017-07-02T20:27:10Z","body":"Well, it's a bit tricky:\r\nWhen `IO.Select()` is called, JRuby create a new Selector whenever you called `IO.Select()` before or not (the selector pool isn't used). On Linux, this will constantly create and destroy a file descriptor with `epoll_create`. It could explain why the UDP input plugin crash when there isn't any file descriptor available.\r\n\r\nThe TCP input plugin doesn't use `IO.Select()` but it uses selectors (and epoll) inside the blocking call `socket.sysread()`. Unlike to `IO.Select()`, the selector pool is used so only one selector is created but for each thread ! Actually, there is one selector pool per thread and the plugin creates one thread for each client. That's why there's so much selectors/file descriptors.\r\n\r\nI made a quick benchmark with a modified TCP plugin to use a single thread (with `IO.Select()`), it can handle thousands of clients and a total of 20000 ev/s (at 100% CPU). A better solution could be a configurable thread pool to distribute the connections.\r\nFinally, a pure java implementation would be better to avoid the creation/destruction of the selector in `IO.select()`."}}
{"actor":{"display_login":"micoq"}}