{"action":"created","issue":{"url":"https://api.github.com/repos/azkaban/azkaban/issues/893","repository_url":"https://api.github.com/repos/azkaban/azkaban","labels_url":"https://api.github.com/repos/azkaban/azkaban/issues/893/labels{/name}","comments_url":"https://api.github.com/repos/azkaban/azkaban/issues/893/comments","events_url":"https://api.github.com/repos/azkaban/azkaban/issues/893/events","html_url":"https://github.com/azkaban/azkaban/issues/893","id":204979871,"number":893,"title":"Azkaban 3.0 leaks NioProcessor threads and over time they consume all the CPU available","user":{"login":"JabbaTheCoder","id":25514528,"avatar_url":"https://avatars.githubusercontent.com/u/25514528?v=3","gravatar_id":"","url":"https://api.github.com/users/JabbaTheCoder","html_url":"https://github.com/JabbaTheCoder","followers_url":"https://api.github.com/users/JabbaTheCoder/followers","following_url":"https://api.github.com/users/JabbaTheCoder/following{/other_user}","gists_url":"https://api.github.com/users/JabbaTheCoder/gists{/gist_id}","starred_url":"https://api.github.com/users/JabbaTheCoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JabbaTheCoder/subscriptions","organizations_url":"https://api.github.com/users/JabbaTheCoder/orgs","repos_url":"https://api.github.com/users/JabbaTheCoder/repos","events_url":"https://api.github.com/users/JabbaTheCoder/events{/privacy}","received_events_url":"https://api.github.com/users/JabbaTheCoder/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-02-02T19:59:59Z","updated_at":"2017-02-06T02:57:07Z","closed_at":null,"body":"I am currently running Azkaban 3.0 in a pre-production environment and I have a cron job that bounces the Azkaban web server once it consumes all the CPU available on the box and slows everything down. There is clearly a bug that causes gradual increase in CPU consumption over time. I enabled the JMX on the Azkaban web server, connected jvisualvm and was found out the following:\r\n\r\nOver time the number of NioProcessor threads grows from 0 to about 10. These threads are constantly consuming CPU eventually consuming all of it and requiring a restart of the Azkaban Web server.\r\n\r\nThe thread dump for these threads looks something like this:\r\n\r\njava.lang.Thread.State: RUNNABLE\r\nat sun.nio.ch.IOUtil.drain(Native Method)\r\nat sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)\r\n- locked <3e5de8a8> (a java.lang.Object)\r\nat sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)\r\n- locked <1b60bc70> (a sun.nio.ch.Util$2)\r\n- locked <3c55e39e> (a java.util.Collections$UnmodifiableSet)\r\n- locked <1f30b1fa> (a sun.nio.ch.EPollSelectorImpl)\r\nat sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\r\nat org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:97)\r\nat org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1074)\r\nat org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)\r\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\nat java.lang.Thread.run(Thread.java:745)\r\n\r\nLocked ownable synchronizers:\r\n- locked <7945192> (a java.util.concurrent.ThreadPoolExecutor$Worker)\r\n\r\n\"NioProcessor-1\" - Thread t@74\r\njava.lang.Thread.State: RUNNABLE\r\nat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)\r\nat sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)\r\nat sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)\r\nat sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)\r\n- locked <28189547> (a sun.nio.ch.Util$2)\r\n- locked <77adfc> (a java.util.Collections$UnmodifiableSet)\r\n- locked <3f0eda24> (a sun.nio.ch.EPollSelectorImpl)\r\nat sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\r\nat org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:97)\r\nat org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1074)\r\nat org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)\r\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\nat java.lang.Thread.run(Thread.java:745)\r\n\r\nLocked ownable synchronizers:\r\n- locked <4ca4266d> (a java.util.concurrent.ThreadPoolExecutor$Worker)"},"comment":{"url":"https://api.github.com/repos/azkaban/azkaban/issues/comments/277576024","html_url":"https://github.com/azkaban/azkaban/issues/893#issuecomment-277576024","issue_url":"https://api.github.com/repos/azkaban/azkaban/issues/893","id":277576024,"user":{"login":"JabbaTheCoder","id":25514528,"avatar_url":"https://avatars.githubusercontent.com/u/25514528?v=3","gravatar_id":"","url":"https://api.github.com/users/JabbaTheCoder","html_url":"https://github.com/JabbaTheCoder","followers_url":"https://api.github.com/users/JabbaTheCoder/followers","following_url":"https://api.github.com/users/JabbaTheCoder/following{/other_user}","gists_url":"https://api.github.com/users/JabbaTheCoder/gists{/gist_id}","starred_url":"https://api.github.com/users/JabbaTheCoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JabbaTheCoder/subscriptions","organizations_url":"https://api.github.com/users/JabbaTheCoder/orgs","repos_url":"https://api.github.com/users/JabbaTheCoder/repos","events_url":"https://api.github.com/users/JabbaTheCoder/events{/privacy}","received_events_url":"https://api.github.com/users/JabbaTheCoder/received_events","type":"User","site_admin":false},"created_at":"2017-02-06T02:57:07Z","updated_at":"2017-02-06T02:57:07Z","body":"Yes, I am referring to that project. I am testing a fix, if it works out I will contribute it. Otherwise I'll file a bug. In any event, I agree that this issue belongs in that project. When I filed the bug I didn't realize that the problem was caused by the plugin and not by Azkaban itself. I realized that that was the case after reading the code which took a little bit of time."}}
{"actor":{"display_login":"JabbaTheCoder"}}