{"action":"created","issue":{"url":"https://api.github.com/repos/rundeck/rundeck/issues/2651","repository_url":"https://api.github.com/repos/rundeck/rundeck","labels_url":"https://api.github.com/repos/rundeck/rundeck/issues/2651/labels{/name}","comments_url":"https://api.github.com/repos/rundeck/rundeck/issues/2651/comments","events_url":"https://api.github.com/repos/rundeck/rundeck/issues/2651/events","html_url":"https://github.com/rundeck/rundeck/issues/2651","id":246797680,"number":2651,"title":"Zombie scp processes and failed jobs","user":{"login":"pbenas","id":10399279,"avatar_url":"https://avatars0.githubusercontent.com/u/10399279?v=4","gravatar_id":"","url":"https://api.github.com/users/pbenas","html_url":"https://github.com/pbenas","followers_url":"https://api.github.com/users/pbenas/followers","following_url":"https://api.github.com/users/pbenas/following{/other_user}","gists_url":"https://api.github.com/users/pbenas/gists{/gist_id}","starred_url":"https://api.github.com/users/pbenas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pbenas/subscriptions","organizations_url":"https://api.github.com/users/pbenas/orgs","repos_url":"https://api.github.com/users/pbenas/repos","events_url":"https://api.github.com/users/pbenas/events{/privacy}","received_events_url":"https://api.github.com/users/pbenas/received_events","type":"User","site_admin":false},"labels":[{"id":7615329,"url":"https://api.github.com/repos/rundeck/rundeck/labels/bug","name":"bug","color":"e10c02","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-07-31T15:29:47Z","updated_at":"2017-09-13T11:36:21Z","closed_at":null,"author_association":"NONE","body":"Issue type: Bug report\r\n\r\n**My Rundeck detail**\r\n\r\n* Rundeck version: 2.8.3\r\n* install type: war\r\n* OS Name/version: CentOS Linux release 7.3.1611 (Core)\r\n* DB Type/version: postgresql-9.2.18-1.el7.x86_64\r\n\r\n**Expected Behavior**\r\nNo failed jobs, no zombie SCP processes. \r\n\r\n**Actual Behavior**\r\nRundeck executes the scp, then fails the job. Couple of zombie scp processes remains.\r\n```\r\ntomcat   22856  0.0  0.0      0     0 ?        Z    12:36   0:00 [scp] <defunct> \r\n```\r\nDigging through the logs, I've found following two errors:\r\n```\r\nJul 31 13:22:10 rundeck server: INFO  stream: Failed: IOFailure: Cannot run program \"/usr/bin/ssh\": error=11, Resource temporarily unavailable severity=info\r\n```\r\nand \r\n```\r\n\r\n\r\nJul 31 13:14:24 rundeck server: ERROR WorkflowService: updateStateForStep([identifier:6/1, index:0, stepStateChange:StepStateChangeImpl{stepState=StepS\r\ntateImpl{executionState=RUNNING, metadata=null, errorMessage='null'}, nodeName='na1-dss-v21n01', nodeState=true}, timestamp:Mon Jul 31 13:14:23 CEST 20\r\n17]): No such property: nodeState for class: java.util.Date severity=info\r\nJul 31 13:14:24 rundeck server: groovy.lang.MissingPropertyException: No such property: nodeState for class: java.util.Date severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl$_updateStateForStep_closure7.doCall(MutableWorkfl\r\nowStateImpl.groovy:221) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateStateForStep(MutableWorkflowStateImpl.groov\r\ny:220) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.descendUpdateStateForStep(MutableWorkflowStateImp\r\nl.groovy:323) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateImpl.updateStateForStep(MutableWorkflowStateImpl.groov\r\ny:140) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.app.internal.workflow.ExceptionHandlingMutableWorkflowState.updateStateForStep(ExceptionHandling\r\nMutableWorkflowState.groovy:36) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.app.internal.workflow.MutableWorkflowStateListener.stepStateChanged(MutableWorkflowStateListener\r\n.groovy:40) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.core.execution.workflow.state.WorkflowExecutionStateListenerAdapter.notifyAllStepState(WorkflowE\r\nxecutionStateListenerAdapter.java:72) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.core.execution.workflow.state.WorkflowExecutionStateListenerAdapter.beginExecuteNodeStep(Workflo\r\nwExecutionStateListenerAdapter.java:203) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.app.internal.workflow.MultiWorkflowExecutionListener.beginExecuteNodeStep(MultiWorkflowExecution\r\nListener.groovy:104) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:109) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.core.execution.dispatch.ParallelNodeDispatcher$ExecNodeStepCallable.call(ParallelNodeDispatcher.\r\njava:215) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.core.execution.dispatch.ParallelNodeDispatcher$ExecNodeStepCallable.call(ParallelNodeDispatcher.\r\njava:190) severity=info\r\nJul 31 13:14:24 rundeck server: at com.dtolabs.rundeck.core.cli.CallableWrapperTask.execute(CallableWrapperTask.java:52) severity=info\r\nJul 31 13:14:24 rundeck server: at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) severity=info\r\nJul 31 13:14:24 rundeck server: at org.apache.tools.ant.Task.perform(Task.java:348) severity=info\r\nJul 31 13:14:24 rundeck server: at org.apache.tools.ant.taskdefs.Parallel$TaskRunnable.run(Parallel.java:435) severity=info\r\nJul 31 13:14:24 rundeck server: at java.lang.Thread.run(Thread.java:745) severity=info\r\n```\r\n\r\n**How to reproduce Behavior**\r\n- configure file copier to scp\r\n- run a job containing a scritpt on about 700 target nodes. \r\nNot reproducible when running over smaller sets of nodes. Our instance has 12 CPUs and 72G of RAM. Short term load reaches small, single-digit numbers in peaks.\r\n"},"comment":{"url":"https://api.github.com/repos/rundeck/rundeck/issues/comments/329140547","html_url":"https://github.com/rundeck/rundeck/issues/2651#issuecomment-329140547","issue_url":"https://api.github.com/repos/rundeck/rundeck/issues/2651","id":329140547,"user":{"login":"pbenas","id":10399279,"avatar_url":"https://avatars0.githubusercontent.com/u/10399279?v=4","gravatar_id":"","url":"https://api.github.com/users/pbenas","html_url":"https://github.com/pbenas","followers_url":"https://api.github.com/users/pbenas/followers","following_url":"https://api.github.com/users/pbenas/following{/other_user}","gists_url":"https://api.github.com/users/pbenas/gists{/gist_id}","starred_url":"https://api.github.com/users/pbenas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pbenas/subscriptions","organizations_url":"https://api.github.com/users/pbenas/orgs","repos_url":"https://api.github.com/users/pbenas/repos","events_url":"https://api.github.com/users/pbenas/events{/privacy}","received_events_url":"https://api.github.com/users/pbenas/received_events","type":"User","site_admin":false},"created_at":"2017-09-13T11:36:21Z","updated_at":"2017-09-13T11:36:21Z","author_association":"NONE","body":"Had time for some deeper testing through different rundeck releases - 2.7.3, through 2.8.* to 2.9.3 and found out this is not a regression, I was just confused by a different issue of rundeck 2.9 - https://github.com/rundeck/rundeck/issues/2756\r\n\r\nFor our 12-cpu instance, the limit of zombie creation remains the same - something between 650 and 700 target nodes being acessed simulaneously (`threadcount`). The zombie creation is associated with the `Fork: resource temporarily unavailable` error, no matter how hard are the limits for max number of processes and max open files pushed. In majority of the times, the zombies were spawned almost immediately after the job was stated. My though is the fork may succeed but rundeck thinks it did not (therefore not knowing about the child process) and some reasonable throttling could help - delay the start of some processes by second or two, so it won't perform so many forks within a short period of time."}}
{"actor":{"display_login":"pbenas"}}