{"action":"created","issue":{"url":"https://api.github.com/repos/radical-cybertools/radical.pilot/issues/1281","repository_url":"https://api.github.com/repos/radical-cybertools/radical.pilot","labels_url":"https://api.github.com/repos/radical-cybertools/radical.pilot/issues/1281/labels{/name}","comments_url":"https://api.github.com/repos/radical-cybertools/radical.pilot/issues/1281/comments","events_url":"https://api.github.com/repos/radical-cybertools/radical.pilot/issues/1281/events","html_url":"https://github.com/radical-cybertools/radical.pilot/issues/1281","id":214552895,"number":1281,"title":"DVM error does not terminate the agent","user":{"login":"andre-merzky","id":1319674,"avatar_url":"https://avatars1.githubusercontent.com/u/1319674?v=3","gravatar_id":"","url":"https://api.github.com/users/andre-merzky","html_url":"https://github.com/andre-merzky","followers_url":"https://api.github.com/users/andre-merzky/followers","following_url":"https://api.github.com/users/andre-merzky/following{/other_user}","gists_url":"https://api.github.com/users/andre-merzky/gists{/gist_id}","starred_url":"https://api.github.com/users/andre-merzky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andre-merzky/subscriptions","organizations_url":"https://api.github.com/users/andre-merzky/orgs","repos_url":"https://api.github.com/users/andre-merzky/repos","events_url":"https://api.github.com/users/andre-merzky/events{/privacy}","received_events_url":"https://api.github.com/users/andre-merzky/received_events","type":"User","site_admin":false},"labels":[{"id":46694419,"url":"https://api.github.com/repos/radical-cybertools/radical.pilot/labels/bug","name":"bug","color":"e11d21","default":true},{"id":559956096,"url":"https://api.github.com/repos/radical-cybertools/radical.pilot/labels/termination","name":"termination","color":"1d76db","default":false}],"state":"closed","locked":false,"assignee":{"login":"andre-merzky","id":1319674,"avatar_url":"https://avatars1.githubusercontent.com/u/1319674?v=3","gravatar_id":"","url":"https://api.github.com/users/andre-merzky","html_url":"https://github.com/andre-merzky","followers_url":"https://api.github.com/users/andre-merzky/followers","following_url":"https://api.github.com/users/andre-merzky/following{/other_user}","gists_url":"https://api.github.com/users/andre-merzky/gists{/gist_id}","starred_url":"https://api.github.com/users/andre-merzky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andre-merzky/subscriptions","organizations_url":"https://api.github.com/users/andre-merzky/orgs","repos_url":"https://api.github.com/users/andre-merzky/repos","events_url":"https://api.github.com/users/andre-merzky/events{/privacy}","received_events_url":"https://api.github.com/users/andre-merzky/received_events","type":"User","site_admin":false},"assignees":[{"login":"andre-merzky","id":1319674,"avatar_url":"https://avatars1.githubusercontent.com/u/1319674?v=3","gravatar_id":"","url":"https://api.github.com/users/andre-merzky","html_url":"https://github.com/andre-merzky","followers_url":"https://api.github.com/users/andre-merzky/followers","following_url":"https://api.github.com/users/andre-merzky/following{/other_user}","gists_url":"https://api.github.com/users/andre-merzky/gists{/gist_id}","starred_url":"https://api.github.com/users/andre-merzky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andre-merzky/subscriptions","organizations_url":"https://api.github.com/users/andre-merzky/orgs","repos_url":"https://api.github.com/users/andre-merzky/repos","events_url":"https://api.github.com/users/andre-merzky/events{/privacy}","received_events_url":"https://api.github.com/users/andre-merzky/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/radical-cybertools/radical.pilot/milestones/34","html_url":"https://github.com/radical-cybertools/radical.pilot/milestone/34","labels_url":"https://api.github.com/repos/radical-cybertools/radical.pilot/milestones/34/labels","id":2346203,"number":34,"title":"0.46","description":"Release 0.46, target date 28-Apr-2017","creator":{"login":"ibethune","id":7913631,"avatar_url":"https://avatars1.githubusercontent.com/u/7913631?v=3","gravatar_id":"","url":"https://api.github.com/users/ibethune","html_url":"https://github.com/ibethune","followers_url":"https://api.github.com/users/ibethune/followers","following_url":"https://api.github.com/users/ibethune/following{/other_user}","gists_url":"https://api.github.com/users/ibethune/gists{/gist_id}","starred_url":"https://api.github.com/users/ibethune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ibethune/subscriptions","organizations_url":"https://api.github.com/users/ibethune/orgs","repos_url":"https://api.github.com/users/ibethune/repos","events_url":"https://api.github.com/users/ibethune/events{/privacy}","received_events_url":"https://api.github.com/users/ibethune/received_events","type":"User","site_admin":false},"open_issues":18,"closed_issues":68,"state":"open","created_at":"2017-02-24T16:12:57Z","updated_at":"2017-06-05T13:34:52Z","due_on":"2017-04-28T07:00:00Z","closed_at":null},"comments":6,"created_at":"2017-03-15T23:08:37Z","updated_at":"2017-06-05T13:34:52Z","closed_at":"2017-06-05T13:34:52Z","body":"```\r\n2017-03-15 17:56:26,034: agent_0             : MainProcess                     : DVMWatcher     : DEBUG   : dvm output: --------------------------------------------------------------------------\r\n2017-03-15 17:56:26,034: agent_0             : MainProcess                     : DVMWatcher     : DEBUG   : dvm output: All nodes which are allocated for this job are already filled.\r\n2017-03-15 17:56:26,035: agent_0             : MainProcess                     : DVMWatcher     : DEBUG   : dvm output: --------------------------------------------------------------------------\r\n2017-03-15 17:56:26,712: agent_0             : MainProcess                     : DVMWatcher     : DEBUG   : dvm output: [smic004:84124] 59 more processes have sent help message help-orte-rmaps-base.txt / orte-rmaps-base:all-available-resources-used\r\n2017-03-15 17:56:26,713: agent_0             : MainProcess                     : DVMWatcher     : DEBUG   : dvm output: [smic004:84124] Set MCA parameter \"orte_base_help_aggregate\" to 0 to see all help / error messages\r\n2017-03-15 17:56:27,039: agent_0             : MainProcess                     : agent_0.idler._check_units_cb: INFO    : units pulled:    0\r\n2017-03-15 17:56:27,718: agent_0             : MainProcess                     : DVMWatcher     : INFO    : DVM stopped (0)\r\n```\r\n\r\nbut the agent lives happily ever after, continues to churn through units, failing then with \r\n\r\n```\r\n$ cat unit.000081/STDOUT\r\n------------------------------------------------------------\r\nA process or daemon was unable to complete a TCP connection\r\nto another process:\r\n  Local host:    smic004\r\n  Remote host:   <unknown>\r\nThis is usually caused by a firewall on the remote host. Please\r\ncheck that any firewall (e.g., iptables) has been disabled and\r\ntry again.\r\n------------------------------------------------------------\r\n```\r\n\r\nThis relates to the TODO at https://github.com/radical-cybertools/radical.pilot/blob/devel/src/radical/pilot/agent/lm/orte.py#L139.\r\n\r\nThis has been observed on supermic in the `xsede.supermic_orte` configuration in this stack:\r\n\r\n```\r\n$ radical-stack\r\npython            : 2.7.12+\r\nvirtualenv        : /home/merzky/radical/radical.pilot.devel/ve\r\nradical.utils     : 0.45\r\nsaga-python       : 0.45\r\nradical.pilot     : split-28-gde1f156@feature-supermic_ortelib\r\n```"},"comment":{"url":"https://api.github.com/repos/radical-cybertools/radical.pilot/issues/comments/306188683","html_url":"https://github.com/radical-cybertools/radical.pilot/issues/1281#issuecomment-306188683","issue_url":"https://api.github.com/repos/radical-cybertools/radical.pilot/issues/1281","id":306188683,"user":{"login":"andre-merzky","id":1319674,"avatar_url":"https://avatars1.githubusercontent.com/u/1319674?v=3","gravatar_id":"","url":"https://api.github.com/users/andre-merzky","html_url":"https://github.com/andre-merzky","followers_url":"https://api.github.com/users/andre-merzky/followers","following_url":"https://api.github.com/users/andre-merzky/following{/other_user}","gists_url":"https://api.github.com/users/andre-merzky/gists{/gist_id}","starred_url":"https://api.github.com/users/andre-merzky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andre-merzky/subscriptions","organizations_url":"https://api.github.com/users/andre-merzky/orgs","repos_url":"https://api.github.com/users/andre-merzky/repos","events_url":"https://api.github.com/users/andre-merzky/events{/privacy}","received_events_url":"https://api.github.com/users/andre-merzky/received_events","type":"User","site_admin":false},"created_at":"2017-06-05T13:34:52Z","updated_at":"2017-06-05T13:34:52Z","body":"Tested, behaves as expected."}}
{"actor":{"display_login":"andre-merzky"}}