{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/3158","repository_url":"https://api.github.com/repos/open-mpi/ompi","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/3158/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/3158/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/3158/events","html_url":"https://github.com/open-mpi/ompi/issues/3158","id":214000568,"number":3158,"title":"Error happenss when launching a subprocess using openmpi","user":{"login":"cshanbo","id":9649941,"avatar_url":"https://avatars0.githubusercontent.com/u/9649941?v=3","gravatar_id":"","url":"https://api.github.com/users/cshanbo","html_url":"https://github.com/cshanbo","followers_url":"https://api.github.com/users/cshanbo/followers","following_url":"https://api.github.com/users/cshanbo/following{/other_user}","gists_url":"https://api.github.com/users/cshanbo/gists{/gist_id}","starred_url":"https://api.github.com/users/cshanbo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cshanbo/subscriptions","organizations_url":"https://api.github.com/users/cshanbo/orgs","repos_url":"https://api.github.com/users/cshanbo/repos","events_url":"https://api.github.com/users/cshanbo/events{/privacy}","received_events_url":"https://api.github.com/users/cshanbo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-03-14T08:37:23Z","updated_at":"2017-03-17T07:55:43Z","closed_at":null,"body":"Hi openmpi team,\r\n\r\nI am attempting run a [Theano](https://github.com/Theano/Theano)-based `Python` code for [neural machine translation](https://github.com/rsennrich/nematus) with openmpi and a [multi-node framework](https://github.com/cshanbo/platoon/tree/fix/multi-node) , but something weird happened. \r\nAny idea will be helpful and appreciated.\r\n\r\nBackground: The process of my codes is like:\r\n1. starting a training process\r\n2. starting a subprocess after some batches are trained (`Popen` or `exec_que` an external validation shell script). \r\n\r\nThe weird things are,\r\n1. Single-node Scenario\r\nwhen I use `single-node` training, which only launch the training process on a single node (machine), without `mpirun`, an exception still encountered in `validation, step 2`.\r\nThe error is like:\r\n```\r\n[nmyjs_104_22:33214] PMIX ERROR: NOT-FOUND in file src/client/pmix_client.c at line 199\r\nTraining time 236.7153s\r\nNumber of samples: 4010\r\n[nmyjs_104_22:33214] [[INVALID],INVALID] ORTE_ERROR_LOG: Unknown data type in file runtime/data_type_support/orte_dt_print_fns.c at line 161\r\n[nmyjs_104_22:33214] *** Process received signal ***\r\n[nmyjs_104_22:33214] Signal: Segmentation fault (11)\r\n[nmyjs_104_22:33214] Signal code: Invalid permissions (2)\r\n[nmyjs_104_22:33214] Failing at address: 0x2568700\r\n[nmyjs_104_22:33214] [ 0] /lib64/libpthread.so.0[0x3da4a0f710]\r\n[nmyjs_104_22:33214] [ 1] [0x2568700]\r\n[nmyjs_104_22:33214] *** End of error message ***\r\n```\r\nI'm not sure why openmpi influenced the experiment in a `single node` scenario. (Although importing some `Theano` related module invokes the `fork` WARNING, I don't think that's related to this issue)\r\n\r\n2. Multi-node scenario\r\nSimilar exception happens when using `mpirun -np 2 -H host1,host2 python xxx`.\r\nThe training process works fine util a subprocess, validation, is called. Similar error is invoked on two nodes, like:\r\n```\r\n*** glibc detected *** python: double free or corruption (!prev): 0x0000000002ba9f10 ***\r\n[nmyjs_104_22:01210] *** Process received signal ***\r\n[nmyjs_104_22:01210] Signal: Segmentation fault (11)\r\n[nmyjs_104_22:01210] Signal code: Invalid permissions (2)\r\n[nmyjs_104_22:01210] Failing at address: 0x29405e0\r\n[nmyjs_104_22:01210] [ 0] /lib64/libpthread.so.0[0x3da4a0f710]\r\n[nmyjs_104_22:01210] [ 1] [0x29405e0][nmyjs_104_22:01210] *** End of error message ***\r\n\r\n--------------------------------------------------------------------------\r\nMPI_ABORT was invoked on rank 0 in communicator <Unknown>\r\nwith errorcode 0.\r\n\r\nNOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.\r\nYou may or may not see output from other processes, depending on\r\nexactly when Open MPI kills them.\r\n\r\n[nmyjs_104_22:51831] OPAL ERROR: Timeout in file base/pmix_base_fns.c at line 193\r\n*** An error occurred in MPI_Init_thread\r\n*** on a NULL communicator\r\n*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,\r\n***    and potentially your MPI job)\r\n--------------------------------------------------------------------------\r\nIt looks like MPI_INIT failed for some reason; your parallel process is\r\nlikely to abort.  There are many reasons that a parallel process can\r\nfail during MPI_INIT; some of which are due to configuration or environment\r\nproblems.  This failure appears to be an internal failure; here's some\r\nadditional information (which may only be relevant to an Open MPI\r\ndeveloper):\r\n\r\n  ompi_dpm_dyn_init() failed\r\n  --> Returned \"Timeout\" (-15) instead of \"Success\" (0)\r\n--------------------------------------------------------------------------\r\n--------------------------------------------------------------------------\r\nMPI_ABORT was invoked on rank 0 in communicator <Unknown>\r\nwith errorcode 0.\r\n\r\nNOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.\r\nYou may or may not see output from other processes, depending on\r\nexactly when Open MPI kills them.\r\n```\r\n\r\n### example code\r\n\r\nWhen the `req` is `valid_err` (examples of what did `req` do can be found [here](https://github.com/cshanbo/platoon/blob/fix/multi-node/platoon/channel/controller.py#L206) ), invoking a subprocess to validate.\r\n```\r\n        elif 'valid_err' in req:\r\n            valid_err = req['valid_err']\r\n            print(\"valid:\", valid_err)\r\n            self.history_errs.append(valid_err)\r\n            harr = numpy.array(self.history_errs)\r\n\r\n            if self.external_validation_script:\r\n                print(\"Calling external validation script\")\r\n                cmd = [self.external_validation_script, str(self.valid_gpu), str('%s/model.npz'%self.model_dir), str(self.last_save_index), str(self.valid_datadir)]\r\n                self.exec_que.put(cmd)\r\n            if valid_err <= harr.min():\r\n                self.bad_counter = 0\r\n                control_response = 'best'\r\n                print(\"Best error valid:\", valid_err)\r\n            elif (len(self.history_errs) > self.patience and valid_err >= harr[:-self.patience].min()):\r\n                self.bad_counter += 1\r\n                print(\"Bad Counter add one, now:\", self.bad_counter)\r\n```\r\n\r\nIf anybody has any idea, please help. Thank you so very much."},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/287290015","html_url":"https://github.com/open-mpi/ompi/issues/3158#issuecomment-287290015","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/3158","id":287290015,"user":{"login":"rhc54","id":2440928,"avatar_url":"https://avatars2.githubusercontent.com/u/2440928?v=3","gravatar_id":"","url":"https://api.github.com/users/rhc54","html_url":"https://github.com/rhc54","followers_url":"https://api.github.com/users/rhc54/followers","following_url":"https://api.github.com/users/rhc54/following{/other_user}","gists_url":"https://api.github.com/users/rhc54/gists{/gist_id}","starred_url":"https://api.github.com/users/rhc54/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhc54/subscriptions","organizations_url":"https://api.github.com/users/rhc54/orgs","repos_url":"https://api.github.com/users/rhc54/repos","events_url":"https://api.github.com/users/rhc54/events{/privacy}","received_events_url":"https://api.github.com/users/rhc54/received_events","type":"User","site_admin":false},"created_at":"2017-03-17T07:55:43Z","updated_at":"2017-03-17T07:55:43Z","body":"@ggouaillardet People have tried your proposed approach, with mixed results. The problem is that you cannot tell which OMPI params should be kept, and which should be removed. If the user intends, for example, that an MCA param set in the environment actually govern all their MPI applications, then it won't work.\r\n\r\nIt also results in a program that isn't portable across MPI implementations, but perhaps that isn't a consideration here.\r\n\r\nSo bottom line is that @cshanbo can certainly try that approach and see if it works. If it does and there is no concern over portability, then it might be \"good enough\"."}}
{"actor":{"display_login":"rhc54"}}