{"action":"created","issue":{"url":"https://api.github.com/repos/singularityware/singularity/issues/494","repository_url":"https://api.github.com/repos/singularityware/singularity","labels_url":"https://api.github.com/repos/singularityware/singularity/issues/494/labels{/name}","comments_url":"https://api.github.com/repos/singularityware/singularity/issues/494/comments","events_url":"https://api.github.com/repos/singularityware/singularity/issues/494/events","html_url":"https://github.com/singularityware/singularity/issues/494","id":208181103,"number":494,"title":"Concurrency issue when running multiple jobs","user":{"login":"belledon","id":19876414,"avatar_url":"https://avatars1.githubusercontent.com/u/19876414?v=3","gravatar_id":"","url":"https://api.github.com/users/belledon","html_url":"https://github.com/belledon","followers_url":"https://api.github.com/users/belledon/followers","following_url":"https://api.github.com/users/belledon/following{/other_user}","gists_url":"https://api.github.com/users/belledon/gists{/gist_id}","starred_url":"https://api.github.com/users/belledon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/belledon/subscriptions","organizations_url":"https://api.github.com/users/belledon/orgs","repos_url":"https://api.github.com/users/belledon/repos","events_url":"https://api.github.com/users/belledon/events{/privacy}","received_events_url":"https://api.github.com/users/belledon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":65,"created_at":"2017-02-16T16:53:51Z","updated_at":"2017-03-05T22:44:27Z","closed_at":null,"body":"This was encountered on the OpenMInd cluster, managed by @satra.\r\n\r\nThe general issue is: Random jobs submitted via sbatch will fail (usually 1-2%).\r\n\r\nJobs submission layout:\r\n500-1000 jobs are submitted via sbatch's stdin. An example is generated below. Essentially, singularity execs a simply python script (test.py) using its own python3 environment. Just a quick note, `xvfb-run` from within the container exacerbates the issue due to creating tmp directories it may not be able to write to and messing up the exit code of the passed command. This is why I call singularity inside the `xvfb-run -a` call in a later example. \r\n```\r\n#!/bin/sh\r\nmodule purge\r\nmodule add openmind/anaconda\r\nmodule add openmind/xvfb-fix/0.1\r\nmodule add openmind/singularity/2.2.1\r\nexport GRASPIT=/graspit\r\nexport PYTHONPATH=$PYTHONPATH:/graspPlanningInstall/include/grasp_planning_graspit/\r\nsingularity exec -B /om:/om -W /om/user/belledon/bash/ /om/user/belledon/singularity_vm/ubuntu.img /usr/bin/python3 /om/user/belledon/git/vision\r\n-touch-objects/py/test.py\r\n```\r\nWhere test.py is:\r\n```\r\nimport argparse\r\nimport autograsp\r\ndef main():\r\n        print(\"HelloPython!\")\r\nif __name__ == '__main__':\r\n        main()\r\n```\r\nThere is one variable, whether or not to use openmind/xvfb-fix/0.1 (which is required for the codebase I would actually be using this for). This appears to be the main issue:\r\n\r\nThe jobs running as xvfb-run -a singularity\r\n```\r\nTraceback (most recent call last):\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/test.py\", line 2, in <module>\r\n    import autograsp\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/autograsp.py\", line 21, in <module>\r\n    import bpyModule as bpym\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/bpyModule.py\", line 22, in <module>\r\n    from scipy import stats\r\n  File \"/usr/local/lib/python3.5/dist-packages/scipy/stats/__init__.py\", line 344, in <module>\r\n    from .stats import *\r\n  File \"/usr/local/lib/python3.5/dist-packages/scipy/stats/stats.py\", line 173, in <module>\r\n    import scipy.special as special\r\n  File \"/usr/local/lib/python3.5/dist-packages/scipy/special/__init__.py\", line 636, in <module>\r\n    from ._ufuncs import *\r\n```\r\n```\r\nTraceback (most recent call last):\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/test.py\", line 2, in <module>\r\n    import autograsp\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/autograsp.py\", line 21, in <module>\r\n    import bpyModule as bpym\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/bpyModule.py\", line 20, in <module>\r\n    import distances as dst\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/distances.py\", line 3, in <module>\r\n    from scipy.spatial.distance import cosine\r\nImportError: No module named 'scipy.spatial'\r\n```\r\n```\r\nTraceback (most recent call last):\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/test.py\", line 2, in <module>\r\n    import autograsp\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/autograsp.py\", line 21, in <module>\r\n    import bpyModule as bpym\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/bpyModule.py\", line 25, in <module>\r\n    import bpy\r\nImportError: /usr/lib/python3.5/site-packages/bpy.so: undefined symbol: \r\n```\r\n```\r\nERROR  : Failed to execv() /.exec: Accessing a corrupted shared library\r\nERROR  : Failed to execvp() /.exec: Accessing a corrupted shared library\r\nABORT  : Retval = 255\r\n```\r\n\r\nWhile running without xvfb results in a lower error rate some still exist:\r\n```\r\nInconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 517: elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed!\r\n```\r\n\r\n```\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/test.py\", line 2, in <module>\r\n    import autograsp\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/autograsp.py\", line 21, in <module>\r\n    import bpyModule as bpym\r\n  File \"/om/user/belledon/git/vision-touch-objects/py/bpyModule.py\", line 25, in <module>\r\n    import bpy\r\nImportError: /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.58.0: symbol , version GLIBCXX_3.4 not defined in file libstdc++.so.6 with link time reference\r\n```\r\n\r\nWhen comparing the singulartiy debug between the last error and a succesfull job (both without xvfb), I found these differences (I will refer to the successful job as passed and the failed job as \"failed\"). I used `diffuse` to find the differences in the two files. The following are all of the differences (after realignment).\r\n\r\nPassed:\r\n\r\n- Could not get exclusive lock:\r\n```\r\nVERBOSE [U=145570,P=44467] loop-control.c:74:singularity_loop_bind()  : Did not get exclusive lock on image loop device cache, assuming it is active\r\nDEBUG   [U=145570,P=44467] loop-control.c:76:singularity_loop_bind()  : Waiting to obtain shared lock on loop_dev lockfile\r\nDEBUG   [U=145570,P=44467] loop-control.c:79:singularity_loop_bind()  : Obtaining cached loop device name\r\nDEBUG   [U=145570,P=44467] util/file.c:346:filecat()                  : Called filecat(/tmp/.singularity-session-145570.-563112314.144116424523225713/image_loop_dev)\r\nDEBUG   [U=145570,P=44467] loop-control.c:85:singularity_loop_bind()  : Returning with active loop device name: /dev/loop0\r\nDEBUG   [U=145570,P=44467] privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=145570)\r\n```\r\nFailed\r\n- Did get exclusive lock:\r\n```\r\nDEBUG   [U=145570,P=16098] loop-control.c:71:singularity_loop_bind()  : Requesting exclusive flock() on loop_dev lockfile\r\n\r\nDEBUG   [U=145570,P=16098] loop-control.c:94:singularity_loop_bind()  : Calculating image offset\r\nVERBOSE [U=145570,P=16098] image-util.c:48:singularity_image_check()  : Checking file is a Singularity image\r\nDEBUG   [U=145570,P=16098] image-util.c:59:singularity_image_check()  : Checking if first line matches key\r\nVERBOSE [U=145570,P=16098] image-util.c:62:singularity_image_check()  : File is a valid Singularity image\r\nVERBOSE [U=145570,P=16098] image-util.c:82:singularity_image_offset() : Calculating image offset\r\nVERBOSE [U=145570,P=16098] image-util.c:91:singularity_image_offset() : Found image at an offset of 31 bytes\r\nDEBUG   [U=145570,P=16098] image-util.c:96:singularity_image_offset() : Returning image_offset(image_fp) = 31\r\nDEBUG   [U=145570,P=16098] privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=16098]      loop-control.c:101:singularity_loop_bind() : Finding next available loop device...\r\nVERBOSE [U=0,P=16098]      loop-control.c:133:singularity_loop_bind() : Found avaialble loop device: /dev/loop0\r\nDEBUG   [U=0,P=16098]      loop-control.c:135:singularity_loop_bind() : Setting loop device flags\r\nDEBUG   [U=0,P=16098]      privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=145570, GID=88895\r\nDEBUG   [U=145570,P=16098] privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID\r\nVERBOSE [U=145570,P=16098] loop-control.c:145:singularity_loop_bind() : Using loop device: /dev/loop0\r\nDEBUG   [U=145570,P=16098] loop-control.c:147:singularity_loop_bind() : Writing active loop device name (/dev/loop0) to loop file cache: /tmp/.singularity-session-145570.-563112314.144116424523225713/image_loop_dev\r\nDEBUG   [U=145570,P=16098] util/file.c:327:fileput()                  : Called fileput(/tmp/.singularity-session-145570.-563112314.144116424523225713/image_loop_dev, /dev/loop0)\r\nDEBUG   [U=145570,P=16098] loop-control.c:153:singularity_loop_bind() : Resetting exclusive flock() to shared on image_loop_file\r\n```\r\n\r\nFailed\r\n- During mounting, it creates a directory (while the passed file does not)\r\n```\r\nDEBUG   [U=145570,P=16098] mount-util.c:48:check_mounted()            : Iterating through /proc/mounts\r\nDEBUG   [U=145570,P=16098] util/file.c:245:s_mkpath()                 : Creating directory: /tmp/.singularity-session-145570.-563112314.144116424523225713/home\r\nDEBUG   [U=145570,P=16098] util/file.c:245:s_mkpath()                 : Creating directory: /tmp/.singularity-session-145570.-563112314.144116424523225713/home/belledon\r\n```\r\n\r\nThe only other difference is in this line:\r\nFailed:\r\n```\r\nDEBUG   [U=145570,P=16098] util/file.c:297:copy_file()                : Calling fstat() on source file descriptor: 10\r\n```\r\nWhere  passed has a descriptor of 9. \r\n\r\nI will be happy to post the full debug files. I am also doing the same analysis with jobs that ran with xvfb. "},"comment":{"url":"https://api.github.com/repos/singularityware/singularity/issues/comments/284269691","html_url":"https://github.com/singularityware/singularity/issues/494#issuecomment-284269691","issue_url":"https://api.github.com/repos/singularityware/singularity/issues/494","id":284269691,"user":{"login":"belledon","id":19876414,"avatar_url":"https://avatars1.githubusercontent.com/u/19876414?v=3","gravatar_id":"","url":"https://api.github.com/users/belledon","html_url":"https://github.com/belledon","followers_url":"https://api.github.com/users/belledon/followers","following_url":"https://api.github.com/users/belledon/following{/other_user}","gists_url":"https://api.github.com/users/belledon/gists{/gist_id}","starred_url":"https://api.github.com/users/belledon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/belledon/subscriptions","organizations_url":"https://api.github.com/users/belledon/orgs","repos_url":"https://api.github.com/users/belledon/repos","events_url":"https://api.github.com/users/belledon/events{/privacy}","received_events_url":"https://api.github.com/users/belledon/received_events","type":"User","site_admin":false},"created_at":"2017-03-05T22:44:27Z","updated_at":"2017-03-05T22:44:27Z","body":"```\r\nenabling debugging\r\nincreasing verbosity level (6)\r\nending argument loop\r\nExec'ing: /cm/shared/openmind/singularity/singularity-librefactor-1g397644e/libexec/singularity/cli/exec.exec -BVERBOSE [U=0,P=]      message_init()                            Set messagelevel to: 6\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Initialize configuration file: /cm/shared/openmind/singularity/singularity-librefactor-1g397644e/etc/singularity/singularity.conf\r\nDEBUG   [U=0,P=]      singularity_config_parse()                Starting parse of configuration file /cm/shared/openmind/singularity/singularity-librefactor-1g397644e/etc/singularity/singularity.conf\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key allow setuid = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key max loop devices = '256'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key allow pid ns = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key config passwd = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key config group = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key config resolv_conf = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key mount proc = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key mount sys = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key mount dev = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key mount home = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key mount tmp = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key mount hostfs = 'no'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key bind path = '/etc/hosts'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key user bind control = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key enable overlay = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key mount slave = 'yes'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key container dir = '/var/lib/singularity/mnt'\r\nVERBOSE [U=0,P=]      singularity_config_parse()                Got config key singularity user = 'singularity'\r\nDEBUG   [U=0,P=]      singularity_config_parse()                Finished parsing configuration file '/cm/shared/openmind/singularity/singularity-librefactor-1g397644e/etc/singularity/singularity.conf'\r\nVERBOSE [U=0,P=]      singularity_suid_init()                   Running SUID program workflow\r\nVERBOSE [U=0,P=]      singularity_suid_init()                   Checking program has appropriate permissions\r\nVERBOSE [U=0,P=]      singularity_suid_init()                   Checking configuration file is properly owned by root\r\nVERBOSE [U=0,P=]      singularity_suid_init()                   Checking if singularity.conf allows us to run as suid\r\nDEBUG   [U=0,P=]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow setuid, yes)\r\nDEBUG   [U=0,P=]      singularity_config_get_value_impl()       Returning configuration value allow setuid='yes'\r\nDEBUG   [U=0,P=]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow setuid, yes) = 1\r\nVERBOSE [U=0,P=]      singularity_suid_init()                   Checking if we were requested to run as NOSUID by user\r\nVERBOSE [U=0,P=]      singularity_registry_init()               Initializing Singularity Registry\r\nVERBOSE [U=0,P=]      singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/cm/shared/openmind/singularity/singularity-librefactor-1g397644e/libexec'\r\nDEBUG   [U=0,P=]      singularity_registry_set()                Returning singularity_registry_set(libexecdir, /cm/shared/openmind/singularity/singularity-librefactor-1g397644e/libexec) = 0\r\nVERBOSE [U=0,P=]      singularity_registry_set()                Adding value to registry: 'COMMAND' = 'exec'\r\nDEBUG   [U=0,P=]      singularity_registry_set()                Returning singularity_registry_set(COMMAND, exec) = 0\r\nVERBOSE [U=0,P=]      singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '6'\r\nDEBUG   [U=0,P=]      singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 6) = 0\r\nVERBOSE [U=0,P=]      singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/cm/shared/openmind/singularity/singularity-librefactor-1g397644e/var'\r\nDEBUG   [U=0,P=]      singularity_registry_set()                Returning singularity_registry_set(localstatedir, /cm/shared/openmind/singularity/singularity-librefactor-1g397644e/var) = 0\r\nVERBOSE [U=0,P=]      singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/cm/shared/openmind/singularity/singularity-librefactor-1g397644e/etc'\r\nDEBUG   [U=0,P=]      singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /cm/shared/openmind/singularity/singularity-librefactor-1g397644e/etc) = 0\r\nVERBOSE [U=0,P=]      singularity_registry_set()                Adding value to registry: 'BINDPATH' = '/om:/om,'\r\nDEBUG   [U=0,P=]      singularity_registry_set()                Returning singularity_registry_set(BINDPATH, /om:/om,) = 0\r\nVERBOSE [U=0,P=]      singularity_registry_set()                Adding value to registry: 'IMAGE' = '/om/user/belledon/singularity_test/test.img'\r\nDEBUG   [U=0,P=]      singularity_registry_set()                Returning singularity_registry_set(IMAGE, /om/user/belledon/singularity_test/test.img) = 0\r\nDEBUG   [U=0,P=]      singularity_priv_init()                   Initializing user info\r\nDEBUG   [U=0,P=]      singularity_priv_init()                   Set the calling user's username to: belledon\r\nDEBUG   [U=0,P=]      singularity_priv_init()                   Marking uinfo structure as ready\r\nDEBUG   [U=0,P=]      singularity_priv_init()                   Obtaining home directory\r\nVERBOSE [U=0,P=]      singularity_priv_init()                   Set home (via getpwuid()) to: /home/belledon\r\nVERBOSE [U=0,P=]      singularity_priv_userns()                 Invoking the user namespace\r\nDEBUG   [U=0,P=]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)\r\nDEBUG   [U=0,P=]      singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'\r\nDEBUG   [U=0,P=]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1\r\nVERBOSE [U=0,P=]      singularity_priv_userns()                 Not virtualizing USER namespace: running as SUID\r\nDEBUG   [U=0,P=]      singularity_priv_userns()                 Returning singularity_priv_init(void)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       No configuration entry found for 'sessiondir prefix'; returning default value '/tmp/.singularity-session-'\r\nDEBUG   [U=145570,P=] singularity_sessiondir()                  Got sessiondir from configuration: '/tmp/.singularity-session-'\r\nVERBOSE [U=145570,P=] singularity_registry_set()                Adding value to registry: 'SESSIONDIR' = '/tmp/.singularity-session-'\r\nDEBUG   [U=145570,P=] singularity_registry_set()                Returning singularity_registry_set(SESSIONDIR, /tmp/.singularity-session-) = 0\r\nVERBOSE [U=145570,P=] singularity_sessiondir()                  Creating session directory: /tmp/.singularity-session-\r\nDEBUG   [U=145570,P=] s_mkpath()                                Creating directory: /tmp/.singularity-session-\r\nDEBUG   [U=145570,P=] singularity_sessiondir()                  Opening sessiondir file descriptor\r\nDEBUG   [U=145570,P=] singularity_sessiondir()                  Setting shared flock() on session directory\r\nVERBOSE [U=145570,P=] message_init()                            Set messagelevel to: 6\r\nDEBUG   [U=145570,P=] envar_path()                              Checking environment variable is valid path: 'SINGULARITY_CLEANDIR'\r\nVERBOSE [U=145570,P=] envar()                                   Checking input from environment: 'SINGULARITY_CLEANDIR'\r\nDEBUG   [U=145570,P=] envar()                                   Checking environment variable is defined: SINGULARITY_CLEANDIR\r\nDEBUG   [U=145570,P=] envar()                                   Checking environment variable length (<= 4096): SINGULARITY_CLEANDIR\r\nDEBUG   [U=145570,P=] envar()                                   Checking environment variable has allowed characters: SINGULARITY_CLEANDIR\r\nVERBOSE [U=145570,P=] envar()                                   Obtained input from environment 'SINGULARITY_CLEANDIR' = '/tmp/.singularity-session-'\r\nDEBUG   [U=145570,P=] main()                                    Starting cleanup process\r\nDEBUG   [U=145570,P=] main()                                    Opening cleandir file descriptor\r\nDEBUG   [U=145570,P=] main()                                    Opening cleandir_lock file descriptor\r\nVERBOSE [U=145570,P=] main()                                    Daemonizing cleandir cleanup process\r\nDEBUG   [U=145570,P=] singularity_registry_get()                Retriving value from registry: 'IMAGE' = '/om/user/belledon/singularity_test/test.img'\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       No configuration entry found for 'limit container owners'; returning default value 'NULL'\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       No configuration entry found for 'limit container paths'; returning default value 'NULL'\r\nDEBUG   [U=145570,P=] singularity_image_open()                  Opening file descriptor to image: /om/user/belledon/singularity_test/test.img\r\nDEBUG   [U=145570,P=] singularity_registry_get()                Retriving value from registry: 'SESSIONDIR' = '/tmp/.singularity-session-'\r\nDEBUG   [U=145570,P=] singularity_runtime_ns()                  Calling: _singularity_runtime_ns_pid()\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow pid ns, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value allow pid ns='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow pid ns, yes) = 1\r\nVERBOSE [U=145570,P=] singularity_runtime_ns_pid()              Not virtualizing PID namespace on user request\r\nDEBUG   [U=145570,P=] singularity_runtime_ns()                  Calling: _singularity_runtime_ns_ipc()\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow ipc ns, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       No configuration entry found for 'allow ipc ns'; returning default value 'yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow ipc ns, yes) = 1\r\nVERBOSE [U=145570,P=] singularity_runtime_ns_ipc()              Not virtualizing IPC namespace on user request\r\nDEBUG   [U=145570,P=] singularity_runtime_ns()                  Calling: _singularity_runtime_ns_mnt()\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount slave, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount slave='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount slave, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nDEBUG   [U=0,P=]      singularity_runtime_ns_mnt()              Virtualizing FS namespace\r\nDEBUG   [U=0,P=]      singularity_runtime_ns_mnt()              Virtualizing mount namespace\r\nDEBUG   [U=0,P=]      singularity_runtime_ns_mnt()              Making mounts slave\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value max loop devices='256'\r\nDEBUG   [U=145570,P=] singularity_image_bind()                  Entered singularity_image_bind()\r\nDEBUG   [U=145570,P=] singularity_image_bind()                  Converting max_loop_devs_string to int: '256'\r\nDEBUG   [U=145570,P=] singularity_image_bind()                  Converted max_loop_devs_string to int: '256' -> 256\r\nDEBUG   [U=145570,P=] singularity_image_bind()                  Checking if this image has been properly opened\r\nDEBUG   [U=145570,P=] singularity_image_bind()                  Checking if image is valid file\r\nDEBUG   [U=145570,P=] singularity_image_bind()                  Checking if image is already bound to a loop device\r\nDEBUG   [U=145570,P=] singularity_image_bind()                  Calculating image offset\r\nVERBOSE [U=145570,P=] singularity_image_check()                 Checking that file pointer is a Singularity image\r\nDEBUG   [U=145570,P=] singularity_image_check()                 First line of image(fd=5): #!/usr/bin/env run-singularity\r\n\r\nDEBUG   [U=145570,P=] singularity_image_check()                 Checking if first line matches key\r\nVERBOSE [U=145570,P=] singularity_image_check()                 File is a valid Singularity image\r\nVERBOSE [U=145570,P=] singularity_image_offset()                Calculating image offset\r\nVERBOSE [U=145570,P=] singularity_image_offset()                Found image at an offset of 31 bytes\r\nDEBUG   [U=145570,P=] singularity_image_offset()                Returning image_offset(image_fp) = 31\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nDEBUG   [U=0,P=]      singularity_image_bind()                  Finding next available loop device...\r\nVERBOSE [U=0,P=]      singularity_image_bind()                  Found available loop device: /dev/loop1\r\nDEBUG   [U=0,P=]      singularity_image_bind()                  Setting loop device flags\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nVERBOSE [U=145570,P=] singularity_image_bind()                  Using loop device: /dev/loop1\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value container dir='/var/lib/singularity/mnt'\r\nDEBUG   [U=145570,P=] singularity_runtime_rootfs()              Setting container directory to: /var/lib/singularity/mnt/source\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nDEBUG   [U=0,P=]      singularity_runtime_rootfs()              Creating top level source mount directory to: /var/lib/singularity/mnt/source\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nVERBOSE [U=145570,P=] singularity_image_mount()                 Checking what kind of image we are mounting\r\nDEBUG   [U=145570,P=] singularity_image_mount_squashfs_check()  Image does not appear to be of type '.sqsh': /om/user/belledon/singularity_test/test.img\r\nDEBUG   [U=145570,P=] singularity_image_mount_dir_check()       Checking if image is a directory\r\nVERBOSE [U=145570,P=] singularity_image_mount_dir_check()       Source path is not a directory: /om/user/belledon/singularity_test/test.img\r\nVERBOSE [U=145570,P=] singularity_image_mount()                 Attempting to mount as singularity image\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_image_mount_image_mount()     Mounting /dev/loop1 in read/only to: /var/lib/singularity/mnt/source\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] action_ready()                            Checking if container is valid at: /var/lib/singularity/mnt/source\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value container dir='/var/lib/singularity/mnt'\r\nDEBUG   [U=145570,P=] singularity_runtime_overlayfs()           Checking if overlayfs should be used\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(enable overlay, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value enable overlay='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(enable overlay, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_runtime_overlayfs()           OverlayFS enabled by host build\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nDEBUG   [U=0,P=]      singularity_runtime_overlayfs()           Creating top level overlay mount directory: /var/lib/singularity/mnt/overlay\r\nDEBUG   [U=0,P=]      singularity_runtime_overlayfs()           Mounting overlay tmpfs: /var/lib/singularity/mnt/overlay\r\nDEBUG   [U=0,P=]      singularity_runtime_overlayfs()           Creating upper overlay directory: /var/lib/singularity/mnt/overlay/upper\r\nDEBUG   [U=0,P=]      s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay/upper\r\nDEBUG   [U=0,P=]      singularity_runtime_overlayfs()           Creating overlay work directory: /var/lib/singularity/mnt/overlay/work\r\nDEBUG   [U=0,P=]      s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay/work\r\nDEBUG   [U=0,P=]      singularity_runtime_overlayfs()           Creating mount_final directory: /var/lib/singularity/mnt/final\r\nVERBOSE [U=0,P=]      singularity_runtime_overlayfs()           Mounting overlay with options: lowerdir=/var/lib/singularity/mnt/source,upperdir=/var/lib/singularity/mnt/overlay/upper,workdir=/var/lib/singularity/mnt/overlay/work\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nVERBOSE [U=145570,P=] singularity_registry_set()                Adding value to registry: 'OVERLAYFS_ENABLED' = '1'\r\nDEBUG   [U=145570,P=] singularity_registry_set()                Returning singularity_registry_set(OVERLAYFS_ENABLED, 1) = 0\r\nVERBOSE [U=145570,P=] singularity_runtime_overlayfs()           Updating the containerdir to: /var/lib/singularity/mnt/final\r\nVERBOSE [U=145570,P=] singularity_runtime_mounts()              Running all mount components\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount hostfs, no)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount hostfs='no'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount hostfs, no) = 0\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_hostfs()        Not mounting host file systems per configuration\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_binds()         Checking configuration file for 'bind path'\r\nVERBOSE [U=145570,P=] singularity_runtime_mount_binds()         Found 'bind path' = /etc/hosts, /etc/hosts\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_binds()         Checking if bind point is already mounted: /etc/hosts\r\nDEBUG   [U=145570,P=] check_mounted()                           Opening /proc/mounts\r\nDEBUG   [U=145570,P=] check_mounted()                           Iterating through /proc/mounts\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_binds()         Binding '/etc/hosts' to '/var/lib/singularity/mnt/final//etc/hosts'\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount proc'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount proc, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount proc='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount proc, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_kernelfs()      Mounting /proc\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount sys'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount sys, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount sys='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount sys, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_kernelfs()      Mounting /sys\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount dev='yes'\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_dev()           Checking configuration file for 'mount dev'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount dev, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount dev='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount dev, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_dev()           Bind mounting /dev\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount tmp, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount tmp='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount tmp, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_tmp()           Mounting directory: /tmp\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_tmp()           Mounting directory: /var/tmp\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount home, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value mount home='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount home, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Checking that home directry is configured: /home/belledon\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Identifying the base directory of homedir: /home/belledon\r\nDEBUG   [U=145570,P=] basedir()                                 Obtaining basedir for: /home/belledon\r\nDEBUG   [U=145570,P=] basedir()                                 Iterating basedir: /home/belledon\r\nDEBUG   [U=145570,P=] basedir()                                 Iterating basedir: /home\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Checking if home directory is already mounted: /home/belledon\r\nDEBUG   [U=145570,P=] check_mounted()                           Opening /proc/mounts\r\nDEBUG   [U=145570,P=] check_mounted()                           Iterating through /proc/mounts\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Creating directory to stage tmpdir home: /tmp/.singularity-session-/home/belledon\r\nDEBUG   [U=145570,P=] s_mkpath()                                Creating directory: /tmp/.singularity-session-/home\r\nDEBUG   [U=145570,P=] s_mkpath()                                Creating directory: /tmp/.singularity-session-/home/belledon\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Configuring the source of the home directory\r\nVERBOSE [U=145570,P=] singularity_runtime_mount_home()          Setting home directory source from singularity_priv_homedir()\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Set home directory source to: /home/belledon\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Checking to make sure that the home directory exists: /home/belledon\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_home()          Checking ownership of physical home directory: /home/belledon\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_home()          Mounting home directory source to stage: /home/belledon->/tmp/.singularity-session-/home/belledon\r\nDEBUG   [U=0,P=]      singularity_runtime_mount_home()          Remounting home directory with necessary options: /home/belledon\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_home()          Mounting home directory base into container: /tmp/.singularity-session-/home->/var/lib/singularity/mnt/final/home\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_userbinds()     Checking for environment variable 'SINGULARITY_BINDPATH'\r\nDEBUG   [U=145570,P=] singularity_registry_get()                Retriving value from registry: 'BINDPATH' = '/om:/om,'\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_userbinds()     Checking for 'user bind control' in config\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(user bind control, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value user bind control='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(user bind control, yes) = 1\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_userbinds()     Parsing SINGULARITY_BINDPATH for user-specified bind mounts.\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_userbinds()     Found bind: /om -> container:/om\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_userbinds()     Checking if bind point is already mounted: /om\r\nDEBUG   [U=145570,P=] check_mounted()                           Opening /proc/mounts\r\nDEBUG   [U=145570,P=] check_mounted()                           Iterating through /proc/mounts\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_mount_userbinds()     Binding '/om' to '/var/lib/singularity/mnt/final//om'\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_userbinds()     Unsetting environment variable 'SINGULARITY_BINDPATH'\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_scratch()       Getting SINGULARITY_SCRATCHDIR from environment\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_scratch()       Not mounting scratch directory: Not requested\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_cwd()           Checking to see if we should mount current working directory\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_cwd()           Getting current working directory\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_cwd()           Checking if current directory exists in container\r\nDEBUG   [U=145570,P=] file_devino()                             Called file_devino(/om/user/belledon/twilight_zone)\r\nDEBUG   [U=145570,P=] file_devino()                             Returning file_devino(/om/user/belledon/twilight_zone) = -563112314.144116535219193287\r\nDEBUG   [U=145570,P=] file_devino()                             Called file_devino(/var/lib/singularity/mnt/final/om/user/belledon/twilight_zone)\r\nDEBUG   [U=145570,P=] file_devino()                             Returning file_devino(/var/lib/singularity/mnt/final/om/user/belledon/twilight_zone) = -563112314.144116535219193287\r\nDEBUG   [U=145570,P=] singularity_runtime_mount_cwd()           Checking if container's cwd == host's cwd\r\nVERBOSE [U=145570,P=] singularity_runtime_mount_cwd()           Not mounting current directory: location already available within container\r\nVERBOSE [U=145570,P=] singularity_runtime_files()               Running file components\r\nDEBUG   [U=145570,P=] singularity_runtime_files_passwd()        Called singularity_file_passwd_create()\r\nDEBUG   [U=145570,P=] singularity_runtime_files_passwd()        Checking configuration option: 'config passwd'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config passwd, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value config passwd='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config passwd, yes) = 1\r\nVERBOSE [U=145570,P=] singularity_runtime_files_passwd()        Checking for template passwd file: /var/lib/singularity/mnt/final/etc/passwd\r\nVERBOSE [U=145570,P=] singularity_runtime_files_passwd()        Creating template of /etc/passwd\r\nDEBUG   [U=145570,P=] copy_file()                               Called copy_file(/var/lib/singularity/mnt/final/etc/passwd, /tmp/.singularity-session-/passwd)\r\nDEBUG   [U=145570,P=] copy_file()                               Opening source file: /var/lib/singularity/mnt/final/etc/passwd\r\nDEBUG   [U=145570,P=] copy_file()                               Opening destination file: /tmp/.singularity-session-/passwd\r\nDEBUG   [U=145570,P=] copy_file()                               Calling fstat() on source file descriptor: 7\r\nDEBUG   [U=145570,P=] copy_file()                               Cloning permission string of source to dest\r\nDEBUG   [U=145570,P=] copy_file()                               Copying file data...\r\nDEBUG   [U=145570,P=] copy_file()                               Done copying data, closing file pointers\r\nDEBUG   [U=145570,P=] copy_file()                               Returning copy_file(/var/lib/singularity/mnt/final/etc/passwd, /tmp/.singularity-session-/passwd) = 0\r\nVERBOSE [U=145570,P=] singularity_runtime_files_passwd()        Creating template passwd file and appending user data: /tmp/.singularity-session-/passwd\r\nDEBUG   [U=145570,P=] container_file_bind()                     Called file_bind(/tmp/.singularity-session-/passwd, /etc/passwd()\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      container_file_bind()                     Binding file '/tmp/.singularity-session-/passwd' to '/var/lib/singularity/mnt/final/etc/passwd'\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_runtime_files_group()         Called singularity_file_group_create()\r\nDEBUG   [U=145570,P=] singularity_runtime_files_group()         Checking configuration option: 'config group'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config group, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value config group='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config group, yes) = 1\r\nVERBOSE [U=145570,P=] singularity_runtime_files_group()         Creating template of /etc/group for containment\r\nDEBUG   [U=145570,P=] copy_file()                               Called copy_file(/var/lib/singularity/mnt/final/etc/group, /tmp/.singularity-session-/group)\r\nDEBUG   [U=145570,P=] copy_file()                               Opening source file: /var/lib/singularity/mnt/final/etc/group\r\nDEBUG   [U=145570,P=] copy_file()                               Opening destination file: /tmp/.singularity-session-/group\r\nDEBUG   [U=145570,P=] copy_file()                               Calling fstat() on source file descriptor: 7\r\nDEBUG   [U=145570,P=] copy_file()                               Cloning permission string of source to dest\r\nDEBUG   [U=145570,P=] copy_file()                               Copying file data...\r\nDEBUG   [U=145570,P=] copy_file()                               Done copying data, closing file pointers\r\nDEBUG   [U=145570,P=] copy_file()                               Returning copy_file(/var/lib/singularity/mnt/final/etc/group, /tmp/.singularity-session-/group) = 0\r\nVERBOSE [U=145570,P=] singularity_runtime_files_group()         Updating group file with user info\r\nDEBUG   [U=145570,P=] singularity_runtime_files_group()         Getting supplementary group info\r\nVERBOSE [U=145570,P=] singularity_runtime_files_group()         Found supplementary group membership in: 1002\r\nVERBOSE [U=145570,P=] singularity_runtime_files_group()         Adding user's supplementary group ('om_user') info to template group file\r\nDEBUG   [U=145570,P=] singularity_runtime_files_group()         Skipping duplicate supplementary group\r\nDEBUG   [U=145570,P=] container_file_bind()                     Called file_bind(/tmp/.singularity-session-/group, /etc/group()\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      container_file_bind()                     Binding file '/tmp/.singularity-session-/group' to '/var/lib/singularity/mnt/final/etc/group'\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_runtime_files_resolvconf()    Checking configuration option\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config resolv_conf, yes)\r\nDEBUG   [U=145570,P=] singularity_config_get_value_impl()       Returning configuration value config resolv_conf='yes'\r\nDEBUG   [U=145570,P=] singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config resolv_conf, yes) = 1\r\nDEBUG   [U=145570,P=] container_file_bind()                     Called file_bind(/etc/resolv.conf, /etc/resolv.conf()\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      container_file_bind()                     Binding file '/etc/resolv.conf' to '/var/lib/singularity/mnt/final/etc/resolv.conf'\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nVERBOSE [U=145570,P=] singularity_runtime_enter()               Containing all rootfs components\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nVERBOSE [U=0,P=]      singularity_runtime_enter_chroot()        Entering container file system root: /var/lib/singularity/mnt/final\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Dropping privileges to UID=145570, GID=88895 (2 supplementary GIDs)\r\nDEBUG   [U=0,P=]      singularity_priv_drop()                   Restoring supplementary groups\r\nDEBUG   [U=145570,P=] singularity_priv_drop()                   Confirming we have correct UID/GID\r\nDEBUG   [U=145570,P=] singularity_runtime_enter_chroot()        Changing dir to '/' within the new root\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_NODELIST\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_CHECKPOINT_IMAGE_DIR\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: XDG_SESSION_ID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: HOSTNAME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_TOPOLOGY_ADDR\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURMD_NODENAME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_PRIO_PROCESS\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_NODE_ALIASES\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: TERM\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SHELL\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: HISTSIZE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_QOS\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SSH_CLIENT\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_TOPOLOGY_ADDR_PATTERN\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NCARG_FONTCAPS\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: QTDIR\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: QTINC\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SSH_TTY\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: GRASPIT\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_libexecdir\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_COMMAND\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: QT_GRAPHICSSYSTEM_CHECKED\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: USER\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_NNODES\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_MESSAGELEVEL\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: LS_COLORS\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOBID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NCARG_GRAPHCAPS\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: TMUX\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: PATH\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: MAIL\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_TASKS_PER_NODE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_localstatedir\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_CPUS_PER_TASK\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_ID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NEW_HOME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: PWD\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_USER\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NCARG_ROOT\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: CUDA_VISIBLE_DEVICES\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: _LMFILES_\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: LANG\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NCARG_DATABASE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: MODULEPATH\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NUM_CORES\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_UMASK\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_UID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: LOADEDMODULES\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: TMUX_PANE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_NODEID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_SUBMIT_DIR\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_TASK_PID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_CPUS_ON_NODE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: RETVAL\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SSH_ASKPASS\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NCARG_LIB\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: KRB5CCNAME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: HISTCONTROL\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: ENVIRONMENT\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_PROCID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_NODELIST\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SHLVL\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: NCARG_NCARG\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: HOME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_LOCALID\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_CLUSTER_NAME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_CPUS_PER_NODE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_SUBMIT_HOST\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_GTIDS\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_PARTITION\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_sysconfdir\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: LOGNAME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_PROFILE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: QTLIB\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: CVS_RSH\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SSH_CONNECTION\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_ACCOUNT\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: GPU_DEVICE_ORDINAL\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: MODULESHOME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_JOB_NUM_NODES\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: LESSOPEN\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: DISPLAY\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: XDG_RUNTIME_DIR\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: CHANGE_HOME\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_IMAGE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: HISTFILE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: SLURM_MEM_PER_NODE\r\nDEBUG   [U=145570,P=] singularity_runtime_environment()         Leaving environment variable set: XAUTHORITY\r\nDEBUG   [U=145570,P=] singularity_priv_drop_perm()              Called singularity_priv_drop_perm(void)\r\nDEBUG   [U=145570,P=] singularity_priv_drop_perm()              Escalating permissison so we can properly drop permission\r\nDEBUG   [U=145570,P=] singularity_priv_escalate()               Temporarily escalating privileges (U=145570)\r\nDEBUG   [U=0,P=]      singularity_priv_escalate()               Clearing supplementary GIDs.\r\nDEBUG   [U=0,P=]      singularity_priv_drop_perm()              Resetting supplementary groups\r\nDEBUG   [U=0,P=]      singularity_priv_drop_perm()              Dropping to group ID '88895'\r\nDEBUG   [U=0,P=]      singularity_priv_drop_perm()              Dropping real and effective privileges to GID = '88895'\r\nDEBUG   [U=0,P=]      singularity_priv_drop_perm()              Dropping real and effective privileges to UID = '145570'\r\nDEBUG   [U=145570,P=] singularity_priv_drop_perm()              Confirming we have correct GID\r\nDEBUG   [U=145570,P=] singularity_priv_drop_perm()              Confirming we have correct UID\r\nDEBUG   [U=145570,P=] singularity_priv_drop_perm()              Setting NO_NEW_PRIVS to prevent future privilege escalations.\r\nDEBUG   [U=145570,P=] singularity_priv_drop_perm()              Finished dropping privileges\r\nDEBUG   [U=145570,P=] singularity_registry_get()                Retriving value from registry: 'COMMAND' = 'exec'\r\nVERBOSE [U=145570,P=] action_exec()                             Exec'ing /.exec\r\nHelloPython!\r\n```"}}
{"actor":{"display_login":"belledon"}}