{"action":"created","issue":{"url":"https://api.github.com/repos/singularityware/singularity/issues/39","repository_url":"https://api.github.com/repos/singularityware/singularity","labels_url":"https://api.github.com/repos/singularityware/singularity/issues/39/labels{/name}","comments_url":"https://api.github.com/repos/singularityware/singularity/issues/39/comments","events_url":"https://api.github.com/repos/singularityware/singularity/issues/39/events","html_url":"https://github.com/singularityware/singularity/issues/39","id":157755537,"number":39,"title":"strange mount/unmount behavior","user":{"login":"kellyrowland","id":5440574,"avatar_url":"https://avatars3.githubusercontent.com/u/5440574?v=3","gravatar_id":"","url":"https://api.github.com/users/kellyrowland","html_url":"https://github.com/kellyrowland","followers_url":"https://api.github.com/users/kellyrowland/followers","following_url":"https://api.github.com/users/kellyrowland/following{/other_user}","gists_url":"https://api.github.com/users/kellyrowland/gists{/gist_id}","starred_url":"https://api.github.com/users/kellyrowland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kellyrowland/subscriptions","organizations_url":"https://api.github.com/users/kellyrowland/orgs","repos_url":"https://api.github.com/users/kellyrowland/repos","events_url":"https://api.github.com/users/kellyrowland/events{/privacy}","received_events_url":"https://api.github.com/users/kellyrowland/received_events","type":"User","site_admin":false},"labels":[{"id":271060106,"url":"https://api.github.com/repos/singularityware/singularity/labels/help%20wanted","name":"help wanted","color":"159818","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2016-05-31T19:53:04Z","updated_at":"2017-05-10T02:32:48Z","closed_at":null,"body":"seen in setup meeting on 5/31/16 on ubuntu 14.\n"},"comment":{"url":"https://api.github.com/repos/singularityware/singularity/issues/comments/300356717","html_url":"https://github.com/singularityware/singularity/issues/39#issuecomment-300356717","issue_url":"https://api.github.com/repos/singularityware/singularity/issues/39","id":300356717,"user":{"login":"thiell","id":1494929,"avatar_url":"https://avatars2.githubusercontent.com/u/1494929?v=3","gravatar_id":"","url":"https://api.github.com/users/thiell","html_url":"https://github.com/thiell","followers_url":"https://api.github.com/users/thiell/followers","following_url":"https://api.github.com/users/thiell/following{/other_user}","gists_url":"https://api.github.com/users/thiell/gists{/gist_id}","starred_url":"https://api.github.com/users/thiell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thiell/subscriptions","organizations_url":"https://api.github.com/users/thiell/orgs","repos_url":"https://api.github.com/users/thiell/repos","events_url":"https://api.github.com/users/thiell/events{/privacy}","received_events_url":"https://api.github.com/users/thiell/received_events","type":"User","site_admin":false},"created_at":"2017-05-10T02:32:47Z","updated_at":"2017-05-10T02:32:47Z","body":"Hi @gmkurtzer and sorry for the delay!!\r\n\r\nThanks for adding `--nv`! I tried it (using 2.2.99-development.g945c6ee) and played a bit with  `docker://tensorflow/tensorflow:latest-gpu`...\r\n\r\nThe good news is that it seems to work well in jobs that allocate a single GPU (in slurm, so a single GPU device is visible - note: we're using device cgroup).\r\n\r\nExample (single GPU allocation):\r\n```\r\nSingularity tensorflow:latest-gpu:~> python\r\nPython 2.7.12 (default, Nov 19 2016, 06:48:10) \r\n[GCC 5.4.0 20160609] on linux2\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import tensorflow as tf\r\n>>> with tf.device('/gpu:0'):\r\n...     a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')\r\n...     b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')\r\n...     c = tf.matmul(a, b)\r\n... \r\n>>> sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))\r\n2017-05-10 02:23:25.640377: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-05-10 02:23:25.640407: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-05-10 02:23:25.640416: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-05-10 02:23:26.163631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: \r\nname: Tesla K80\r\nmajor: 3 minor: 7 memoryClockRate (GHz) 0.8235\r\npciBusID 0000:85:00.0\r\nTotal memory: 11.17GiB\r\nFree memory: 11.11GiB\r\n2017-05-10 02:23:26.163686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 \r\n2017-05-10 02:23:26.163696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y \r\n2017-05-10 02:23:26.163712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:85:00.0)\r\nDevice mapping:\r\n/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: Tesla K80, pci bus id: 0000:85:00.0\r\n2017-05-10 02:23:26.227264: I tensorflow/core/common_runtime/direct_session.cc:257] Device mapping:\r\n/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: Tesla K80, pci bus id: 0000:85:00.0\r\n\r\n>>> print(sess.run(c))\r\nMatMul: (MatMul): /job:localhost/replica:0/task:0/gpu:0\r\n2017-05-10 02:23:47.507606: I tensorflow/core/common_runtime/simple_placer.cc:841] MatMul: (MatMul)/job:localhost/replica:0/task:0/gpu:0\r\nb: (Const): /job:localhost/replica:0/task:0/gpu:0\r\n2017-05-10 02:23:47.507659: I tensorflow/core/common_runtime/simple_placer.cc:841] b: (Const)/job:localhost/replica:0/task:0/gpu:0\r\na: (Const): /job:localhost/replica:0/task:0/gpu:0\r\n2017-05-10 02:23:47.507679: I tensorflow/core/common_runtime/simple_placer.cc:841] a: (Const)/job:localhost/replica:0/task:0/gpu:0\r\n[[ 22.  28.]\r\n [ 49.  64.]]\r\n```\r\n\r\nHowever, for jobs using multiple GPUs, even when effectively using only one GPU in TensorFlow, I get the following error messages:\r\n\r\n```\r\nSingularity tensorflow:latest-gpu:~> python\r\nPython 2.7.12 (default, Nov 19 2016, 06:48:10) \r\n[GCC 5.4.0 20160609] on linux2\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import tensorflow as tf\r\n>>> with tf.device('/gpu:0'):\r\n...     a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')\r\n...     b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')\r\n...     c = tf.matmul(a, b)\r\n... \r\n>>> sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))\r\n2017-05-10 02:27:16.126277: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-05-10 02:27:16.126330: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-05-10 02:27:16.126353: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-05-10 02:27:16.139414: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_UNKNOWN\r\n2017-05-10 02:27:16.139496: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: xs-0007\r\n2017-05-10 02:27:16.139523: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: xs-0007\r\n2017-05-10 02:27:16.139594: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: Invalid argument: expected %d.%d or %d.%d.%d form for driver version; got \"1\"\r\n2017-05-10 02:27:16.139657: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: \"\"\"NVRM version: NVIDIA UNIX x86_64 Kernel Module  361.93.03  Tue Sep 27 22:40:25 PDT 2016\r\nGCC version:  gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) \r\n\"\"\"\r\n2017-05-10 02:27:16.139718: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 361.93.3\r\nDevice mapping: no known devices.\r\n2017-05-10 02:27:16.143122: I tensorflow/core/common_runtime/direct_session.cc:257] Device mapping:\r\n```\r\n\r\nI tried with several different GPU indexes, same issue."}}
{"actor":{"display_login":"thiell"}}