title: PyTorch with CUDA and Nvidia card: RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable, but torch.cuda.is_available() is True
body: <h2>Problem:</h2>
  <p>I occasionally get the following CUDA error when running PyTorch scripts with CUDA on an Nvidia GPU, running on CentOS 7.</p>
  <p>If I run:</p>
  <pre><code>python3 -c 'import torch; print(torch.cuda.is_available()); torch.randn(1).to(&quot;cuda&quot;)'​
  </code></pre>
  <p>I get the following output:</p>
  <pre><code>True​
  Traceback (most recent call last):​
    File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;​
  RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable​
  CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.​
  For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
  </code></pre>
  <p>PyTorch seems to think the GPU is available, but I can't put anything onto it's memory. When I restart the computer, the error goes away. I can't seem to get the error to come back consistently.</p>