title: My python app using tensorflow often hangs
body: <p>My app using TensorFlow often hangs without a specific condition.</p>
  <p>When stopped, there is one child process of my app spawned, and this child process seems to have something to do with ptxas.</p>
  <p>Because if I kill this process, I get the following error message.</p>
  <pre><code>2022-01-28 18:27:50.164615: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:70] Can't find ptxas binary in ${CUDA_DIR}/bin.  Will back to the GPU driver for PTX -&gt; sass compilation.  This is OK so long as you don't see a warning below about an out-of-date driver version.
  2022-01-28 18:27:50.164665: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:71] Searched for CUDA in the following directories:
  2022-01-28 18:27:50.164675: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:74]   ./cuda_sdk_lib
  2022-01-28 18:27:50.164680: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:74]   /usr/local/cuda-10.1
  2022-01-28 18:27:50.164685: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:74]   .
  2022-01-28 18:27:50.164689: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:76] You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
  </code></pre>
  <p>If I don't kill the child process, my app keeps hanging with the error message below.</p>
  <pre><code>2022-01-28 18:26:53.379479: E tensorflow/compiler/xla/service/slow_operation_alarm.cc:55] 
  ********************************
  Very slow compile?  If you want to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
  ********************************
  </code></pre>
  <p>The important thing is that this doesn't always happen when I launch the app.</p>
  <p>Besides, this app works well in different environments (OS, H/W, etc.).</p>
  <p>So, i think it seems to be a problem like a conflict with the OS and something rather than a simple bug.</p>
  <p>Below is the environment we are currently testing.</p>
  <p>OS: RHEL 7.9 <strong>(I have to use RHEL.)</strong></p>
  <p>GPU: nvidia Tesla T4 x 1 (Azure Standard_NC16as_T4_v3)</p>
  <p>Python 3.7.7</p>
  <p>tensorflow 2.1 <strong>(can't be changed)</strong></p>
  <p>cuda 10.1</p>
  <p>CuDNN 7.6.5</p>