title: Pytorch C++ (Libtroch), using inter-op parallelism
body: <p>I am working on a machine learning system using the C++ API of PyTorch (<code>libtorch</code>).</p>
  <p>One thing that I have been recently working on is researching performance, CPU utilization and GPU usage of <code>libtorch</code>. Trough my research I understand that Torch utilizes two ways of parallelization on CPUs:</p>
  <ul>
  <li><code>inter-op</code> parallelization</li>
  <li><code>intra-op</code> parallelization</li>
  </ul>
  <p><strong>My main questions are</strong>:</p>
  <ul>
  <li>difference between these two</li>
  <li>how can I utilize <code>inter-op</code> parallelism</li>
  </ul>
  <p>I know that I can specify the number of threads used for <code>intra-op</code> parallelism (which from my understanding is performed using the <code>openmp</code> backend) using the <code>torch::set_num_threads()</code> function, as I monitor the performance of my models, I can see clearly that it utilizes the number of threads I specify using this function, and I can see clear performance difference by changing the number of <code>intra-op</code> threads.</p>
  <p>There is also another function <code>torch::set_num_interop_threads()</code>, but it seems that no matter how many interop threads I specify, I never see any difference in performance.</p>
  <p>Now I have read <a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html" rel="nofollow noreferrer">this PyTorch documentation article</a> but it is still is unclear to me how to utilize the inter op thread pool.</p>
  <p>The docs say:</p>
  <blockquote>
  <p>PyTorch uses a single thread pool for the inter-op parallelism, this thread pool is shared by all inference tasks that are forked within the application process.</p>
  </blockquote>
  <p><strong>I have two questions to this part:</strong></p>
  <ul>
  <li>do I need to create new threads myself to utilize the <code>interop</code> threads, or does torch do it somehow for me internally?</li>
  <li>If I need to create new threads myself, how do I do it in C++, so that I create a new thread form the <code>interop</code> thread pool?</li>
  </ul>
  <p>In <code>python</code> example they use a <code>fork</code> function from <code>torch.jit</code> module, but I cant find anything similar in the C++ API.</p>