title: Detect memory swapping in Python
body: <p>How to detect when OS starts swapping some resources of the running process to disk?</p>
  <p>I came here from basically the same <a href="https://stackoverflow.com/questions/37760854/detect-swapping-in-python">question</a>. The <a href="https://psutil.readthedocs.io/en/latest/" rel="nofollow noreferrer"><code>psutil</code></a> library is obviously great and provides a lot of information, yet, I don't know how to use it to solve the problem.</p>
  <p>I created a simple test script.</p>
  <pre class="lang-py prettyprint-override"><code>import psutil
  import os
  import numpy as np
  
  MAX = 45000
  
  ar = []
  pr_mem = []
  swap_mem = []
  virt_mem = []
  
  process = psutil.Process()
  
  for i in range(MAX):
      ar.append(np.zeros(100_000))
      pr_mem.append(process.memory_info())
      swap_mem.append(psutil.swap_memory())
      virt_mem.append(psutil.virtual_memory())
  </code></pre>
  <p>Then, I plotted the course of those statistics.</p>
  <pre class="lang-py prettyprint-override"><code>import matplotlib.pyplot as plt
  
  plt.figure(figsize=(16,12))
  
  plt.plot([x.rss for x in pr_mem], label='Resident Set Size')
  plt.plot([x.vms for x in pr_mem], label='Virtual Memory Size')
  plt.plot([x.available for x in virt_mem], label='Available Memory')
  plt.plot([x.total for x in swap_mem], label='Total Swap Memory')
  plt.plot([x.used for x in swap_mem], label='Used Swap Memory')
  plt.plot([x.free for x in swap_mem], label='Free Swap Memory')
  
  plt.legend(loc='best')
  plt.show()
  </code></pre>
  <p><a href="https://i.stack.imgur.com/jYh9t.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jYh9t.png" alt="Resulting plot" /></a></p>
  <p>I cannot see, how I can use the information about swap memory to detect the swapping of my process.</p>
  <ul>
  <li><p>The 'Used Swap Memory' information is kind of meaningless, as it is high from the very first moment (it counts global consumption) when the data of the process are obviously not swapped.</p>
  </li>
  <li><p>It seems best to look at the difference between 'Virtual Memory Size' and 'Resident Set Size' of the process. If VMS greatly exceeds RSS, it is a sign that the data are not on RAM but on disk.</p>
  <ul>
  <li>However, <a href="https://github.com/dask/distributed/issues/1409" rel="nofollow noreferrer">here</a> is described a problem with sudden explosion of VMS which makes it irrelevant in some cases, if I understand it correctly.</li>
  </ul>
  </li>
  <li><p>Other approach is to watch 'Available Memory' and be sure that it does not drop below a certain threshold, as <a href="https://psutil.readthedocs.io/en/latest/index.html?highlight=Process#psutil.virtual_memory" rel="nofollow noreferrer"><code>psutil</code> documentation suggests</a>. But to me it seems complicated to set the threshold properly. They suggests (in the small snippet) 100 MB, but on my machine it is something like 500 MB.</p>
  </li>
  </ul>
  <p>So the question stands. How to detect when OS starts swapping some resources of the running process to disk?</p>
  <p>I work on Windows, but the solution needs to be cross-platform (or at least as cross-platform as possible).</p>
  <p>Any suggestion, advice or useful link is welcomed. Thank you!</p>
  <hr />
  <p>For the context, I write a library which needs to manage its memory consumption.
  I believe that with the knowledge of the program logic, my manual swapping (serializing data to disk) can work better (faster) than OS swapping. More importantly, the OS swap space it limited, thus, sometimes it is necessary to do the manual swapping which does not utilize the OS swap space.</p>
  <p>In order to start the manual swapping at the right time, it is crucial to know when OS swapping starts.</p>