title: Prometheus track long running processes in a HA environment
body: <p>I am running Prometheus to monitor my system and am currently building in application-level metrics.</p>
  
  <p>The issue I am having is with long running processes. I want to know how many are running at any time, and I can't seem to find a good solution for this with Prometheus.</p>
  
  <p>The processes are running async from multiple redundant applications that both receive traffic at the same time. The processes run between 10 minutes and a few hours so my initial naive approach was to count a counter up whenever a process was started and increment another counter when the processes stopped.</p>
  
  <p>The issue I am having here is that I see the uptick when a process starts in Grafana with the <code>rate</code> operator, but I can't monitor how many are running at any time.</p>
  
  <p>Since two applications are managing the same pool of processes I can't really use a gauge to report the current number of running processes because a process might be started on instance A and then stop on instance B (or any other of the running application instances). (And the number of processes would be requested from a shared database so it would be inflated).</p>
  
  <p>Now an approach I tried was to subtract <code>started - finished</code> counters to get to the currently running instance. But this would run out of sync quite quickly if the application reporting a process as finishing at some point would be restarted or die before the metric was scraped by prometheus (so I would get into a state where 0 is no longer the baseline).</p>
  
  <p>Any suggestions on how to handle this?</p>