title: Scaling Prometheus federation
body: <p>I am trying to determine what can be tuned to get our federation collection intervals completing faster.  Federation from our collector pairs is surpassing the 60 scrape interval for federation, ~59 sec based on scrape_duration_seconds{job="federation"}</p>
  
  <p><strong>Our setup within data center A</strong>
  two collectors each pointed to all servers (~1,500) and a single federation server collecting from the one of the two collectors.  Collectors have an interval of 30s and the federation server 60s. For the data center where we have issues there are roughly 800,000 metrics.  The federation server is running 6proc, 16GB ram and version 1.6.2.  </p>
  
  <p>-storage.local.target-heap-size=10737418240 
  -storage.local.num-fingerprint-mutexes=8192</p>
  
  <p>Are there other obvious customization's I am missing? I can always deploy more collector pairs but I am worried that the federation tier cannot collect from a single collector within the 60 interval.  Our design assumes a single federation server per data center (for 1..N environments) which federates from multiple collector pairs.  Is there a better design to consider?</p>
  
  <p>I have reviewed these links but am unsure of how to proceed. 
  <a href="https://prometheus.io/docs/operating/storage/#settings-for-high-numbers-of-time-series" rel="nofollow noreferrer">https://prometheus.io/docs/operating/storage/#settings-for-high-numbers-of-time-series</a>
  <a href="https://prometheus.io/docs/operating/storage/#helpful-metrics" rel="nofollow noreferrer">https://prometheus.io/docs/operating/storage/#helpful-metrics</a></p>
  
  <p><a href="https://i.stack.imgur.com/Tylgo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Tylgo.png" alt="enter image description here"></a></p>