title: Elasticsearch 7.x circuit breaker - data too large - troubleshoot
body: <p><strong>The problem:</strong><br>
  Since the upgrading from ES-5.4 to ES-7.2 I started getting "data too large" errors, when trying to write concurrent bulk request (or/and search requests) from my multi-threaded Java application (using <code>elasticsearch-rest-high-level-client-7.2.0.jar</code> java client) to an ES cluster of 2-4 nodes.</p>
  
  <p><strong>My ES configuration:</strong></p>
  
  <pre><code>Elasticsearch version: 7.2
  
  custom configuration in elasticsearch.yml:   
      thread_pool.search.queue_size = 20000  
      thread_pool.write.queue_size = 500
  
  I use only the default 7.x circuit-breaker values, such as:  
      indices.breaker.total.limit = 95%  
      indices.breaker.total.use_real_memory = true  
      network.breaker.inflight_requests.limit = 100%  
      network.breaker.inflight_requests.overhead = 2  
  </code></pre>
  
  <p><strong>The error from elasticsearch.log:</strong></p>
  
  <pre><code>    {
        "error": {
          "root_cause": [
            {
              "type": "circuit_breaking_exception",
              "reason": "[parent] Data too large, data for [&lt;http_request&gt;] would be [3144831050/2.9gb], which is larger than the limit of [3060164198/2.8gb], real usage: [3144829848/2.9gb], new bytes reserved: [1202/1.1kb]",
              "bytes_wanted": 3144831050,
              "bytes_limit": 3060164198,
              "durability": "PERMANENT"
            }
          ],
          "type": "circuit_breaking_exception",
          "reason": "[parent] Data too large, data for [&lt;http_request&gt;] would be [3144831050/2.9gb], which is larger than the limit of [3060164198/2.8gb], real usage: [3144829848/2.9gb], new bytes reserved: [1202/1.1kb]",
          "bytes_wanted": 3144831050,
          "bytes_limit": 3060164198,
          "durability": "PERMANENT"
        },
        "status": 429
      }
  </code></pre>
  
  <p><strong>Thoughts:</strong><br>
  I'm having hard time to pin point the source of the issue.<br>
  When using ES cluster nodes with &lt;=8gb heap size (on a &lt;=16gb vm), the problem become very visible, so, one obvious solution is to increase the memory of the nodes.<br>
  But I feel that increasing the memory only hides the issue.</p>
  
  <p><strong>Questions:</strong><br>
  I would like to understand what scenarios could have led to this error?<br>
  and what action can I take in order to handle it properly?<br>
  (change circuit-breaker values, change es.yml configuration, change/limit my ES requests)</p>