title: Why is elasticsearch python parallel_bulk using so much memory?
body: <p>I have a pretty simple indexing task that reads line-delimited json from a file, does simple manipulation of incoming data, and then indexes using elasticsearch <code>parallel_bulk</code>. I am using python3, ES 6.7 on the server (2 nodes, 16G each). </p>
  
  <p>When I run the code below, I see pretty steadily-increasing memory usage until the machine starts swapping. The data are not huge (&lt;500 characters per line), so I am surprised by the memory usage. </p>
  
  <p>I have done a little tuning of queue size, thread count, and chunk size and there is quite a bit of sensitivity. I'd like to optimize indexing speed on the client (I have done the work on the server side to speed things up) while keeping memory to a reasonable amount. </p>
  
  <p>Code:</p>
  
  <pre><code>def yield_records(idx='sra_bulk_test_biosample'):
      for fname in list(filter(lambda f: f.endswith('.json.gz'),os.listdir())):
          print(fname)
          with gzip.open(fname, 'r') as f:
              n = 0
              for line in f:
                  n+=1
                  if(n % 10000 == 0):
                      print(n)
                  res = json.loads(line)
                  yield {
                      "_op_type": "index",                   # this is the default
                      "_index": idx,
                      "_type": 'doc',                        # no more doc types after es_6
                      "_id": res['accession'], # extract _id from record
                      "_source": res                         # use entire record as "source"
                  }
  
  def main():
      for success, info in parallel_bulk(es, actions = yield_records(), chunk_size=1000, queue_size=4, thread_count=4, yield_ok = False):
          if(not success):
              print(info)
  
  if __name__ == '__main__':
      main()
  </code></pre>