title: "Lazy" version of asyncio.gather?
body: <p>I'm using Python's asyncio module and <code>async</code>/<code>await</code> to process a character sequence in chunks concurrently and collect the results in a list. For that I'm using a chunker function (<code>split</code>) and a chunk processing function (<code>process_chunk</code>). They both come from a third-party library, and I would prefer not to change them.</p>
  
  <p>Chunking is slow, and the number of chunks is not known up front, which is why I don't want to consume the whole chunk generator at once. Ideally, the code should advance the generator in sync with the <code>process_chunk</code>'s semaphore, i.e., every time that function returns.</p>
  
  <p>My code</p>
  
  <pre><code>import asyncio
  
  def split(sequence):
      for x in sequence:
          print('Getting the next chunk:', x)
          yield x
      print('Finished chunking')
  
  async def process_chunk(chunk, *, semaphore=asyncio.Semaphore(2)):
      async with semaphore:
          print('Processing chunk:', chunk)
          await asyncio.sleep(3)
          return 'OK'
  
  async def process_in_chunks(sequence):
      gen = split(sequence)
      coro = [process_chunk(chunk) for chunk in gen]
      results = await asyncio.gather(*coro)
  
  if __name__ == '__main__':
      loop = asyncio.get_event_loop()
      loop.run_until_complete(process_in_chunks('ABC'))
  </code></pre>
  
  <p>kind of works and prints</p>
  
  <pre class="lang-none prettyprint-override"><code>Getting the next chunk: A
  Getting the next chunk: B
  Getting the next chunk: C
  Finished chunking
  Processing chunk: C
  Processing chunk: B
  Processing chunk: A
  </code></pre>
  
  <p>although that means that the <code>gen</code> generator is exhausted before the processing begins. I know why it happens, but how do change that?</p>