title: Celery add tasks to queue when ready
body: <p>I am new to Celery . I am trying to make a distributed task work with celery. </p>
  
  <p>Lets say I  have a single task in my tasks file <code>task.py</code>: </p>
  
  <pre><code>@celery.task
  def generate_sorts(params,meta_data_dict):
      '''
      This will generate some sorts .
      '''
      pass
  </code></pre>
  
  <p>And I am doing the following as part of some distributed processing:</p>
  
  <pre><code>taskset = TaskSet(
      tasks.generate_sorts.subtask(args = (params, meta_data_dict))
      for meta_data_dict in chunk_generator)
  
  print "Dispatching tasks"
  taskset_result = taskset.apply_async()
  
  print "Waiting for results"
  results = taskset_result.join_native()
  print "Results:"
  #Process the results. 
  </code></pre>
  
  <p>Now chunk_generator is basically a generator pattern which goes to the database and fetches some metadata. My problem now is these tasks get accumulated before eventually sent to the task queue. My generator takes somewhere around 30 minutes to fetch all the meta data before the tasks are actually added to queue. I know that is how <code>TaskSet</code> is intend to perform. I am looking for some alternate to <code>TaskSet</code>, i.e. will I be able to perform the equivalent of the below in distributed manner. </p>
  
  <pre><code>pool.imap_unordered(generate_sorts, chunk_generator)
  </code></pre>
  
  <p>The above will perform generate_sots as soon as a generator yields a result. In other words is there anything alternate to <code>TaskSet</code> where I can add from generator as soon as the generator yield the first job, instead of waiting for generator to fetch everything before I could eventually start doing some work. </p>