title: Architecture for luigi tasks with multiple inputs
body: <p>I have number of pickle files, one for each date between 2005 and 2010. Each file contains a dictionary of words with their respective frequencies for that date. I also have a "master file" with all unique words for the whole period. There are about 5 million words in total.</p>
  
  <p>I need to take all that data and produce one CSV file per word, which will have one row per date. E.g., for example file <code>some_word.txt</code>:</p>
  
  <pre><code>2005-01-01,0.0003
  2005-01-02,0.00034
  2005-01-03,0.008
  </code></pre>
  
  <p>I'm having trouble organizing this process with the luigi framework. My current top-level task takes a word, looks up it's associated frequency for every date and stores the result in a CSV file. I guess I could just loop through every word in my master file and run the task with that word, but I estimate that would take months, if not longer. Here's my top-level <code>AggregateTokenFreqs</code> task in a simplified version.</p>
  
  <pre><code>class AggregateTokenFreqs(luigi.Task):
      word = luigi.Parameter()
  
      def requires(self):
          pass  # not sure what to require here, master file?
  
      def output(self):
          return luigi.LocalTarget('data/{}.csv'.format(self.word))
  
      def run(self):
          results = []
          for date_ in some_list_of_dates:
              with open('pickles/{}.p'.format(date_), 'rb') as f:
                  freqs = pickle.load(f)
                  results.append((date_, freqs.get(self.word))
  
          # Write results list to output CSV file
  </code></pre>