title: Is this an appropriate use case for PySpark? Airflow? Apache Beam?
body: <p>I have a complex data processing pipeline that's currently implemented on a single machine in Python.  </p>
  
  <p>The pipeline is built around processing objects belonging to a series of custom classes implementing Documents, Pages, Words, etc.  Most of the operations in this pipeline are embarrassingly parallel -- they work on individual documents -- though a few require gathering statistics on the whole set.  </p>
  
  <p>The dataset doesn't fit in memory, and the pipeline currently relies on pickling/depickling objects to and from the disk at each stages.  This works but is slow, and it's time to move to a more scalable, distributed architecture.  </p>
  
  <p>Ideally <strong>I'd like to be able to write something like this</strong>:</p>
  
  <pre><code>distributed_list = read_objects()
  for transform in transformations:
      distributed_list = transform(distributed_list)
  custom_dataset_object = convert_to_dataset(distributed_list)
  custom_dataset_object.save()
  </code></pre>
  
  <p>Where the whole processing stream is broken up into a DAG and parallelized across a cluster.</p>
  
  <p>I've been evaluating <strong>pyspark</strong> for this purpose but am not sure it's the right tool for the task given that I'm not working complex custom objects with lots of methods on them rather than simple datatypes.</p>
  
  <p><strong>Apache Beam</strong>'s DoFns look like they might accomplish what I'm looking for, but it doesn't seem very widely adopted and I'd like to use the most portable technologies possible.</p>
  
  <p>I could also break this up into a series of <strong>Airflow</strong> jobs, but that seams more manual and brittle than I'd like.</p>
  
  <p>For more experienced big-data engineers, <strong>are these evaluations accurate</strong>?  Would one of these tools (or some other) be best for my job?</p>