title: Why we need to distribute files in Spark at all, e.g. --py-files?
body: <p>As I read from many blogs and posts here in SO, for example <a href="https://stackoverflow.com/a/22596875/2235936">this one</a> (in the first a few paragraphs), quoted as follows:</p>
  
  <blockquote>
    <p>Not to get into too many details, but when you run different
    transformations on a RDD (map, flatMap, filter and others), your
    transformation code (closure) is:</p>
    
    <ol>
    <li>serialized on the driver node,</li>
    <li>shipped to the appropriate nodes in the cluster,</li>
    <li>deserialized,</li>
    <li>and finally executed on the nodes</li>
    </ol>
  </blockquote>
  
  <p>OK, here is my take at this: </p>
  
  <p>I define some custom transformation/action functions in the driver, and then those custom functions will be serialized to all the executors to run the job.</p>
  
  <p>Then what's the point of shipping extra py-files to all the nodes? Since all that executors need will be serialized to them, what the heck is going on here?</p>