title: Apache Spark: Applying a function from sklearn parallel on partitions
body: <p>I'm new to Big Data and Apache Spark (and an undergrad doing work under a supervisor).</p>
  
  <p>Is it possible to apply a function (i.e. a spline) to only partitions of the RDD? I'm trying to implement some of the work in the paper <a href="http://sites.stat.psu.edu/~rli/research/huge.pdf" rel="nofollow">here</a>.</p>
  
  <p>The book "Learning Spark" seems to indicate that this is possible, but doesn't explain how.</p>
  
  <blockquote>
    <p>"If you instead have many small datasets on which you want to train different learning models, it would be better to use a single- node learning library (e.g., Weka or SciKit-Learn) <strong>on each node</strong>, perhaps calling it in <strong>parallel across nodes</strong> using a Spark <code>map()</code>."</p>
  </blockquote>