title: spark-streaming and connection pool implementation
body: <p>The spark-streaming website at <a href="https://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations-on-dstreams">https://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations-on-dstreams</a> mentions the following code:</p>
  
  <pre><code>dstream.foreachRDD { rdd =&gt;
    rdd.foreachPartition { partitionOfRecords =&gt;
      // ConnectionPool is a static, lazily initialized pool of connections
      val connection = ConnectionPool.getConnection()
      partitionOfRecords.foreach(record =&gt; connection.send(record))
      ConnectionPool.returnConnection(connection)  // return to the pool for future reuse
    }
  }
  </code></pre>
  
  <p>I have tried to implement this using org.apache.commons.pool2 but running the application fails with the expected java.io.NotSerializableException: </p>
  
  <pre><code>15/05/26 08:06:21 ERROR OneForOneStrategy: org.apache.commons.pool2.impl.GenericObjectPool
  java.io.NotSerializableException: org.apache.commons.pool2.impl.GenericObjectPool
          at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
   ...
  </code></pre>
  
  <p>I am wondering how realistic it is to implement a connection pool that is serializable. Has anyone succeeded in doing this ?</p>
  
  <p>Thank you. </p>