title: Why won't this Spark sample code load in spark-shell?
body: <p>The example code below is from the book Advanced Analytics with Spark.  When I load it into spark-shell (version 1.4.1) it gives the following error, indicating that it can't find StatCounter:</p>
  
  <pre><code>import org.apache.spark.util.StatCounter
  &lt;console&gt;:9: error: not found: type StatCounter
          val stats: StatCounter = new StatCounter()
                     ^
  &lt;console&gt;:9: error: not found: type StatCounter
          val stats: StatCounter = new StatCounter()
                                       ^
  &lt;console&gt;:23: error: not found: type NAStatCounter
          def apply(x: Double) = new NAStatCounter().add(x)
  </code></pre>
  
  <p>If I just do the following in spark-shell there is no problem:</p>
  
  <pre><code>scala&gt; import org.apache.spark.util.StatCounter
  import org.apache.spark.util.StatCounter
  
  scala&gt; val statsCounter: StatCounter = new StatCounter()
  statsCounter: org.apache.spark.util.StatCounter = (count: 0, mean: 0.000000, stdev: NaN, max: -Infinity, min: Infinity)
  </code></pre>
  
  <p>The problem seems to be with the :load command in spark-shell.</p>
  
  <p>Here's the code:</p>
  
  <pre><code>import org.apache.spark.util.StatCounter
  class NAStatCounter extends Serializable {
      val stats: StatCounter = new StatCounter()
      var missing: Long = 0
  
      def add(x: Double): NAStatCounter = {
          if (java.lang.Double.isNaN(x)) {
              missing += 1
          } else {
          stats.merge(x)
          }
          this
      }
  
      def merge(other: NAStatCounter): NAStatCounter = {
          stats.merge(other.stats)
          missing += other.missing
          this
      }
  
      override def toString = {
          "stats: " + stats.toString + " NaN: " + missing
      }
  }
  
  object NAStatCounter extends Serializable {
      def apply(x: Double) = new NAStatCounter().add(x)
  }
  </code></pre>