title: How do I avoid counting new lines as words in spark?
body: <p>I'm trying to run the word count example on lorem ipsum; i.e. count <em>words</em> frequencies in a given text file. As the rule of words splitting I would like to use any instance of non-character. I have the following python code:</p>
  
  <pre><code>import re
  from pyspark import SparkContext
  print "-----------------===========================-----------------"
  print "-----------------==========Staring==========-----------------"
  print "-----------------===========================-----------------"
  sc = SparkContext(appName = "simple app")
  
  print "-----------------===========================-----------------"
  print "-----------------==========Loaded file======-----------------"
  print "-----------------===========================-----------------"
  text_file = sc.textFile("lorem.txt")
  
  print "-----------------===========================-----------------"
  print "-----------------==========  Process  ======-----------------"
  print "-----------------===========================-----------------"
  counts = text_file.flatMap(lambda line: re.split(r'\W*', line.rstrip())) \
           .map(lambda word: (word, 1)) \
           .reduceByKey(lambda a, b: a + b) \
           .map(lambda (a,b): (b, a)) \
           .sortByKey(False)
  
  output = counts.collect()
  counts.saveAsTextFile("test.txt")
  sc.stop()
  for x in output:
      print (x[0], x[1])
  </code></pre>
  
  <p>It works almost as expected. The main problem is that it counts the new lines. If I understand correctly this is due to the way the regex works, but I cannot find a workaround. What am I doing wrong?</p>