title: Bypassing org.apache.hadoop.mapred.InvalidInputException: Input Pattern s3n://[...] matches 0 files
body: <p>This is a question i've <a href="http://apache-spark-user-list.1001560.n3.nabble.com/Ignoring-S3-0-files-exception-tt6101.html">already asked</a> on the spark user mailing list and i hope to get more success here.</p>
  
  <p>I'm not sure it's directly related to spark though spark has something to do with the fact I can't easily resolve that problem.</p>
  
  <p>I'm trying to get some files from S3 using various patterns. My problem is that some of those patterns may return nothing, and when they do so, i get the following exception:</p>
  
  <pre><code>org.apache.hadoop.mapred.InvalidInputException: Input Pattern s3n://bucket/mypattern matches 0 files
      at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
      at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
      at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:140)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
      at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
      at org.apache.spark.rdd.FlatMappedRDD.getPartitions(FlatMappedRDD.scala:30)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
      at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:52)
      at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:52)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
      at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      at scala.collection.AbstractTraversable.map(Traversable.scala:105)
      at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:52)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
      at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:58)
      at org.apache.spark.api.java.JavaPairRDD.reduceByKey(JavaPairRDD.scala:335)
      ... 2 more
  </code></pre>
  
  <p>I would like a way to ignore missing files and just do nothing in that case. The problem here IMO is that i don't know if a pattern will return something until it's actually executed and spark starts processing data only when an action occurs (here, the <code>reduceByKey</code> part). So i can't just catch an error somewhere and let things continue on.</p>
  
  <p>One solution would be to force spark to process each path individually but that will probably cost allot in terms of speed and/or memory so i'm looking for an other option that would be efficient.</p>
  
  <p>I'm using spark 0.9.1.
  Thanks</p>