title: Spark Streaming: java.io.FileNotFoundException: File does not exist: <input_filename>._COPYING_
body: <p>I am writing a spark streaming application which reads input from HDFS. I submit spark application to yarn and then run a script which copies data from local fs to HDFS. </p>
  
  <p>But Spark application starts throwing fileNotFoundException. 
  I believe this is happening because spark is picking up files before it is being copied fully onto HDFS. </p>
  
  <p>Following is the some part of exception trace:</p>
  
  <pre><code>java.io.FileNotFoundException: File does not exist: &lt;filename&gt;._COPYING_
          at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
          at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
          at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
          at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
          at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
          at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
          at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
          at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
  
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
          at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
          at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
  </code></pre>
  
  <p>Any suggestion how to resolve this?</p>
  
  <p>Thanks</p>