title: How to let Apache Spark on Windows access Hadoop on Linux?
body: <p>First, I have almost no experience on Apache Hadoop and Apache Spark.</p>
  
  <p>What I want for now is as follows:</p>
  
  <ul>
  <li>Hadoop is running on Hortonworks Sandbox 2.1, which is installed on a Windows 7 machine.</li>
  <li>Spark shell and Spark programs are running on the Windows 7 machine, which is the same machine as above.</li>
  <li>Spark shell and Spark programs can use the Hadoop on Hortonworks Sandbox 2.1.</li>
  </ul>
  
  <p>The reason I want to run Spark on Windows is, I want to develop Spark programs on a Windows 7 machine, which is familiar to me.</p>
  
  <p>Installing Hortonworks Sandbox 2.1 was straightforward.</p>
  
  <p>Then I tried to build Spark on Windows 7 machine as follows:</p>
  
  <ul>
  <li>Install JDK 7, Git, sbt (JDK 8 is not working for sbt, at least for now: 2014.5.8)</li>
  <li><code>git clone git://github.com/apache/spark.git</code> (spark-0.9.1 does not compile with Hadoop 2.4.0 when SPARK_YARN=true, as described in <a href="https://issues.apache.org/jira/browse/SPARK-1465" rel="nofollow">https://issues.apache.org/jira/browse/SPARK-1465</a>)</li>
  <li>Run <code>cmd</code></li>
  <li><code>cd</code> to spark root directory</li>
  <li><code>set SPARK_HADOOP_VERSION=2.4.0</code> (Hadoop on Hortonworks Sandbox 2.1 is 2.4.0.2.1.1.0-385)</li>
  <li><code>set SPARK_YARN=true</code></li>
  <li><code>sbt assembly</code></li>
  </ul>
  
  <p>Done. Successful.<br>
  But when I ran bin\spark-shell, the following warning appeared.</p>
  
  <pre><code>14/05/08 11:26:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  14/05/08 11:26:15 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
  java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
      at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
      at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
      ...
  </code></pre>
  
  <p>Apparently spark shell requires winutil.exe.<br>
  I found that winutils.exe is part of Hadoop, but it was not included in the normal prebuilt distribution.<br>
  So I built Hadoop on Windows.<br>
  With some trouble, I managed to build <code>hadoop-2.4.0.tar.gz</code>, which includes winutils.exe and Hadoop.dll.<br></p>
  
  <p>But... I don't know what to do from now on.<br>
  How can I install or apply the built Hadoop package so that Spark can get it to access Hadoop on the Hortonworks Sandbox 2.1?</p>
  
  <p>Any suggestions are welcome.</p>