title: I can't get Hadoop to start using Amazon EC2/S3
body: <p>I have created an AMI image and installed Hadoop from the Cloudera CDH2 build. I configured my core-site.xml as so:</p>
  
  <pre><code>&lt;property&gt;
     &lt;name&gt;fs.default.name&lt;/name&gt;
     &lt;value&gt;s3://&lt;BUCKET NAME&gt;/&lt;/value&gt;
   &lt;/property&gt;
   &lt;property&gt;
      &lt;name&gt;fs.s3.awsAccessKeyId&lt;/name&gt;
      &lt;value&gt;&lt;ACCESS ID&gt;&lt;/value&gt;
   &lt;/property&gt;
   &lt;property&gt;
      &lt;name&gt;fs.s3.awsSecretAccessKey&lt;/name&gt;
      &lt;value&gt;&lt;SECRET KEY&gt;&lt;/value&gt;
   &lt;/property&gt;
   &lt;property&gt;
      &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;
      &lt;value&gt;/var/lib/hadoop-0.20/cache/${user.name}&lt;/value&gt;
   &lt;/property&gt;
  </code></pre>
  
  <p>But I get the following error message when I start up the hadoop daemons in the namenode log:</p>
  
  <pre><code>2010-11-03 23:45:21,680 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:      java.lang.IllegalArgumentException: Invalid URI for NameNode address (check     fs.default.name): s3://&lt;BUCKET NAME&gt;/ is not of scheme 'hdfs'.
      at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:177)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:198)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.&lt;init&gt;(NameNode.java:306)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1006)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1015)
  
  2010-11-03 23:45:21,691 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
  </code></pre>
  
  <p>However, I am able to execute hadoop commands from the command line like so:</p>
  
  <pre><code>hadoop fs -put sun-javadb-common-10.5.3-0.2.i386.rpm s3://&lt;BUCKET NAME&gt;/
  
   hadoop fs -ls s3://poc-jwt-ci/
  Found 3 items
  drwxrwxrwx   -          0 1970-01-01 00:00 /
  -rwxrwxrwx   1      16307 1970-01-01 00:00 /sun-javadb-common-10.5.3-0.2.i386.rpm
  drwxrwxrwx   -          0 1970-01-01 00:00 /var
  </code></pre>
  
  <p>You will notice there is a <code>/</code> and a <code>/var</code> folders in the bucket. I ran the hadoop namenode -format when I first saw this error, then restarted all services, but still receive the weird <code>Invalid URI for NameNode address (check     fs.default.name): s3://&lt;BUCKET NAME&gt;/ is not of scheme 'hdfs'.</code></p>
  
  <p>I also notice that the file system created looks like this:</p>
  
  <pre><code> hadoop fs -ls s3://&lt;BUCKET NAME&gt;/var/lib/hadoop-0.20/cache/hadoop/mapred/system
  Found 1 items
  -rwxrwxrwx   1          4 1970-01-01 00:00 /var/lib/hadoop0.20/cache/hadoop/mapred/system/jobtracker.info
  </code></pre>
  
  <p>Any ideas of what's going on?</p>