title: Running shell script through oozie
body: <p>I'm trying to execute a shell script through oozie but I'm having some issues.</p>
  
  <p>I have a property file like this (import.properties):</p>
  
  <pre><code>startIndex=2000
  chunkSize=2000
  </code></pre>
  
  <p>The idea is, in every single execution the startIndex value will be updated by the chunk size. So if I execute it, it should have</p>
  
  <pre><code>startIndex=4000
  chunkSize=2000
  </code></pre>
  
  <p>I have tested the script separately and it works fine. Here are my other related files.</p>
  
  <p>job.properties</p>
  
  <pre><code>nameNode=hdfs://192.168.56.101:8020
  jobTracker=192.168.56.101:50300
  wfeRoot=wfe
  queueName=default
  EXEC=script.sh
  propertyLoc=import.properties
  
  oozie.use.system.libpath=true
  oozie.wf.application.path=${nameNode}/user/${user.name}/${wfeRoot}/coordinator
  </code></pre>
  
  <p>workflow.xml</p>
  
  <pre><code>&lt;workflow-app xmlns='uri:oozie:workflow:0.2' name='shell-wf'&gt;
  &lt;start to='shell1' /&gt;
  &lt;action name='shell1'&gt;
      &lt;shell xmlns="uri:oozie:shell-action:0.1"&gt;
          &lt;job-tracker&gt;${jobTracker}&lt;/job-tracker&gt;
          &lt;name-node&gt;${nameNode}&lt;/name-node&gt;
          &lt;configuration&gt;
              &lt;property&gt;
                &lt;name&gt;mapred.job.queue.name&lt;/name&gt;
                &lt;value&gt;${queueName}&lt;/value&gt;
              &lt;/property&gt;
          &lt;/configuration&gt;
          &lt;exec&gt;${EXEC}&lt;/exec&gt;
       &lt;file&gt;${EXEC}#${EXEC}&lt;/file&gt;
          &lt;file&gt;${propertyLoc}#${propertyLoc}&lt;/file&gt;
      &lt;/shell&gt;
      &lt;ok to="end" /&gt;
      &lt;error to="fail" /&gt;
  &lt;/action&gt;
  &lt;kill name="fail"&gt;
      &lt;message&gt;Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]&lt;/message&gt;
  &lt;/kill&gt;
  &lt;end name='end' /&gt;
  </code></pre>
  
  <p></p>
  
  <p>script.sh</p>
  
  <pre><code>#!/bin/sh
  file=import.properties
  . $file
  
  SCRIPT=$(readlink -f $file)
  SCRIPTPATH=$(dirname $SCRIPT)
  echo $SCRIPTPATH
  
  newStartIndex=`expr $chunkSize + $startIndex`
  newStartIndexStr=startIndex=$newStartIndex
  
  oldStartIndexStr=startIndex=$startIndex
  chunkSizeStr=chunkSize=$chunkSize
  
  sed -i "s|$oldStartIndexStr|$newStartIndexStr|g" $file
  </code></pre>
  
  <p>And I put all these files inside my HDFS working directory:</p>
  
  <pre><code>[ambari_qa@sandbox coordinator]$ hadoop fs -lsr /user/ambari_qa/wfe/coordinator
  -rw-rw-rw-   1 ambari_qa hdfs         32 2013-05-09 00:12    /user/ambari_qa/wfe/coordinator/import.properties
  -rw-rw-rw-   1 ambari_qa hdfs        533 2013-05-09 01:19 /user/ambari_qa/wfe/coordinator/script.sh
  -rw-------   1 ambari_qa hdfs        852 2013-05-09 00:50 /user/ambari_qa/wfe/coordinator/workflow.xml
  </code></pre>
  
  <p>I was expecting that the import.properties file will be changed after each execution. But I see it's not changing even though the oozie job is successful. For debugging purpose, I printed out the location of the file during execution and found out that it copied to another location (from log):</p>
  
  <pre><code>&gt;&gt;&gt; Invoking Shell command line now &gt;&gt;
  
  Stdoutput /hadoop/mapred/taskTracker/ambari_qa/distcache/-5756672768810005023_889271025_125659265/192.168.56.101/user/ambari_qa/wfe/coordinator
  Stdoutput startIndex=4000
  Stdoutput startIndex=2000
  Exit code of the Shell command 0
  &lt;&lt;&lt; Invocation of Shell command completed &lt;&lt;&lt;
  </code></pre>
  
  <p>What I need to do so that it effects the working directory of HDFS? Thanks in advance.</p>
  
  <p>Update:</p>
  
  <p>After changing the script based on Chris's suggestion, it becomes (last 3 lines):</p>
  
  <pre><code>hadoop fs -rm hdfs://ip-10-0-0-92:8020/user/ambari_qa/wfe/shell-oozie/$file
  sed -i "s|$oldStartIndexStr|$newStartIndexStr|g" $file
  hadoop fs -put $file /user/ambari_qa/wfe/shell-oozie
  </code></pre>
  
  <p>But then I started facing permission issue. I gave write permission on that file and folder.</p>
  
  <pre><code>[ambari_qa@ip-10-0-0-91 shell-oozie]$  hadoop fs -ls /user/ambari_qa/wfe/shell-oozie
  </code></pre>
  
  <p>Found 3 items:</p>
  
  <pre><code>-rw-rw-rw-   3 ambari_qa hdfs         32 2013-05-10 16:55 /user/ambari_qa/wfe/shell-oozie/import.properties
  -rw-rw-rw-   3 ambari_qa hdfs        540 2013-05-10 16:48 /user/ambari_qa/wfe/shell-oozie/script.sh
  -rw-rw-rw-   3 ambari_qa hdfs        826 2013-05-10 15:29 /user/ambari_qa/wfe/shell-oozie/workflow.xml
  </code></pre>
  
  <p>Here is the error log:</p>
  
  <pre><code>rm: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=EXECUTE, inode="ambari_qa":ambari_qa:hdfs:rwxrwx---
  put: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=EXECUTE, inode="ambari_qa":ambari_qa:hdfs:rwxrwx---
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
  </code></pre>