title: The touch of death
body: <p>Should the file <code>foo.txt</code> already exist before running the following Snakefile</p>
  
  <pre><code>rule master :
          output :
                  touch('foo.txt')
  
          input : 'task.done'
  
  rule task :
          output :
                  touch('task.done')
  
          shell : 'echo "bar" &gt; foo.txt'
  </code></pre>
  
  <p>it is clear that there is</p>
  
  <pre><code>Nothing to be done.
  </code></pre>
  
  <p>and the contents of <code>foo.txt</code> remain unaltered.  Should the file <code>foo.txt</code> <em>not</em> exist before running this Snakefile, it is created with the text "bar" by <code>rule task</code>, but then this is erased by the touch of <code>rule master</code>.  There is even the appropriate warning</p>
  
  <pre><code>Warning: the following output files of rule master were not present when the DAG was created:
  {'foo.txt'}
  Touching output file foo.txt.
  </code></pre>
  
  <p>I understand perfectly what is going on, but <em>why</em> does snakemake do this?  Why can <code>touch</code> do more than just "touch" a file in certain contexts?</p>
  
  <h2>UPDATE:</h2>
  
  <p>Ok, thanks to a helpful comment, this does not really have to do specifically with <code>touch</code>, and is rather a question about untracked files (maybe a change of title should be in order), in particular those <em>not present when the DAG was created</em>, as clearly mentioned in the Warning.  For example, if the above Snakefile is instead</p>
  
  <pre><code>rule master :
          output : 'foo.txt'
  
          input : 'task.done'
  
  rule task :
          output :
                  touch('task.done')
  
          shell : 'echo "bar" &gt; foo.txt'
  </code></pre>
  
  <p>I now get:</p>
  
  <pre><code>Warning: the following output files of rule master were not present when the DAG was created:
  {'foo.txt'}
  Waiting at most 5 seconds for missing files.
  MissingOutputException in line 1 of /home/patterson/snaketest/touch-of-death.snake:
  Missing files after 5 seconds:
  foo.txt
  This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
  </code></pre>
  
  <p>Which again, is perfectly understandable behavior: I get the warning about the file <code>foo.txt</code> which was not there before the creation of the DAG, and then it waits for <code>foo.txt</code>, which not the target of any rule, and so snakemake does not see it: it is technically missing according to snakemake.</p>
  
  <p>I agree that the above is an example of bad design, and that it is because I am playing the devil's advocate in trying to understand why snakemake operates like this.</p>
  
  <p>Let's say then, for argument, that one could have snakemake pay attention to files that are created during the workflow --- again, I believe that one could somehow force this behavior with <code>checkpoints</code> (re-evaluation of the DAG at runtime), but let's say I do not want to do this either --- even an example, where one could really shoot oneself badly in the foot with a (bad) design as it is above, would be welcome.</p>
  
  <p>I realize that this could be related to why <code>dynamic</code> files is now deprecated in snakemake (in favor of <code>checkpoints</code>), but also because of inherent limitations regarding concurrent processes in python (cf. <a href="https://sebastianraschka.com/Articles/2014_multiprocessing.html" rel="nofollow noreferrer">https://sebastianraschka.com/Articles/2014_multiprocessing.html</a>), since snakemake is written in python after all.</p>