title: SLURM `srun` vs `sbatch` and their parameters
body: <p>I am trying to understand what the difference is between SLURM's <a href="https://slurm.schedmd.com/srun.html" rel="noreferrer"><code>srun</code></a> and <a href="https://slurm.schedmd.com/sbatch.html" rel="noreferrer"><code>sbatch</code></a> commands. I will be happy with a general explanation, rather than specific answers to the following questions, but here are some specific points of confusion that can be a starting point and give an idea of what I'm looking for. </p>
  
  <p>According to the <a href="https://slurm.schedmd.com/quickstart.html" rel="noreferrer">documentation</a>, <code>srun</code> is for submitting jobs, and <code>sbatch</code> is for submitting jobs for later execution, but the practical difference is unclear to me, and their behavior seems to be the same. For example, I have a cluster with 2 nodes, each with 2 CPUs. If I execute <code>srun testjob.sh &amp;</code> 5x in a row, it will nicely queue up the fifth job until a CPU becomes available, as will executing <code>sbatch testjob.sh</code>.</p>
  
  <p>To make the question more concrete, I think a good place to start might be: <strong>What are some things that I can do with one that I cannot do with the other, and why?</strong></p>
  
  <p>Many of the arguments to both commands are the same. The ones that seem the most relevant are <code>--ntasks</code>, <code>--nodes</code>, <code>--cpus-per-task</code>, <code>--ntasks-per-node</code>. <strong>How are these related to each other, and how do they differ for <code>srun</code> vs <code>sbatch</code>?</strong></p>
  
  <p>One particular difference is that <code>srun</code> will cause an error if <code>testjob.sh</code> does not have executable permission i.e. <code>chmod +x testjob.sh</code> whereas <code>sbatch</code> will happily run it. <strong>What is happening "under the hood" that causes this to be the case?</strong></p>
  
  <p>The documentation also mentions that <code>srun</code> is commonly used inside of <code>sbatch</code> scripts. This leads to the question: <strong>How do they interact with each other, and what is the "canonical" usecase for each them? Specifically, would I ever use <code>srun</code> by itself?</strong></p>