{"action":"created","issue":{"url":"https://api.github.com/repos/nextflow-io/nextflow/issues/361","repository_url":"https://api.github.com/repos/nextflow-io/nextflow","labels_url":"https://api.github.com/repos/nextflow-io/nextflow/issues/361/labels{/name}","comments_url":"https://api.github.com/repos/nextflow-io/nextflow/issues/361/comments","events_url":"https://api.github.com/repos/nextflow-io/nextflow/issues/361/events","html_url":"https://github.com/nextflow-io/nextflow/issues/361","id":232695238,"number":361,"title":"nextflow hanging with slurm executor","user":{"login":"msmootsgi","id":22308243,"avatar_url":"https://avatars0.githubusercontent.com/u/22308243?v=3","gravatar_id":"","url":"https://api.github.com/users/msmootsgi","html_url":"https://github.com/msmootsgi","followers_url":"https://api.github.com/users/msmootsgi/followers","following_url":"https://api.github.com/users/msmootsgi/following{/other_user}","gists_url":"https://api.github.com/users/msmootsgi/gists{/gist_id}","starred_url":"https://api.github.com/users/msmootsgi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msmootsgi/subscriptions","organizations_url":"https://api.github.com/users/msmootsgi/orgs","repos_url":"https://api.github.com/users/msmootsgi/repos","events_url":"https://api.github.com/users/msmootsgi/events{/privacy}","received_events_url":"https://api.github.com/users/msmootsgi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-05-31T21:02:41Z","updated_at":"2017-06-23T13:06:41Z","closed_at":null,"body":"Nextflow seems to be hanging after the pipeline fails when running on a slurm cluster.  Here is the full .nextflow.log.  This particular pipeline has a `workflow.onComplete` block that also isn't being called.\r\n\r\n```\r\nMay-31 19:17:58.816 [main] DEBUG nextflow.cli.Launcher - $> /usr/local/bin/nextflow run http://git.l.synthgeno.global/SGI-Pipelines/GeneFlagging.git -resume -hub gitlab -r aws_changes -latest -params-file /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/params.yaml -process.executor slurm\r\nMay-31 19:17:58.964 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 0.24.4\r\nMay-31 19:17:59.521 [main] DEBUG nextflow.scm.AssetManager - Repository URL: http://git.l.synthgeno.global/SGI-Pipelines/GeneFlagging.git; Project: SGI-Pipelines/GeneFlagging; Hub provider: gitlab\r\nMay-31 19:17:59.529 [main] INFO  nextflow.cli.CmdRun - Pulling SGI-Pipelines/GeneFlagging ...\r\nMay-31 19:17:59.533 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> http://git.l.synthgeno.global/api/v3/projects/SGI-Pipelines%2FGeneFlagging\r\nMay-31 19:17:59.884 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> http://git.l.synthgeno.global/api/v3/projects/SGI-Pipelines%2FGeneFlagging/repository/files?file_path=nextflow.config&ref=master\r\nMay-31 19:18:00.018 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> http://git.l.synthgeno.global/api/v3/projects/SGI-Pipelines%2FGeneFlagging/repository/files?file_path=main.nf&ref=master\r\nMay-31 19:18:00.139 [main] DEBUG nextflow.scm.AssetManager - Pulling SGI-Pipelines/GeneFlagging -- Using remote clone url: http://git.l.synthgeno.global/SGI-Pipelines/GeneFlagging.git\r\nMay-31 19:18:01.244 [main] INFO  nextflow.cli.CmdRun -  downloaded from http://git.l.synthgeno.global/SGI-Pipelines/GeneFlagging.git\r\nMay-31 19:18:01.675 [main] DEBUG nextflow.scm.AssetManager - Git config: /tools/nextflow/assets/SGI-Pipelines/GeneFlagging/.git/config; branch: master; remote: origin; url: http://git.l.synthgeno.global/SGI-Pipelines/GeneFlagging.git\r\nMay-31 19:18:01.675 [main] INFO  nextflow.cli.CmdRun - Launching `SGI-Pipelines/GeneFlagging` [big_jennings] - revision: e8dc1db4ec [aws_changes]\r\nMay-31 19:18:01.686 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /tools/nextflow/assets/SGI-Pipelines/GeneFlagging/nextflow.config\r\nMay-31 19:18:01.690 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /tools/nextflow/assets/SGI-Pipelines/GeneFlagging/nextflow.config\r\nMay-31 19:18:01.825 [main] DEBUG nextflow.config.ConfigBuilder - Setting config profile: 'standard'\r\nMay-31 19:18:01.874 [main] WARN  nextflow.config.ConfigBuilder - It seems you never run this project before -- Option `-resume` is ignored\r\nMay-31 19:18:01.936 [main] DEBUG nextflow.Session - Session uuid: 57c9f9a1-cc45-41cd-be69-4bc9818b2b3e\r\nMay-31 19:18:01.936 [main] DEBUG nextflow.Session - Run name: big_jennings\r\nMay-31 19:18:01.936 [main] DEBUG nextflow.Session - Executor pool size: 4\r\nMay-31 19:18:01.951 [main] DEBUG nextflow.cli.CmdRun -\r\n  Version: 0.24.4 build 4341\r\n  Modified: 22-05-2017 11:18 UTC\r\n  System: Linux 4.4.0-64-generic\r\n  Runtime: Groovy 2.4.10 on OpenJDK 64-Bit Server VM 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11\r\n  Encoding: UTF-8 (UTF-8)\r\n  Process: 4001@ip-172-20-22-81 [172.20.22.81]\r\n  CPUs: 4 - Mem: 15.7 GB (14.3 GB) - Swap: 0 (0)\r\nMay-31 19:18:02.043 [main] DEBUG nextflow.Session - Work-dir: /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/work [nfs]\r\nMay-31 19:18:02.044 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /tools/nextflow/assets/SGI-Pipelines/GeneFlagging/bin\r\nMay-31 19:18:02.428 [main] DEBUG nextflow.Session - Session start invoked\r\nMay-31 19:18:02.432 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start\r\nMay-31 19:18:02.433 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/trace.tsv\r\nMay-31 19:18:02.446 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing\r\nMay-31 19:18:10.194 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution\r\nMay-31 19:18:10.197 [main] WARN  nextflow.script.ScriptBinding - Access to undefined parameter `clean_up` -- Initialise it to a default value eg. `params.clean_up = some_value`\r\nMay-31 19:18:10.234 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider\r\nMay-31 19:18:10.245 [main] DEBUG nextflow.file.FileHelper - AWS S3 config details: {region=us-west-2}\r\nMay-31 19:18:10.989 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: slurm\r\nMay-31 19:18:10.989 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'slurm'\r\nMay-31 19:18:11.000 [main] DEBUG nextflow.executor.Executor - Initializing executor: slurm\r\nMay-31 19:18:11.002 [main] INFO  nextflow.executor.Executor - [warm up] executor > slurm\r\nMay-31 19:18:11.009 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 100; pollInterval: 5s; dumpInterval: 5m\r\nMay-31 19:18:11.012 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: TaskPollingMonitor\r\nMay-31 19:18:11.013 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: slurm)\r\nMay-31 19:18:11.016 [main] DEBUG nextflow.executor.Executor - Invoke register for executor: slurm\r\nMay-31 19:18:11.017 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m\r\nMay-31 19:18:11.056 [main] DEBUG nextflow.Session - >>> barrier register (process: fetch_blast_nr_db)\r\nMay-31 19:18:11.069 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > fetch_blast_nr_db -- maxForks: 4\r\nMay-31 19:18:11.091 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: slurm\r\nMay-31 19:18:11.091 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'slurm'\r\nMay-31 19:18:11.091 [main] DEBUG nextflow.executor.Executor - Initializing executor: slurm\r\nMay-31 19:18:11.092 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m\r\nMay-31 19:18:11.092 [main] DEBUG nextflow.Session - >>> barrier register (process: fetch_pfam_db)\r\nMay-31 19:18:11.095 [Actor Thread 2] DEBUG nextflow.processor.TaskProcessor - <fetch_blast_nr_db> Poison pill arrived\r\nMay-31 19:18:11.102 [Actor Thread 1] DEBUG nextflow.processor.StateObj - <fetch_blast_nr_db> State before poison: StateObj[submitted: 1; completed: 0; poisoned: false ]\r\nMay-31 19:18:11.110 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > fetch_pfam_db -- maxForks: 4\r\nMay-31 19:18:11.128 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: slurm\r\nMay-31 19:18:11.128 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'slurm'\r\nMay-31 19:18:11.128 [main] DEBUG nextflow.executor.Executor - Initializing executor: slurm\r\nMay-31 19:18:11.129 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m\r\nMay-31 19:18:11.129 [main] DEBUG nextflow.Session - >>> barrier register (process: fetch_superfam_db)\r\nMay-31 19:18:11.136 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > fetch_superfam_db -- maxForks: 4\r\nMay-31 19:18:11.138 [Actor Thread 4] DEBUG nextflow.processor.TaskProcessor - <fetch_pfam_db> Poison pill arrived\r\nMay-31 19:18:11.138 [Actor Thread 1] DEBUG nextflow.processor.StateObj - <fetch_pfam_db> State before poison: StateObj[submitted: 1; completed: 0; poisoned: false ]\r\nMay-31 19:18:11.157 [Actor Thread 6] DEBUG nextflow.processor.TaskProcessor - <fetch_superfam_db> Poison pill arrived\r\nMay-31 19:18:11.162 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: slurm\r\nMay-31 19:18:11.162 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'slurm'\r\nMay-31 19:18:11.162 [Actor Thread 1] DEBUG nextflow.processor.StateObj - <fetch_superfam_db> State before poison: StateObj[submitted: 1; completed: 0; poisoned: false ]\r\nMay-31 19:18:11.163 [main] DEBUG nextflow.executor.Executor - Initializing executor: slurm\r\nMay-31 19:18:11.163 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m\r\nMay-31 19:18:11.164 [main] DEBUG nextflow.Session - >>> barrier register (process: order_by_gc)\r\nMay-31 19:18:11.172 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > order_by_gc -- maxForks: 4\r\nMay-31 19:18:11.197 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: slurm\r\nMay-31 19:18:11.198 [Actor Thread 8] DEBUG nextflow.processor.TaskProcessor - <order_by_gc> Poison pill arrived\r\nMay-31 19:18:11.203 [Actor Thread 1] DEBUG nextflow.processor.StateObj - <order_by_gc> State before poison: StateObj[submitted: 1; completed: 0; poisoned: false ]\r\nMay-31 19:18:11.198 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'slurm'\r\nMay-31 19:18:11.205 [main] DEBUG nextflow.executor.Executor - Initializing executor: slurm\r\nMay-31 19:18:11.205 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m\r\nMay-31 19:18:11.206 [main] DEBUG nextflow.Session - >>> barrier register (process: contig_stats)\r\nMay-31 19:18:11.224 [Actor Thread 9] DEBUG nextflow.processor.TaskProcessor - Copying to process workdir foreign file: s3:///sgi-pipeline-dev/geneflagging_data/contigs.fa\r\nMay-31 19:18:11.240 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > contig_stats -- maxForks: 4\r\nMay-31 19:18:11.269 [Actor Thread 10] DEBUG nextflow.processor.TaskProcessor - <contig_stats> Poison pill arrived\r\nMay-31 19:18:11.271 [Actor Thread 12] DEBUG nextflow.processor.TaskProcessor - Copying to process workdir foreign file: s3:///sgi-pipeline-dev/geneflagging_data/contigs.fa\r\nMay-31 19:18:11.278 [Actor Thread 1] DEBUG nextflow.processor.StateObj - <contig_stats> State before poison: StateObj[submitted: 1; completed: 0; poisoned: false ]\r\nMay-31 19:18:11.288 [Actor Thread 3] INFO  nextflow.processor.TaskProcessor - [skipping] Stored process > fetch_blast_nr_db\r\nMay-31 19:18:11.293 [Actor Thread 7] INFO  nextflow.processor.TaskProcessor - [skipping] Stored process > fetch_superfam_db\r\nMay-31 19:18:11.293 [Actor Thread 5] INFO  nextflow.processor.TaskProcessor - [skipping] Stored process > fetch_pfam_db\r\nMay-31 19:18:11.295 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <fetch_pfam_db> Sending poison pills and terminating process\r\nMay-31 19:18:11.298 [Actor Thread 13] DEBUG nextflow.processor.TaskProcessor - <fetch_superfam_db> Sending poison pills and terminating process\r\nMay-31 19:18:11.299 [Actor Thread 6] DEBUG nextflow.processor.TaskProcessor - <fetch_superfam_db> After stop\r\nMay-31 19:18:11.299 [Actor Thread 2] DEBUG nextflow.processor.TaskProcessor - <fetch_blast_nr_db> After stop\r\nMay-31 19:18:11.299 [Actor Thread 14] DEBUG nextflow.processor.TaskProcessor - <fetch_blast_nr_db> Sending poison pills and terminating process\r\nMay-31 19:18:11.301 [Actor Thread 4] DEBUG nextflow.processor.TaskProcessor - <fetch_pfam_db> After stop\r\nMay-31 19:18:11.303 [Actor Thread 1] DEBUG nextflow.Session - <<< barrier arrive (process: fetch_pfam_db)\r\nMay-31 19:18:11.304 [Actor Thread 14] DEBUG nextflow.Session - <<< barrier arrive (process: fetch_blast_nr_db)\r\nMay-31 19:18:11.308 [Actor Thread 13] DEBUG nextflow.Session - <<< barrier arrive (process: fetch_superfam_db)\r\nMay-31 19:18:11.892 [Actor Thread 9] DEBUG nextflow.executor.GridTaskHandler - Launching process > order_by_gc (1) -- work folder: /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/work/c2/2abdaf28c7057602e6edbf6dd85444\r\nMay-31 19:18:12.072 [Actor Thread 12] DEBUG nextflow.executor.GridTaskHandler - Launching process > contig_stats (1) -- work folder: /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/work/c3/9e29e54e126749a4cda9a366422f24\r\nMay-31 19:18:12.073 [Actor Thread 9] INFO  nextflow.Session - [c2/2abdaf] Submitted process > order_by_gc (1)\r\nMay-31 19:18:12.090 [Actor Thread 8] DEBUG nextflow.processor.TaskProcessor - <order_by_gc> After stop\r\nMay-31 19:18:12.181 [Actor Thread 12] INFO  nextflow.Session - [c3/9e29e5] Submitted process > contig_stats (1)\r\nMay-31 19:18:12.185 [Actor Thread 10] DEBUG nextflow.processor.TaskProcessor - <contig_stats> After stop\r\nMay-31 19:23:16.031 [Thread-3] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 2 -- first: TaskHandler[jobId: 2; id: 4; name: order_by_gc (1); status: RUNNING; exit: -; workDir: /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/work/c2/2abdaf28c7057602e6edbf6dd85444 started: 1496258571038; exited: -; ]\r\nMay-31 19:23:26.058 [Thread-3] WARN  nextflow.processor.TaskProcessor - Process `order_by_gc (1)` terminated with an error exit status (125) -- Execution is retried (1)\r\nMay-31 19:23:26.077 [pool-2-thread-1] DEBUG nextflow.executor.GridTaskHandler - Launching process > order_by_gc (1) -- work folder: /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/work/88/a767bf3eb8c490cc267e6e25ec21b2\r\nMay-31 19:23:26.201 [pool-2-thread-1] INFO  nextflow.Session - [88/a767bf] Re-submitted process > order_by_gc (1)\r\nMay-31 19:23:26.203 [Thread-3] WARN  nextflow.processor.TaskProcessor - Process `contig_stats (1)` terminated with an error exit status (125) -- Execution is retried (1)\r\nMay-31 19:23:26.220 [pool-2-thread-2] DEBUG nextflow.executor.GridTaskHandler - Launching process > contig_stats (1) -- work folder: /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/work/23/d2a1c73986ef90e0e685cf32def4df\r\nMay-31 19:23:26.329 [pool-2-thread-2] INFO  nextflow.Session - [23/d2a1c7] Re-submitted process > contig_stats (1)\r\nMay-31 19:24:01.085 [Thread-3] ERROR nextflow.processor.TaskProcessor - Error executing process > 'order_by_gc (1)'\r\n\r\nCaused by:\r\n  Process `order_by_gc (1)` terminated with an error exit status (125)\r\n\r\nCommand executed [/tools/nextflow/assets/SGI-Pipelines/GeneFlagging/templates/orderFastaByGc.py]:\r\n\r\n  #!/usr/bin/env python\r\n\r\n  from gene_annotation.fasta.sortFastaByGc import sortFastaByGc\r\n\r\n\r\n  def main():\r\n\r\n      inputFile = 'contigs.fa'\r\n      outputFile = 'contigsByGc.fa'\r\n\r\n      sortFastaByGc(inputFile, outputFile)\r\n\r\n\r\n  if __name__ == '__main__':\r\n      main()\r\n\r\nCommand exit status:\r\n  125\r\n\r\nCommand output:\r\n  (empty)\r\n\r\nCommand error:\r\n  Unable to find image 'dockreg-dev01.awsv.l.synthgeno.global/compbio/gene_annotation:0.1.0' locally\r\n  docker: Error response from daemon: Get https://dockreg-dev01.awsv.l.synthgeno.global/v1/_ping: dial tcp 172.17.0.30:443: getsockopt: no route to host.\r\n  See 'docker run --help'.\r\n\r\nWork dir:\r\n  /mnt/efs/ubuntu/nextflow.run.47342766-b597-4f67-8726-82c4e9a0d267/work/88/a767bf3eb8c490cc267e6e25ec21b2\r\n\r\nTip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`\r\nMay-31 19:24:01.088 [Actor Thread 13] DEBUG nextflow.processor.TaskProcessor - <order_by_gc> Sending poison pills and terminating process\r\nMay-31 19:24:01.088 [Actor Thread 13] DEBUG nextflow.Session - <<< barrier arrive (process: order_by_gc)\r\nMay-31 19:24:01.097 [Thread-3] DEBUG nextflow.Session - Session aborted -- Cause: Process `order_by_gc (1)` terminated with an error exit status (125)\r\nMay-31 19:24:01.119 [Actor Thread 13] DEBUG nextflow.processor.TaskProcessor - <contig_stats> Sending poison pills and terminating process\r\nMay-31 19:24:01.120 [Actor Thread 13] DEBUG nextflow.Session - <<< barrier arrive (process: contig_stats)\r\nMay-31 19:24:01.122 [Thread-3] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm)\r\n~\r\n```"},"comment":{"url":"https://api.github.com/repos/nextflow-io/nextflow/issues/comments/310660498","html_url":"https://github.com/nextflow-io/nextflow/issues/361#issuecomment-310660498","issue_url":"https://api.github.com/repos/nextflow-io/nextflow/issues/361","id":310660498,"user":{"login":"pditommaso","id":816968,"avatar_url":"https://avatars3.githubusercontent.com/u/816968?v=3","gravatar_id":"","url":"https://api.github.com/users/pditommaso","html_url":"https://github.com/pditommaso","followers_url":"https://api.github.com/users/pditommaso/followers","following_url":"https://api.github.com/users/pditommaso/following{/other_user}","gists_url":"https://api.github.com/users/pditommaso/gists{/gist_id}","starred_url":"https://api.github.com/users/pditommaso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pditommaso/subscriptions","organizations_url":"https://api.github.com/users/pditommaso/orgs","repos_url":"https://api.github.com/users/pditommaso/repos","events_url":"https://api.github.com/users/pditommaso/events{/privacy}","received_events_url":"https://api.github.com/users/pditommaso/received_events","type":"User","site_admin":false},"created_at":"2017-06-23T13:06:41Z","updated_at":"2017-06-23T13:06:41Z","body":"There's at least a task which does not start as expected. These are the entry in the log: \r\n\r\n```\r\nJun-21 22:43:07.301 [Pending tasks thread] DEBUG nextflow.executor.GridTaskHandler - Launching process > blast_clusters_parse (173) -- work folder: /mnt/efs/nextflow/run.b0b42c39-12cf-411a-b4d5-bd8730d0fe59/work/bc/0709426de81ca3dcb79e4827d78605\r\nJun-21 22:43:07.419 [Pending tasks thread] INFO  nextflow.Session - [bc/070942] Submitted process > blast_clusters_parse (173)\r\n: \r\nJun-21 22:56:02.200 [Running tasks thread] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 12 -- first: TaskHandler[jobId: 14953; id: 4834; name: blast_clusters_parse (173); status: SUBMITTED; exit: -; workDir: /mnt/efs/nextflow/run.b0b42c39-12cf-411a-b4d5-bd8730d0fe59/work/bc/0709426de81ca3dcb79e4827d78605 started: -; exited: -; ]\r\n:\r\nJun-21 23:41:02.737 [Running tasks thread] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 2 -- first: TaskHandler[jobId: 14953; id: 4834; name: blast_clusters_parse (173); status: SUBMITTED; exit: -; workDir: /mnt/efs/nextflow/run.b0b42c39-12cf-411a-b4d5-bd8730d0fe59/work/bc/0709426de81ca3dcb79e4827d78605 started: -; exited: -; ]\r\n```\r\n\r\nThe task remains in `SUBMITTED` status, this means the it has been submitted for execution to SLURM which assigned the job-id `14953`. When a task is executed the first thing the `.command.run` wrapper does is to create the `.command.begin` marker. NF uses this file or the `.exitcode` file to detect that the job has started. Hence, there could be three possibilities: \r\n\r\n1) SLURM for some reason didn't execute the task\r\n2) The task was executed but it failed immediately (but at least the `.exitcode` should exist)\r\n3) For some reason the files on the file system where not written / got lost. \r\n\r\nHave you any way to troubleshot this conditions? Does SLURM an history/accounting of the jobs executed?  "}}
{"actor":{"display_login":"pditommaso"}}