User documentation for the PLEIADES cluster at the University of Wuppertal
shellcheck.net is very useful to highlight problems with your shell scripts! You can use it to check and improve your job scripts.
Every job submission in Slurm introduces some overhead to the batch system. If you have many short jobs of the same kind, e.g. 2000 x 30 minutes, you should combine your workload in fewer submission scripts or consider using Slurms job arrays (check our example job). This way you bundle all of these jobs in a single job submission, but still can treat the items individually as job steps.
Please try to estimate a maximum execution time and set your job time limits accordingly.
-t
, shorter than the partitions maximum timesbatch -d <jobid>
to set a job dependency to the original job, to continue processinglong
partition allows for longer jobs, but there are a couple of risks involved:
Previously, we used SGE to submit batch jobs.
Since many concepts are similar between these kind of tools, you just have to look up the new tool names in many cases. This rosetta stone, provided by SchedMD is a good starting point.