SLURM for Bioinformatics: Submitting Cluster Jobs You Can Walk Away From

Quick answer: SLURM is the job scheduler on most university and institute clusters: you describe what a job needs (time, memory, cores, partition) in a short submit script, hand it over with sbatch, and SLURM runs it when the resources are free — across hundreds of nodes, in parallel with everyone else’s work. The working skill is small: write a correct #SBATCH header, submit, watch with squeue, read failures with sacct, and use array jobs to fan one script across a whole cohort overnight instead of babysitting a login shell.

Companion to the Linux for bioinformatics guide on this blog — SSH, modules, and file systems are the ground this runs on.

Why you cannot just run the pipeline on the login node

Login nodes are shared front doors: one heavy job there is poor etiquette, and on most clusters it is grounds for having your account suspended. A cluster’s point is that the scheduler arbitrates scarce resources fairly — cores, memory, walltime — across every user. The trade is explicit: you declare what you need, wait in the queue, and in return get parallel scale you could never buy for one desk.

The submit script

Everything SLURM needs lives in comments at the top; the rest is an ordinary bash script (the same defensive preamble applies):

#!/usr/bin/env bash
#SBATCH --job-name=rnaseq
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err
#SBATCH --partition=compute
#SBATCH --time=04:00:00
#SBATCH --mem=16G
#SBATCH --cpus-per-task=8
#SBATCH --array=1-24
#SBATCH --mail-type=END,FAIL

set -euo pipefail
module load bwa samtools multiqc
conda activate rnaseq

sid=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt)
echo "host=$(hostname) sample=$sid"

fastp -i data/${sid}_1.fq.gz -I data/${sid}_2.fq.gz \
      -o out/${sid}_1.clean.fq.gz -O out/${sid}_2.clean.fq.gz --thread 8
bwa mem -t 8 ref.fa out/${sid}_1.clean.fq.gz out/${sid}_2.clean.fq.gz \
  | samtools sort -@ 4 -o out/${sid}.bam -
samtools index out/${sid}.bam
multiqc out/${sid}.fastp.json -o logs/

The header fields worth understanding rather than copying:

• --time — the wall clock limit. Exceed it and the job is killed with CANCELLED at your declared limit; the number-one cause of “my job died mysteriously” is asking for 2 hours and needing 5.
• --mem — memory per job (a whole node’s worth if shared). Guessing low is the second killer: the OOM killer terminates the process, and sacct will show OUT_OF_MEMORY.
• --cpus-per-task — cores handed to this job. Your tool must actually be told to use them (-t 8, --thread 8); requesting 16 cores and running a single-threaded command wastes the allocation and your queue position.
• --array=1-24 — one script, 24 independent jobs, one line in the queue. SLURM_ARRAY_TASK_ID selects the sample. This is the single highest-leverage flag in SLURM.
• --partition — the queue (compute, highmem, gpu). Names are site-specific; ask or read the cluster docs before guessing.
• Logs with %x_%j — job name plus job ID, so output files never overwrite each other.

The five commands you will actually type

sbatch script.sh          # submit
squeue -u $USER           # what of mine is pending/running
scancel 123456            # kill a job (or a whole array: 123456_[1-24])
sacct -j 123456           # history: state, exit code, max RSS, elapsed
scontrol show job 123456  # why is this PENDING? (resources, reservation)

Reading a pending job is a skill in itself: scontrol reports the reason — resources, priority, requested-node features — and “Priority” simply means waiting, while “Resources” means your requested combination (say 32 cores and 128 GB) is rarer than you assumed. Requests should be justified by measurement: run once, check sacct‘s max RSS and elapsed time, then declare those numbers plus a margin instead of a superstition.

Dependencies, logs, and finishing without babysitting

• Chain stages with dependency flags: sbatch --dependency=afterok:123456 align.sh — alignment waits for QC and fails if QC fails.
• Hold-and-release: submit with sbatch --hold, inspect with scontrol, then scontrol release <jobid> when satisfied; cheaper than cancelling mistakes.
• Capture failures early. Check sacct states: FAILED with exit 1 is your script’s own error (read the .err file — set -euo pipefail surfaces it there), CANCELLED at exactly your time limit is walltime, OUT_OF_MEMORY is the memory request.
• Scratch, not home. Home directories are small and backed up; intermediates (BAMs, fastqs) go to the cluster’s scratch/project space, which is usually not backed up — so final results get copied back.

Where clusters and pipelines meet

SLURM decides where and when a command runs; a workflow manager decides what depends on what. Snakemake can emit sbatch scripts per rule and Nextflow has a native SLURM executor, which means your pipeline gets cluster scale without you writing --array bookkeeping by hand — the natural next step once the batch crosses a few dozen samples, as covered in the workflows article. Learn plain sbatch first: when the cluster returns an obscure error, it is the only way to know whether the scheduler or your pipeline is at fault.

Key takeaways

• Never run heavy jobs on the login node — submit through the scheduler; it is the difference between a shared resource and a suspended account.
• The #SBATCH header is a contract: time, memory, and cores must match what the job actually uses, verified afterwards with sacct.
• --array turns one script into a cohort-wide batch; SLURM_ARRAY_TASK_ID maps each task to a sample.
• Measure one run (max RSS, elapsed), then request those numbers plus margin — over-requests stall the queue, under-requests get killed.
• Chain stages with --dependency=afterok, keep per-job logs named by job ID, and use scratch for intermediates.
• When hand-rolled arrays become dependency puzzles, let Snakemake or Nextflow drive SLURM for you.

Challenge

Take the per-sample script you wrote in the bash tutorial, add a correct #SBATCH header, and submit it as an array job over your sample list. Then: check squeue while it runs, force one failure (a deliberately wrong path) and find the cause in its .err file, and after completion read sacct -j <id> to confirm the actual memory and runtime — then resubmit the header with those measured numbers. That loop of submit, measure, adjust is the whole craft of cluster work.

Want guided practice running analyses on real HPC infrastructure? Ampersand Academy offers one-to-one mentoring in bioinformatics — part of Mahadhi, a development and digital marketing company.

FAQ