Quick answer: A bash script is a saved sequence of terminal commands with arguments, loops, and error handling — the thing you write when you realize you have retyped the same six commands for sample 7. In bioinformatics the payoff is immediate: one script turns twenty FASTQ files into twenty trimmed, aligned, QC-reported results while you do something else. The essential craft is small and learnable: quote your variables, fail loudly (set -euo pipefail), loop over files instead of copying commands, log everything, and know the point where shell scripting stops scaling and a workflow manager should take over.
Companion to the Linux for bioinformatics guide on this blog — this article assumes you can already navigate, pipe, and grep your way around a terminal.
Table of Contents
Why a script beats terminal history
Your shell history is a log of intent, not a pipeline: it has no error handling, no parameters, and no record of which input produced which output. Three properties make a script the correct tool:
- Reproducibility. The script is the methods section, executable. Same commands, same order, every run.
- Safety. With the right settings, a script stops at the first failure instead of cheerfully running steps 3 through 9 against nothing.
- Parallelism. Wrapping a per-sample command in a loop (or
xargs -P) turns a two-hour serial afternoon into a ten-minute batch.
The habit worth building early: never run a per-sample command you have typed twice — put the third run into a script.
The skeleton that saves you at 2 a.m.
Every script you write in this field should start with the same defensive preamble:
#!/usr/bin/env bash
set -euo pipefail
# -e: exit on any failed command
# -u: error on unset variables (catches typos immediately)
# -o pipefail: a failure anywhere in a pipe fails the whole pipe
usage() { echo "Usage: $0 <reads_dir> <outdir>" >&2; exit 1; }
[[ $# -eq 2 ]] || usage
reads_dir=$1; outdir=$2
mkdir -p "$outdir"
log="$outdir/run_$(date +%Y%m%d_%H%M%S).log"
exec > >(tee -a "$log") 2>&1 # everything also lands in a log file
echo "host=$(hostname) date=$(date -Is) args=$*"
Three details there do most of the protecting: set -euo pipefail turns silent breakage into an immediate stop; the usage guard prevents running with wrong arguments; and exec > tee means that when a colleague asks what you actually ran six weeks ago, the answer is a file, not memory.
Loops: the core pattern
The workhorse form processes one file per iteration. Always quote variables — a space in a filename otherwise splits one argument into two:
for r1 in "$reads_dir"/*_1.fastq.gz; do
r2=${r1/_1/_2}
base=$(basename "$r1" _1.fastq.gz)
echo "--- $base"
fastp -i "$r1" -I "$r2" \
-o "$outdir/${base}_1.clean.fq.gz" \
-O "$outdir/${base}_2.clean.fq.gz" \
--thread 8 --json "$outdir/${base}.fastp.json"
bwa mem -t 8 ref.fa \
"$outdir/${base}_1.clean.fq.gz" "$outdir/${base}_2.clean.fq.gz" \
| samtools sort -@ 4 -o "$outdir/${base}.bam" -
samtools index "$outdir/${base}.bam"
done
multiqc "$outdir" -o "$outdir"
Patterns worth internalizing:
- Parameterize, don’t hardcode. Reference paths and thread counts become variables (or better, arguments) at the top — the script then survives the move to another machine or another genome.
- Derive names, don’t list them.
${r1/_1/_2}andbasenamebuild partner filenames; hand-written filename lists are where sample 14 silently gets skipped. - Fail per sample, not per run. For batches where one bad file should not kill the night, wrap the body in
if ! { ... }; then echo "FAILED: $base"; fiand keep a failure list to re-run. - Mind the glob.
for f in *.fastq.gzwith no matches iterates once over the literal pattern;shopt -s nullglobprevents that classic bug.
Doing the same thing to every line: awk, grep, xargs
Much bioinformatics scripting is text surgery on large files, and the shell already ships the tools:
# rows whose 5th column (quality) is above 30
awk -F'\t' '$5 > 30' results.tsv > results.q30.tsv
# sample IDs from a column, one per line, ready for a job array
cut -f1 meta.tsv | tail -n +2 | sort -u > samples.txt
# run an 8-at-a-time conversion across a list
cat samples.txt | xargs -P 8 -I{} sh -c 'samtools index {}.bam'
Use grep/awk for selection, sort/uniq -c for quick counts, and xargs -P (or GNU parallel for fancier scheduling) for bounded concurrency. Two caveats: -P runs processes that may each want several threads, so budget threads × jobs against the machine, and sh -c '...' strings need their own quoting discipline.
Knowing when shell stops being the right tool
Shell is superb at sequencing existing tools and poor at expressing dependencies, resumability, or cloud execution. The tell-tale signs you have outgrown it:
- You are maintaining nested loops with conditional re-run logic to skip completed steps.
- The pipeline has branches (different references per species, optional steps) encoded as commented-out lines.
- A run fails at step 6 and you must reason manually about which outputs are now stale.
That is the handover point to a workflow manager — Snakemake’s rule model or Nextflow’s channels turn each of your script’s stages into an explicit, resumable dependency graph — which the workflows article on this blog covers in full. The good news: well-written scripts are exactly what those tools wrap, so none of this work is wasted.
Key takeaways
- Start every script with
set -euo pipefail, an argument check, and a tee’d log file — those ten lines prevent most 2 a.m. debugging. - Quote every variable; derive filenames programmatically; never hand-maintain a list of samples inside the loop.
- Parameterize references, threads, and paths at the top so the script survives contact with new data.
- Constrain parallelism (
xargs -P) and multiply its jobs by each tool’s own thread count before choosing a number. - Capture the environment too: record tool versions (or a conda environment) next to the script, or reproducibility stops at the commands.
- When branches, caching, or resume logic appear, move the script under Snakemake or Nextflow instead of growing more nested loops.
Challenge
Take the last analysis you ran by copy-pasting, and script it end to end with the defensive preamble: arguments for input and output directories, set -euo pipefail, a timestamped log, and a loop over samples with per-sample failure capture. Then deliberately break one input filename and confirm the script stops (or logs a failure) instead of producing plausible garbage — a pipeline you have never watched fail is a pipeline you cannot trust.
Want help turning ad-hoc commands into reproducible pipelines? Ampersand Academy offers one-to-one mentoring in bioinformatics — part of Mahadhi, a development and digital marketing company.
FAQ
What does set -euo pipefail actually change?
Without it, a failed command in the middle of a pipe is invisible and the script keeps going. With it, the script exits on the first error, errors on unset variables, and treats a failure anywhere in a pipeline as a pipeline failure.
Why do I need quotes around variables like $reads_dir?
Unquoted variables undergo word splitting and glob expansion, so a path containing a space becomes two arguments and an unquoted wildcard can expand to unexpected files. Quoting keeps one value as one argument.
How do I run one script step across hundreds of samples quickly?
Wrap the per-sample command in a for loop reading a file list, or feed the list to xargs -P N for concurrent execution. Cap the concurrency by multiplying it by each tool’s thread count so you do not oversubscribe the machine.
My script failed halfway. Can I just rerun it?
Only if the steps are idempotent. Safer patterns: write to temporary files and move them into place on success, keep a completed-samples list and skip those, or move the pipeline into Snakemake or Nextflow, which re-runs only stale outputs.
Shell or Python — which should I use?
Shell excels at gluing existing command-line tools with pipes and loops; Python is better for parsing structured formats, arithmetic, and anything with real logic. A common arrangement is a shell driver calling small Python helpers.
How do I make my scripts runnable on the lab server unchanged?
Avoid absolute paths inside the script, take them as arguments or variables, use env in the shebang instead of a hardcoded interpreter path, and pin tool versions with a conda environment file shared alongside the script.
When exactly should I switch to a workflow manager?
When you find yourself writing skip-if-exists logic, nested loops with branches, or manual bookkeeping about which outputs a failure invalidated. Those are dependency-graph problems, and Snakemake or Nextflow exists to hold the graph for you.
Related tutorials
- Git and GitHub for Bioinformatics: Version Control for Analysis Projects — tracking script changes and recovering a working version
- NCBI EDirect Tutorial: Query PubMed, GenBank and SRA from the Command Line — automating database queries instead of copying accessions by hand

