Site icon Ampersand Tutorials

Bash Scripting for Bioinformatics: Automate the Same Steps Across Every Sample

Quick answer: A bash script is a saved sequence of terminal commands with arguments, loops, and error handling — the thing you write when you realize you have retyped the same six commands for sample 7. In bioinformatics the payoff is immediate: one script turns twenty FASTQ files into twenty trimmed, aligned, QC-reported results while you do something else. The essential craft is small and learnable: quote your variables, fail loudly (set -euo pipefail), loop over files instead of copying commands, log everything, and know the point where shell scripting stops scaling and a workflow manager should take over.

Companion to the Linux for bioinformatics guide on this blog — this article assumes you can already navigate, pipe, and grep your way around a terminal.

Why a script beats terminal history

Your shell history is a log of intent, not a pipeline: it has no error handling, no parameters, and no record of which input produced which output. Three properties make a script the correct tool:

The habit worth building early: never run a per-sample command you have typed twice — put the third run into a script.

The skeleton that saves you at 2 a.m.

Every script you write in this field should start with the same defensive preamble:

#!/usr/bin/env bash
set -euo pipefail
# -e: exit on any failed command
# -u: error on unset variables (catches typos immediately)
# -o pipefail: a failure anywhere in a pipe fails the whole pipe

usage() { echo "Usage: $0 <reads_dir> <outdir>" >&2; exit 1; }
[[ $# -eq 2 ]] || usage
reads_dir=$1; outdir=$2
mkdir -p "$outdir"

log="$outdir/run_$(date +%Y%m%d_%H%M%S).log"
exec > >(tee -a "$log") 2>&1          # everything also lands in a log file

echo "host=$(hostname) date=$(date -Is) args=$*"

Three details there do most of the protecting: set -euo pipefail turns silent breakage into an immediate stop; the usage guard prevents running with wrong arguments; and exec > tee means that when a colleague asks what you actually ran six weeks ago, the answer is a file, not memory.

Loops: the core pattern

The workhorse form processes one file per iteration. Always quote variables — a space in a filename otherwise splits one argument into two:

for r1 in "$reads_dir"/*_1.fastq.gz; do
    r2=${r1/_1/_2}
    base=$(basename "$r1" _1.fastq.gz)

    echo "--- $base"
    fastp -i "$r1" -I "$r2" \
          -o "$outdir/${base}_1.clean.fq.gz" \
          -O "$outdir/${base}_2.clean.fq.gz" \
          --thread 8 --json "$outdir/${base}.fastp.json"

    bwa mem -t 8 ref.fa \
          "$outdir/${base}_1.clean.fq.gz" "$outdir/${base}_2.clean.fq.gz" \
      | samtools sort -@ 4 -o "$outdir/${base}.bam" -
    samtools index "$outdir/${base}.bam"
done
multiqc "$outdir" -o "$outdir"

Patterns worth internalizing:

Doing the same thing to every line: awk, grep, xargs

Much bioinformatics scripting is text surgery on large files, and the shell already ships the tools:

# rows whose 5th column (quality) is above 30
awk -F'\t' '$5 > 30' results.tsv > results.q30.tsv

# sample IDs from a column, one per line, ready for a job array
cut -f1 meta.tsv | tail -n +2 | sort -u > samples.txt

# run an 8-at-a-time conversion across a list
cat samples.txt | xargs -P 8 -I{} sh -c 'samtools index {}.bam'

Use grep/awk for selection, sort/uniq -c for quick counts, and xargs -P (or GNU parallel for fancier scheduling) for bounded concurrency. Two caveats: -P runs processes that may each want several threads, so budget threads × jobs against the machine, and sh -c '...' strings need their own quoting discipline.

Knowing when shell stops being the right tool

Shell is superb at sequencing existing tools and poor at expressing dependencies, resumability, or cloud execution. The tell-tale signs you have outgrown it:

That is the handover point to a workflow manager — Snakemake’s rule model or Nextflow’s channels turn each of your script’s stages into an explicit, resumable dependency graph — which the workflows article on this blog covers in full. The good news: well-written scripts are exactly what those tools wrap, so none of this work is wasted.

Key takeaways

Challenge

Take the last analysis you ran by copy-pasting, and script it end to end with the defensive preamble: arguments for input and output directories, set -euo pipefail, a timestamped log, and a loop over samples with per-sample failure capture. Then deliberately break one input filename and confirm the script stops (or logs a failure) instead of producing plausible garbage — a pipeline you have never watched fail is a pipeline you cannot trust.

Want help turning ad-hoc commands into reproducible pipelines? Ampersand Academy offers one-to-one mentoring in bioinformatics — part of Mahadhi, a development and digital marketing company.

FAQ

What does set -euo pipefail actually change?

Without it, a failed command in the middle of a pipe is invisible and the script keeps going. With it, the script exits on the first error, errors on unset variables, and treats a failure anywhere in a pipeline as a pipeline failure.

Why do I need quotes around variables like $reads_dir?

Unquoted variables undergo word splitting and glob expansion, so a path containing a space becomes two arguments and an unquoted wildcard can expand to unexpected files. Quoting keeps one value as one argument.

How do I run one script step across hundreds of samples quickly?

Wrap the per-sample command in a for loop reading a file list, or feed the list to xargs -P N for concurrent execution. Cap the concurrency by multiplying it by each tool’s thread count so you do not oversubscribe the machine.

My script failed halfway. Can I just rerun it?

Only if the steps are idempotent. Safer patterns: write to temporary files and move them into place on success, keep a completed-samples list and skip those, or move the pipeline into Snakemake or Nextflow, which re-runs only stale outputs.

Shell or Python — which should I use?

Shell excels at gluing existing command-line tools with pipes and loops; Python is better for parsing structured formats, arithmetic, and anything with real logic. A common arrangement is a shell driver calling small Python helpers.

How do I make my scripts runnable on the lab server unchanged?

Avoid absolute paths inside the script, take them as arguments or variables, use env in the shebang instead of a hardcoded interpreter path, and pin tool versions with a conda environment file shared alongside the script.

When exactly should I switch to a workflow manager?

When you find yourself writing skip-if-exists logic, nested loops with branches, or manual bookkeeping about which outputs a failure invalidated. Those are dependency-graph problems, and Snakemake or Nextflow exists to hold the graph for you.

Exit mobile version