Jupyter Notebooks for Bioinformatics: Reproducible Interactive Analysis

A notebook is a document that mixes code, its output, and your reasoning in one scrollable page. For exploratory bioinformatics — checking whether a count matrix looks sane, trying three normalisation strategies, plotting a distribution before deciding what to do — this is a better fit than any script, because you see the result next to the code that produced it.

It is also the format most likely to produce results you cannot defend, because notebooks remember state in a way that scripts do not. This tutorial sets up notebooks properly and shows the discipline that keeps them trustworthy.

Install Jupyter inside a project environment

Never install Jupyter into your base environment. Make an environment for the project, install Jupyter there, and register it as a named kernel. That way the notebook, the packages and the Python version travel together.

conda create -n amprna -c conda-forge python=3.12 jupyterlab
conda activate amprna

# Register this environment as a selectable kernel
python -m ipykernel install --user \
  --name amprna \
  --display-name "AmpRNA (Python 3.12)"

jupyter lab

The --user flag writes the kernel spec into your home directory rather than the environment, which is what makes it visible to every JupyterLab instance you launch. Without --display-name you get a list of python3, python3, python3 and no idea which is which.

Adding the analysis libraries

conda install -c conda-forge -c bioconda \
  pandas numpy matplotlib seaborn \
  biopython pysam scanpy

# Confirm which Python the kernel is actually using
jupyter kernelspec list

If a notebook cannot import a package that is definitely installed, the usual cause is a kernel pointing at a different environment than the terminal you installed into. import sys; print(sys.executable) in the notebook settles the argument immediately.

Reading real data in cells

Start every notebook with the same three cells: imports, configuration, and a data sanity check. The sanity check is the one people skip and later regret.

import pandas as pd
import numpy as np
from Bio import SeqIO

pd.set_option("display.max_columns", 50)
counts = pd.read_csv("counts/matrix.tsv", sep="\t", index_col=0)

print(counts.shape)          # (genes, samples)
print(counts.head(3))
counts.isna().sum().sum()    # how many missing values?

That last line catches the most common silent failure in omics analysis: a join that produced thousands of NaNs because sample names had trailing whitespace. Seeing it in cell one costs two seconds; discovering it after a differential expression run costs a day.

Sequence files without writing a parser

from Bio import SeqIO

records = list(SeqIO.parse("data/sample.fasta", "fasta"))
print(len(records), records[0].id, len(records[0].seq))

lengths = np.array([len(r.seq) for r in records])
print(f"N50: {np.median(lengths):.0f}  max: {lengths.max()}  total: {lengths.sum():,} bp")

For FASTQ files larger than a few hundred megabytes, do not load them into a notebook at all. Compute the summary on the command line and read only the summary:

%%bash
seqkit stats -T data/*.fastq.gz > qc/seqkit_stats.tsv

stats = pd.read_csv("qc/seqkit_stats.tsv", sep="\t")
stats[["file", "num_seqs", "sum_len", "avg_len", "Q20(%)", "Q30(%)"]]

Magics worth knowing

%who is the quiet hero here. When you suspect a variable is not what you think it is, %who tells you what actually exists in the namespace right now, rather than what the notebook’s scrollback implies.

The hidden-state problem, and how to stay honest

Notebook cells can be run in any order, any number of times. A result that appears on screen may depend on a variable you overwrote four cells ago and forgot about. This is the single reason reviewers distrust notebooks, and it is entirely fixable.

The rule that solves it: before showing a result to anyone, use Kernel → Restart & Run All. If the notebook does not produce the same output from a clean kernel, the output was never real. This takes one keystroke and catches nearly every instance of hidden state.

# From the terminal, the same check without the mouse
jupyter nbconvert --to notebook --execute \
  --ExecutePreprocessor.timeout=900 \
  analysis.ipynb --output analysis_verified.ipynb

Running that in CI or before submission is the professional version of the habit. If it succeeds, the notebook is reproducible from top to bottom by construction.

Kill the memory leaks

Notebook kernels hold every object you have ever created. A 2 GB count matrix from an hour ago is still resident while you wonder why the machine is swapping.

del big_matrix
import gc; gc.collect()

# Check what is actually large
import sys
sorted(((k, sys.getsizeof(v)) for k, v in globals().items()),
       key=lambda x: -x[1])[:5]

Making notebooks reviewable and versionable

Notebooks are JSON documents containing outputs, which makes them terrible in Git: a re-run changes thousands of lines of base64 image data. Two tools fix this.

# Strip outputs before committing
pip install nbstripout
nbstripout --install          # sets up a Git filter for this repo

# Or keep notebooks and scripts in sync via jupytext
pip install jupytext
jupytext --set-formats ipynb,py:percent analysis.ipynb

nbstripout leaves the code and removes outputs before each commit, so diffs show only what you changed. jupytext goes further and maintains a paired .py file, which is what you want when the notebook must eventually become a pipeline step.

Exporting something a collaborator can read

jupyter nbconvert --to html analysis.ipynb
jupyter nbconvert --to pdf analysis.ipynb   # needs LaTeX installed

HTML is the reliable format: it embeds the plots and needs no LaTeX installation. Send that alongside the .ipynb file and the environment file, never instead of them.

When not to use a notebook

Notebooks are excellent for exploration and poor for production. Once an analysis needs to run on every sample, move the logic into a script or a workflow step and keep the notebook as the report layer that reads its outputs. The tell is copy-paste: if you are pasting the same cell for the fifth sample, that cell is a function that belongs in a module.

That split — notebook for investigation, script for repetition, report for communication — is the same structure used in every well-run analysis group. If you want to see how it fits with environments, containers and workflow managers across a whole project, Ampersand Academy covers the workflow end to end.

Practice

1. Create a conda environment, register it as a named kernel, and confirm the kernel path differs from your base Python.
2. Load a count matrix and print its shape, missing-value count and per-sample library sizes in one cell.
3. Run seqkit stats through %%bash and read the result into pandas.
4. Deliberately create hidden state — compute a value, change an input, re-run only the plotting cell — then fix it with Restart & Run All.

Related tutorials

• Python for Bioinformatics: Easy Getting-Started Guide — the Python fundamentals these notebooks run on
• Pandas for Bioinformatics: Reading, Filtering, and Summarizing Omics Tables — table work inside notebook cells, applied to omics data
• Bulk RNA-Seq Data Analysis: From FASTQ to a Trustworthy Differential-Expression Result — a complete analysis you can work through interactively