Reproducibility and experiment tracking

Three weeks in you will have forty model variants and one number in a slide deck, and you will not know which variant produced it. Fix that on day one; retrofitting it is miserable.

Determinism first

SEED = 42

import random, numpy as np
random.seed(SEED)
np.random.seed(SEED)

train_test_split(..., random_state=SEED)
RandomForestClassifier(random_state=SEED)
KFold(shuffle=True, random_state=SEED)

Pass random_state to every object that accepts it. Not for good luck — so that a score change is caused by your change, and not by a different shuffle.

Configuration out of the code

# config.yaml
seed: 42
data:
  raw: data/raw/students_2026_09_01.csv
  label_cutoff: 50
split:
  test_size: 0.2
  strategy: group
  group_column: student_id
model:
  kind: hist_gradient_boosting
  learning_rate: 0.05
  max_leaf_nodes: 31
metric: recall_at_precision_50

import yaml, json, hashlib, subprocess, datetime as dt
from pathlib import Path

cfg = yaml.safe_load(Path("config.yaml").read_text())

Hyperparameters buried in the middle of a notebook cannot be diffed, reviewed or reproduced. In a config file, git log becomes a record of every experiment you ran.

Record the run

def data_hash(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()[:12]


def git_commit():
    try:
        return subprocess.check_output(
            ["git", "rev-parse", "--short", "HEAD"], text=True).strip()
    except Exception:
        return "unknown"


run = {
    "when": dt.datetime.now().isoformat(timespec="seconds"),
    "commit": git_commit(),
    "config": cfg,
    "data_sha": data_hash(cfg["data"]["raw"]),
    "rows": len(df),
    "metrics": {"recall": 0.68, "precision": 0.52, "roc_auc": 0.81},
}

Path("runs").mkdir(exist_ok=True)
Path(f"runs/{run['when'].replace(':', '')}.json").write_text(
    json.dumps(run, indent=2))

Forty lines of code and you can answer the question that otherwise sinks projects: which code, which data and which settings produced this number? The data hash matters most — "the same file" changes underneath you more often than anyone expects.

Tools, when the JSON files get unwieldy

import mlflow

mlflow.set_experiment("at-risk-student")
with mlflow.start_run():
    mlflow.log_params(cfg["model"])
    mlflow.log_metric("recall", 0.68)
    mlflow.log_artifact("outputs/confusion_matrix.png")
    mlflow.sklearn.log_model(model, "model")

MLflow, Weights & Biases and DVC all do this properly, with a UI for comparison. Start with JSON files; adopt a tool when you have enough runs to need searching, not before.

Version the data, not just the code

• Never overwrite a raw extract. Name files with the date they were pulled: students_2026_09_01.csv.
• Store the hash of every input in the run record.
• For a database source, save the exact query and a snapshot, or use a timestamped table. "SELECT * FROM students" run twice returns different data.
• Keep raw data out of git. Use DVC, a bucket, or a shared drive with a path recorded in config.

The environment

pip freeze > requirements.txt        # exact versions

# or a lock file if you use poetry / uv / pipenv

# Dockerfile: the strongest guarantee
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "-m", "src.train"]

A scikit-learn pickle is not guaranteed to load in a different library version. Record versions beside every saved model, and put them in the container image if the model will be served.

The reproducibility test

Clone your repository into a clean directory, follow your own README, and regenerate a specific past number. If you cannot, your project is not reproducible — and the person who will discover that is usually an auditor, a regulator or your successor.

Try it yourself

Move every hyperparameter and path into config.yaml, add the run-recording function, and generate five runs with different settings. Then re-run your best configuration with five different seeds and report mean and standard deviation. Finally do the clean-clone test and fix whatever breaks.