Serving a model: from pickle to API

A model nobody can call is a research artefact. There are three ways to deliver predictions, and picking the right one saves months.

Choose the delivery mode first

• Batch — a scheduled job scores everything and writes to a table the application reads. Start here. Simple, testable, no latency requirement, and it covers most business problems including the at-risk-student case.
• Real-time API — needed when the input only exists at request time (fraud on this transaction, this search query).
• Embedded — the model runs inside the client. Needed offline or where data must not leave the device.

Batch scoring

import joblib, pandas as pd, datetime as dt

def score_batch(model_path="model.joblib", out="data/scores"):
    model = joblib.load(model_path)
    df = load_current_students()               # your own loader

    proba = model.predict_proba(df[FEATURES])[:, 1]

    out_df = pd.DataFrame({
        "student_id": df["student_id"].values,
        "risk": proba.round(4),
        "flagged": proba > THRESHOLD,
        "model_version": MODEL_VERSION,
        "scored_at": dt.datetime.now(),
    })
    out_df.to_parquet(f"{out}/scores_{dt.date.today()}.parquet")
    return out_df

Write the probability, the threshold decision and the model version. Six months later, when someone asks why a student was flagged in March, the version column is the only thing that lets you answer.

A real-time API with FastAPI

# app.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import joblib, pandas as pd, logging

MODEL_VERSION = "2026.09.1"
FEATURES = ["attendance", "previous_mark", "hours_studied", "district"]

app = FastAPI(title="At-risk student API", version=MODEL_VERSION)
model = joblib.load("model.joblib")
log = logging.getLogger("api")


class Student(BaseModel):
    attendance: float = Field(ge=0, le=100)
    previous_mark: float = Field(ge=0, le=100)
    hours_studied: float = Field(ge=0, le=100)
    district: str = Field(min_length=1, max_length=60)


class Prediction(BaseModel):
    risk: float
    flagged: bool
    model_version: str


@app.get("/health")
def health():
    return {"status": "ok", "model_version": MODEL_VERSION}


@app.post("/predict", response_model=Prediction)
def predict(student: Student):
    try:
        row = pd.DataFrame([student.model_dump()])[FEATURES]
        risk = float(model.predict_proba(row)[0, 1])
    except Exception as exc:
        log.exception("prediction failed")
        raise HTTPException(status_code=500, detail="prediction failed") from exc

    log.info("scored", extra={"risk": risk, "version": MODEL_VERSION})
    return Prediction(risk=round(risk, 4), flagged=risk > 0.30,
                      model_version=MODEL_VERSION)


@app.post("/predict-batch", response_model=list[Prediction])
def predict_batch(students: list[Student]):
    rows = pd.DataFrame([s.model_dump() for s in students])[FEATURES]
    risks = model.predict_proba(rows)[:, 1]
    return [Prediction(risk=round(float(r), 4), flagged=r > 0.30,
                       model_version=MODEL_VERSION) for r in risks]

uvicorn app:app --host 0.0.0.0 --port 8000
# interactive docs at http://localhost:8000/docs

What makes it production-grade, not a demo

• Pydantic validation with ranges. An attendance of 5000 is rejected at the door rather than silently producing a confident nonsense prediction.
• Load the model once at startup, not per request. Loading a pickle per call turns 5 ms into 500 ms.
• A /health endpoint that reports the model version. Your load balancer needs it; so do you, when someone asks what is deployed.
• The version in every response. Non-negotiable for debugging.
• Log inputs and outputs (subject to privacy rules). Without a prediction log you cannot monitor drift, and you cannot investigate a complaint.
• Never leak internals in an error. Log the traceback; return a generic message.
• Batch endpoint. One request with 500 rows is far cheaper than 500 requests.

Containerise it

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py model.joblib ./
EXPOSE 8000
HEALTHCHECK CMD python -c "import urllib.request; \
  urllib.request.urlopen('http://localhost:8000/health')"
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

The container pins Python, the libraries and the model together. "It works on my machine" stops being a category of problem.

Shipping a new version safely

• Shadow mode — run the new model alongside the old, log both, act on neither. The cheapest way to find out it is broken.
• Canary — send 5% of traffic to it and watch the metrics.
• A/B — split traffic to measure the business outcome, not just the model score.
• Keep the previous version deployable. Rolling back must take minutes.

Try it yourself

Wrap your best pipeline in a FastAPI app with validated inputs, /health, a batch endpoint and structured logging. Test it with curl, then send deliberately invalid input (negative attendance, a missing field, a district of 500 characters) and confirm each is rejected with a clear 422 rather than a prediction. Then containerise it and run it from the image.