Regression: predicting a number

Start with linear regression. It is old, it is simple, it is often good enough, and — decisively — you can explain it to a head teacher or a loan officer.

Fit one

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

features = ["attendance", "previous_mark", "hours_studied"]
X = df[features]
y = df["final_mark"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)

print("MAE:", round(mean_absolute_error(y_test, pred), 2))
print("R2 :", round(r2_score(y_test, pred), 3))

What it learned

coefs = pd.Series(model.coef_, index=features).sort_values()
print(coefs)
print("intercept:", round(model.intercept_, 2))

# attendance        0.42
# hours_studied     1.85
# previous_mark     0.61
# intercept        12.30

Read it as: holding the other features constant, one extra hour studied is associated with 1.85 more marks. Three cautions, all important:

• "Associated with", not "causes". Students who study more may differ in ten other ways.
• The size depends on the unit. A coefficient on attendance measured 0–1 is a hundred times the same relationship measured 0–100. Standardise before comparing coefficient magnitudes.
• Correlated features scramble the attribution. If attendance and hours studied move together, the model can put the credit on either, and the split between them is unstable.

Residuals: where the model is wrong

import matplotlib.pyplot as plt

residuals = y_test - pred

fig, ax = plt.subplots(1, 2, figsize=(11, 4))
ax[0].scatter(pred, residuals, alpha=0.4)
ax[0].axhline(0, color="red", linestyle="--")
ax[0].set_xlabel("Predicted"); ax[0].set_ylabel("Actual - Predicted")
ax[0].set_title("Residuals vs predicted")

ax[1].hist(residuals, bins=30)
ax[1].set_title("Distribution of residuals")
fig.tight_layout()

What the left panel tells you:

• A shapeless cloud around zero — good. The model captured the structure.
• A curve — the true relationship is not straight. Add a squared term, or use a tree-based model.
• A widening fan — error grows with the prediction. Common with money and counts; try predicting log(y).
• A tilted line — the model is systematically over- or under-predicting across the range. Something is missing.

When straight lines are not enough

# interactions and curvature, still a linear model underneath
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

model = make_pipeline(
    PolynomialFeatures(degree=2, include_bias=False),
    LinearRegression(),
).fit(X_train, y_train)

Degree 2 on ten features produces around sixty-five columns, and degree 3 explodes. Polynomial features overfit quickly — check the test score, not the train score.

Regularisation: linear models that resist overfitting

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.preprocessing import StandardScaler

ridge = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X_train, y_train)
lasso = make_pipeline(StandardScaler(), Lasso(alpha=0.1)).fit(X_train, y_train)

• Ridge shrinks all coefficients toward zero. Good when features are correlated.
• Lasso pushes some coefficients to exactly zero, so it selects features for you.
• ElasticNet is a mix of the two.

alpha controls the strength: higher means simpler. Both require scaling — a penalty applied to unscaled coefficients punishes whichever feature happens to have small units.

Try it yourself

Fit a linear regression on your data, print MAE, RMSE and R², compare all three against the DummyRegressor baseline, and plot the residuals. Then look at your five worst predictions with df.iloc[np.argsort(-np.abs(residuals))[:5]] — those rows usually reveal either a missing feature or a data-quality problem.