Why pandas, and how it thinks

In the beginner track you computed an average per district with about fifteen lines: two dictionaries, a loop, a division. In pandas it is one line.

import pandas as pd

df = pd.read_csv("students.csv")
print(df.groupby("district")["mark"].mean())

That is not just shorter. It is a different way of thinking, and the shift is the real content of this lesson.

Loops versus columns

Plain Python works one row at a time. pandas works one column at a time: you describe an operation on a whole column and the library applies it to every value at C speed.

# row-at-a-time thinking (works, but slow and wordy)
final = []
for m in marks:
    final.append(m * 1.1)

# column-at-a-time thinking (pandas)
df["final"] = df["mark"] * 1.1

That second line is vectorised. On a million rows it is often fifty times faster than the loop, and — more importantly — it says what you mean in one readable line.

The two objects

• A Series is one column: values plus an index (row labels) plus one dtype.
• A DataFrame is a table: several Series sharing one index.

import pandas as pd

s = pd.Series([78, 41, 65], name="mark")
print(s)
# 0    78
# 1    41
# 2    65
# Name: mark, dtype: int64

That left-hand column is the index. It is not data — it is the label of each row. Ignoring it causes half of all pandas confusion, so watch it from the beginning.

Set up

python -m venv venv
source venv/bin/activate          # Windows: venv\Scripts\activate
pip install pandas matplotlib jupyter openpyxl

openpyxl is what lets pandas read .xlsx files. Start a notebook with jupyter notebook and begin every one the same way:

import pandas as pd
import matplotlib.pyplot as plt

pd.set_option("display.max_columns", 50)
pd.set_option("display.width", 200)

Those two options stop pandas hiding columns behind ..., which otherwise wastes an hour the first time a wide table prints.

numpy underneath

pandas is built on numpy, which provides the fast typed arrays. You will meet it directly for two things:

import numpy as np

np.nan                      # the "missing value" marker pandas uses
np.where(df["mark"] >= 50, "Pass", "Fail")   # vectorised if/else

Try it yourself

Install the stack, open a notebook, and build a Series of five marks. Print it and identify the values, the index and the dtype. Then multiply the whole Series by 1.1 in one line and notice that you never wrote a loop.