Pandas & NumPy Documentation
Data analysis in Python — fast NumPy arrays and pandas DataFrames to load, clean, group, join and plot data.
NumPy gives Python fast n-dimensional arrays and vectorised maths; pandas builds on it with DataFrames — labelled tables you can load, clean, filter, group, join and analyse. Together they are the foundation of data analysis, data science and machine learning in Python (scikit-learn, Matplotlib and PyTorch all speak NumPy).
1. Introduction & Installation
pip install numpy pandas matplotlib openpyxl
# or use Jupyter / Google Colab, which already has them
import numpy as np
import pandas as pd
print(np.__version__, pd.__version__)
Output:
2.1.2 2.2.3
Why not plain Python lists?
| Python list | NumPy array |
|---|---|
| any types mixed | one type (int64, float64…) |
| loops in Python (slow) | vectorised C loops — 10–100× faster |
[1,2] * 2 → [1,2,1,2] |
np.array([1,2]) * 2 → [2, 4] |
| more memory | compact memory |
⚠️ Common error:
ModuleNotFoundError: No module named 'pandas'— install it into the same Python/venv your editor or notebook uses (python -m pip install pandas).
2. NumPy Arrays
a = np.array([1, 2, 3, 4])
m = np.array([[1, 2, 3], [4, 5, 6]])
print(m.shape, m.ndim, m.dtype, m.size) # (2, 3) 2 int64 6
np.zeros((2, 3)) # all zeros
np.ones(5)
np.full((2, 2), 7)
np.eye(3) # identity matrix
np.arange(0, 10, 2) # [0 2 4 6 8]
np.linspace(0, 1, 5) # [0. 0.25 0.5 0.75 1. ]
np.random.default_rng(42).integers(1, 100, size=5) # random ints (reproducible)
Reshape and combine:
x = np.arange(12).reshape(3, 4) # 3 rows, 4 columns
x.T # transpose → 4 × 3
x.flatten() # back to 1-D
np.vstack([a, a]); np.hstack([a, a]); np.concatenate([a, a])
3. Indexing, Slicing & Boolean Masks
x = np.arange(12).reshape(3, 4)
# [[ 0 1 2 3]
# [ 4 5 6 7]
# [ 8 9 10 11]]
x[1, 2] # 6 row 1, column 2
x[0] # [0 1 2 3]
x[:, 1] # [1 5 9] column 1
x[1:, 2:] # [[ 6 7] [10 11]]
x[::-1] # rows reversed
scores = np.array([45, 82, 67, 91, 38])
scores[scores >= 60] # [82 67 91] boolean mask
scores[(scores > 40) & (scores < 70)] # [45 67] use & | ~, with brackets
np.where(scores >= 60, "pass", "fail")
⚠️ Slices are views — changing
b = x[0]changesxtoo. Use.copy()when you need an independent array.
4. Vectorised Maths & Broadcasting
a = np.array([1, 2, 3])
a + 10 # [11 12 13]
a * a # [1 4 9] element-wise
a @ a # 14 dot product
np.sqrt(a); np.exp(a); np.log(a); np.round(np.pi, 2)
m = np.array([[1, 2], [3, 4]])
m.sum(), m.sum(axis=0), m.sum(axis=1) # 10, [4 6] (columns), [3 7] (rows)
m.mean(), m.max(), m.argmax(), m.std()
m @ m # matrix multiplication
np.linalg.inv(m); np.linalg.det(m)
Broadcasting: arrays with different shapes are stretched to match when trailing dimensions are equal or 1:
prices = np.array([[100, 200], [300, 400]]) # shape (2, 2)
gst = np.array([1.18, 1.05]) # shape (2,) → applied to each row
prices * gst
# [[118. 210.]
# [354. 420.]]
axis=0 = down the rows (per column), axis=1 = across the columns (per row).
5. Pandas Series & DataFrames
s = pd.Series([90, 75, 60], index=["Asha", "Ravi", "Meena"], name="marks")
s["Ravi"] # 75
s[s > 70] # Asha 90, Ravi 75
df = pd.DataFrame({
"name": ["Asha", "Ravi", "Meena", "John", "Sara"],
"dept": ["CSE", "ECE", "CSE", "ME", "ECE"],
"marks": [91, 78, 85, None, 66],
"year": [3, 2, 3, 4, 2],
})
Load and save data:
df = pd.read_csv("students.csv") # also: sep=";", usecols=[...], parse_dates=["dob"]
df = pd.read_excel("marks.xlsx", sheet_name="2026")
df = pd.read_json("data.json")
df = pd.read_sql("SELECT * FROM users", engine)
df.to_csv("clean.csv", index=False)
df.to_excel("report.xlsx", index=False)
First look at any dataset:
df.head() # first 5 rows (tail() for last)
df.shape # (rows, columns)
df.info() # columns, types, non-null counts
df.describe() # count, mean, std, min, quartiles, max
df.columns; df.dtypes
df["dept"].value_counts()
df["dept"].unique(); df["dept"].nunique()
6. Selecting & Filtering
df["name"] # one column → Series
df[["name", "marks"]] # several columns → DataFrame
df.loc[0] # row by LABEL
df.loc[0:2, ["name", "marks"]] # label slice (inclusive of 2)
df.iloc[0:2, 0:2] # by POSITION (end exclusive)
df.at[0, "marks"] # one value
df[df["marks"] > 80] # filter
df[(df["dept"] == "CSE") & (df["year"] == 3)] # AND (use & | ~ with brackets)
df[df["dept"].isin(["CSE", "ECE"])]
df[df["name"].str.startswith("S")]
df.query("marks > 70 and dept == 'ECE'")
df.sort_values("marks", ascending=False)
df.sort_values(["dept", "marks"], ascending=[True, False])
df.nlargest(3, "marks")
⚠️
SettingWithCopyWarning—df[df.a > 0]["b"] = 1may change a copy. Usedf.loc[df.a > 0, "b"] = 1.
7. Cleaning Data
df.isna().sum() # missing values per column
df.dropna(subset=["marks"]) # drop rows missing marks
df["marks"] = df["marks"].fillna(df["marks"].median())
df.drop_duplicates(subset=["email"])
df.drop(columns=["unused"])
df.rename(columns={"dept": "department"})
df["marks"] = pd.to_numeric(df["marks"], errors="coerce") # bad values → NaN
df["joined"] = pd.to_datetime(df["joined"], dayfirst=True)
df["dept"] = df["dept"].astype("category")
df["name"] = df["name"].str.strip().str.title()
df["email"] = df["email"].str.lower()
df["city"] = df["city"].replace({"Bombay": "Mumbai", "Bangalore": "Bengaluru"})
New columns:
df["passed"] = df["marks"] >= 40
df["grade"] = np.select([df.marks >= 90, df.marks >= 75, df.marks >= 40], ["A", "B", "C"], default="F")
df["band"] = pd.cut(df["marks"], bins=[0, 40, 75, 100], labels=["low", "mid", "high"])
df["initial"] = df["name"].apply(lambda n: n[0]) # apply is slower — prefer vectorised methods
df = df.assign(pct=lambda d: d.marks / 100)
8. Grouping, Aggregation & Pivot Tables
df.groupby("dept")["marks"].mean()
df.groupby("dept").agg(
students=("name", "count"),
avg_marks=("marks", "mean"),
top=("marks", "max"),
).sort_values("avg_marks", ascending=False)
df.groupby(["dept", "year"])["marks"].mean().reset_index()
pd.pivot_table(df, index="dept", columns="year", values="marks", aggfunc="mean", fill_value=0)
pd.crosstab(df["dept"], df["passed"])
df["rank_in_dept"] = df.groupby("dept")["marks"].rank(ascending=False)
Think split → apply → combine: split rows into groups, compute something per group, combine the results.
9. Merging, Reshaping & Time Series
Joining tables (like SQL joins):
students = pd.DataFrame({"id": [1, 2, 3], "name": ["Asha", "Ravi", "Meena"]})
scores = pd.DataFrame({"id": [1, 1, 2, 4], "subject": ["DSA", "OS", "DSA", "DBMS"], "score": [90, 80, 70, 60]})
pd.merge(students, scores, on="id", how="inner") # only matching ids (1, 2)
pd.merge(students, scores, on="id", how="left") # all students; Meena gets NaN
pd.concat([df_jan, df_feb], ignore_index=True) # stack rows
how: inner, left, right, outer, cross.
Reshape:
long = df.melt(id_vars="name", value_vars=["dsa", "os"], var_name="subject", value_name="score") # wide → long
wide = long.pivot(index="name", columns="subject", values="score") # long → wide
Time series:
sales["date"] = pd.to_datetime(sales["date"])
sales = sales.set_index("date")
sales["amount"].resample("ME").sum() # monthly totals ("M" in older pandas)
sales["amount"].rolling(7).mean() # 7-day moving average
sales.loc["2026-01":"2026-03"] # date range
sales.index.dayofweek; sales["amount"].pct_change()
10. Visualisation, Performance & Advanced Concepts
Quick charts (Matplotlib under the hood):
import matplotlib.pyplot as plt
df.groupby("dept")["marks"].mean().plot(kind="bar", title="Average marks by department")
df["marks"].plot(kind="hist", bins=10)
plt.tight_layout(); plt.show()
Performance tips:
- Vectorise — avoid
forloops anditerrows(); use column operations,np.where,np.select,.strand.dtaccessors. - Use the right dtypes:
categoryfor repeated strings, smaller ints (int32),pd.to_datetimeonce. - Read only needed columns:
pd.read_csv(..., usecols=[...], dtype={...}); process huge files inchunksizepieces. df.memory_usage(deep=True)to find heavy columns; Parquet (to_parquet) is faster and smaller than CSV.- For very large data: Polars, DuckDB, or Dask/Spark.
Other useful features: MultiIndex, df.explode() (list → rows), df.pipe() for readable chains, pd.get_dummies() (one-hot encoding for ML), df.corr() (correlations), df.sample(frac=0.2, random_state=1).
From pandas to machine learning:
from sklearn.model_selection import train_test_split
X = pd.get_dummies(df[["dept", "year"]], drop_first=True)
y = df["passed"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
11. Examples with Output
Example 1 — NumPy statistics
marks = np.array([45, 82, 67, 91, 38])
print(marks.mean(), marks.max(), (marks >= 60).sum())
64.6 91 3
Example 2 — Group by
df = pd.DataFrame({"dept": ["CSE", "ECE", "CSE", "ECE"], "marks": [91, 78, 85, 66]})
print(df.groupby("dept")["marks"].mean())
dept
CSE 88.0
ECE 72.0
Name: marks, dtype: float64
Example 3 — Filling missing values
s = pd.Series([10, None, 30, None])
print(s.isna().sum(), s.fillna(s.mean()).tolist())
2 [10.0, 20.0, 30.0, 20.0]
12. Practice Problems
- Create a 5×5 NumPy matrix of random integers; find the row with the largest sum and normalise each column to 0–1.
- Load a CSV of students; report the shape, missing values per column and the 5 top scorers.
- Clean a messy dataset: trim names, fix city spellings, convert dates and numbers, and drop duplicates.
- Find the average, maximum and count of marks per department, sorted by average.
- Merge an orders table with a customers table and find the top 3 customers by total spend.
- Build a pivot table of monthly sales per product and plot it as a line chart.
- Compute a 7-day moving average of daily website visits and mark days 50% above average.
- Prepare a dataset for scikit-learn: handle missing values, one-hot encode categories and split train/test.
13. Interview Questions
1. NumPy array vs Python list? Arrays hold one data type in contiguous memory and support vectorised operations — much faster and smaller than lists.
2. What is broadcasting? NumPy's rule for operating on arrays of different shapes by virtually stretching dimensions of size 1 to match.
3. Series vs DataFrame? A Series is a labelled 1-D array (one column); a DataFrame is a 2-D table of Series sharing an index.
4. loc vs iloc?
loc selects by labels (end inclusive); iloc selects by integer positions (end exclusive).
5. How do you handle missing data?
Find with isna(), then drop (dropna) or fill (fillna with a constant, mean/median, forward-fill) depending on the context.
6. merge vs concat vs join?
merge joins on columns like SQL; concat stacks along an axis; join merges on the index.
7. What does groupby do?
Splits data into groups, applies an aggregation or transformation to each, and combines the results.
8. apply vs vectorised operations?
apply runs a Python function per row/element (slow); vectorised methods run in optimised C and should be preferred.
9. What is a view vs a copy?
A view shares data with the original (changes propagate); a copy is independent. NumPy slices are views; use .copy() to be safe.
10. pivot vs pivot_table?
pivot reshapes without aggregating (needs unique pairs); pivot_table aggregates duplicates with a function like mean or sum.
14. Cheat Sheet & References
NumPy
np.array([1, 2, 3]); np.zeros((2, 3)); np.arange(0, 10, 2); np.linspace(0, 1, 5)
a.shape; a.reshape(3, 4); a.T; a.flatten()
a[a > 5]; np.where(a > 5, 1, 0)
a.sum(axis=0); a.mean(); a.std(); a.argmax()
a @ b; np.linalg.inv(m)
Load and inspect
df = pd.read_csv("file.csv")
df.head(); df.info(); df.describe(); df.shape
df["col"].value_counts(); df.isna().sum()
df.to_csv("out.csv", index=False)
Select and filter
df["a"]; df[["a", "b"]]
df.loc[rows, cols]; df.iloc[0:5, 0:2]
df[(df.a > 1) & (df.b == "x")]
df.query("a > 1"); df[df.c.isin(["x", "y"])]
df.sort_values("a", ascending=False)
Clean and transform
df.dropna(); df.fillna(0); df.drop_duplicates()
df.rename(columns={"a": "b"}); df.drop(columns=["c"])
df["d"] = pd.to_datetime(df["d"])
df["s"] = df["s"].str.strip().str.lower()
df["n"] = pd.to_numeric(df["n"], errors="coerce")
Group, merge, reshape
df.groupby("k")["v"].agg(["mean", "sum", "count"])
pd.pivot_table(df, index="a", columns="b", values="v", aggfunc="sum")
pd.merge(a, b, on="id", how="left")
pd.concat([a, b], ignore_index=True)
df.melt(id_vars="id"); df.set_index("date").resample("ME").sum()
Official documentation