Getting everything ready for you…

Loading courses, problems and your progress. This takes just a moment.

0s

Pandas & NumPy Documentation

Data analysis in Python — fast NumPy arrays and pandas DataFrames to load, clean, group, join and plot data.

NumPy gives Python fast n-dimensional arrays and vectorised maths; pandas builds on it with DataFrames — labelled tables you can load, clean, filter, group, join and analyse. Together they are the foundation of data analysis, data science and machine learning in Python (scikit-learn, Matplotlib and PyTorch all speak NumPy).

1. Introduction & Installation

pip install numpy pandas matplotlib openpyxl
# or use Jupyter / Google Colab, which already has them
import numpy as np
import pandas as pd

print(np.__version__, pd.__version__)

Output:

2.1.2 2.2.3

Why not plain Python lists?

Python list NumPy array
any types mixed one type (int64, float64…)
loops in Python (slow) vectorised C loops — 10–100× faster
[1,2] * 2 → [1,2,1,2] np.array([1,2]) * 2 → [2, 4]
more memory compact memory

⚠️ Common error: ModuleNotFoundError: No module named 'pandas' — install it into the same Python/venv your editor or notebook uses (python -m pip install pandas).

2. NumPy Arrays

a = np.array([1, 2, 3, 4])
m = np.array([[1, 2, 3], [4, 5, 6]])

print(m.shape, m.ndim, m.dtype, m.size)   # (2, 3) 2 int64 6

np.zeros((2, 3))          # all zeros
np.ones(5)
np.full((2, 2), 7)
np.eye(3)                 # identity matrix
np.arange(0, 10, 2)       # [0 2 4 6 8]
np.linspace(0, 1, 5)      # [0.   0.25 0.5  0.75 1.  ]
np.random.default_rng(42).integers(1, 100, size=5)   # random ints (reproducible)

Reshape and combine:

x = np.arange(12).reshape(3, 4)     # 3 rows, 4 columns
x.T                                 # transpose → 4 × 3
x.flatten()                         # back to 1-D
np.vstack([a, a]); np.hstack([a, a]); np.concatenate([a, a])

3. Indexing, Slicing & Boolean Masks

x = np.arange(12).reshape(3, 4)
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]

x[1, 2]          # 6        row 1, column 2
x[0]             # [0 1 2 3]
x[:, 1]          # [1 5 9]  column 1
x[1:, 2:]        # [[ 6  7] [10 11]]
x[::-1]          # rows reversed

scores = np.array([45, 82, 67, 91, 38])
scores[scores >= 60]                 # [82 67 91]   boolean mask
scores[(scores > 40) & (scores < 70)]  # [45 67]    use & | ~, with brackets
np.where(scores >= 60, "pass", "fail")

⚠️ Slices are views — changing b = x[0] changes x too. Use .copy() when you need an independent array.

4. Vectorised Maths & Broadcasting

a = np.array([1, 2, 3])
a + 10            # [11 12 13]
a * a             # [1 4 9]     element-wise
a @ a             # 14          dot product
np.sqrt(a); np.exp(a); np.log(a); np.round(np.pi, 2)

m = np.array([[1, 2], [3, 4]])
m.sum(), m.sum(axis=0), m.sum(axis=1)   # 10, [4 6] (columns), [3 7] (rows)
m.mean(), m.max(), m.argmax(), m.std()
m @ m                                   # matrix multiplication
np.linalg.inv(m); np.linalg.det(m)

Broadcasting: arrays with different shapes are stretched to match when trailing dimensions are equal or 1:

prices = np.array([[100, 200], [300, 400]])   # shape (2, 2)
gst = np.array([1.18, 1.05])                    # shape (2,) → applied to each row
prices * gst
# [[118. 210.]
#  [354. 420.]]

axis=0 = down the rows (per column), axis=1 = across the columns (per row).

5. Pandas Series & DataFrames

s = pd.Series([90, 75, 60], index=["Asha", "Ravi", "Meena"], name="marks")
s["Ravi"]          # 75
s[s > 70]          # Asha 90, Ravi 75

df = pd.DataFrame({
    "name":  ["Asha", "Ravi", "Meena", "John", "Sara"],
    "dept":  ["CSE", "ECE", "CSE", "ME", "ECE"],
    "marks": [91, 78, 85, None, 66],
    "year":  [3, 2, 3, 4, 2],
})

Load and save data:

df = pd.read_csv("students.csv")                   # also: sep=";", usecols=[...], parse_dates=["dob"]
df = pd.read_excel("marks.xlsx", sheet_name="2026")
df = pd.read_json("data.json")
df = pd.read_sql("SELECT * FROM users", engine)

df.to_csv("clean.csv", index=False)
df.to_excel("report.xlsx", index=False)

First look at any dataset:

df.head()          # first 5 rows (tail() for last)
df.shape           # (rows, columns)
df.info()          # columns, types, non-null counts
df.describe()      # count, mean, std, min, quartiles, max
df.columns; df.dtypes
df["dept"].value_counts()
df["dept"].unique(); df["dept"].nunique()

6. Selecting & Filtering

df["name"]                    # one column → Series
df[["name", "marks"]]         # several columns → DataFrame

df.loc[0]                     # row by LABEL
df.loc[0:2, ["name", "marks"]]   # label slice (inclusive of 2)
df.iloc[0:2, 0:2]             # by POSITION (end exclusive)
df.at[0, "marks"]             # one value

df[df["marks"] > 80]                                   # filter
df[(df["dept"] == "CSE") & (df["year"] == 3)]           # AND (use & | ~ with brackets)
df[df["dept"].isin(["CSE", "ECE"])]
df[df["name"].str.startswith("S")]
df.query("marks > 70 and dept == 'ECE'")

df.sort_values("marks", ascending=False)
df.sort_values(["dept", "marks"], ascending=[True, False])
df.nlargest(3, "marks")

⚠️ SettingWithCopyWarning — df[df.a > 0]["b"] = 1 may change a copy. Use df.loc[df.a > 0, "b"] = 1.

7. Cleaning Data

df.isna().sum()                        # missing values per column
df.dropna(subset=["marks"])            # drop rows missing marks
df["marks"] = df["marks"].fillna(df["marks"].median())
df.drop_duplicates(subset=["email"])
df.drop(columns=["unused"])
df.rename(columns={"dept": "department"})

df["marks"] = pd.to_numeric(df["marks"], errors="coerce")   # bad values → NaN
df["joined"] = pd.to_datetime(df["joined"], dayfirst=True)
df["dept"] = df["dept"].astype("category")

df["name"] = df["name"].str.strip().str.title()
df["email"] = df["email"].str.lower()
df["city"] = df["city"].replace({"Bombay": "Mumbai", "Bangalore": "Bengaluru"})

New columns:

df["passed"] = df["marks"] >= 40
df["grade"] = np.select([df.marks >= 90, df.marks >= 75, df.marks >= 40], ["A", "B", "C"], default="F")
df["band"] = pd.cut(df["marks"], bins=[0, 40, 75, 100], labels=["low", "mid", "high"])
df["initial"] = df["name"].apply(lambda n: n[0])     # apply is slower — prefer vectorised methods
df = df.assign(pct=lambda d: d.marks / 100)

8. Grouping, Aggregation & Pivot Tables

df.groupby("dept")["marks"].mean()

df.groupby("dept").agg(
    students=("name", "count"),
    avg_marks=("marks", "mean"),
    top=("marks", "max"),
).sort_values("avg_marks", ascending=False)

df.groupby(["dept", "year"])["marks"].mean().reset_index()

pd.pivot_table(df, index="dept", columns="year", values="marks", aggfunc="mean", fill_value=0)
pd.crosstab(df["dept"], df["passed"])
df["rank_in_dept"] = df.groupby("dept")["marks"].rank(ascending=False)

Think split → apply → combine: split rows into groups, compute something per group, combine the results.

9. Merging, Reshaping & Time Series

Joining tables (like SQL joins):

students = pd.DataFrame({"id": [1, 2, 3], "name": ["Asha", "Ravi", "Meena"]})
scores   = pd.DataFrame({"id": [1, 1, 2, 4], "subject": ["DSA", "OS", "DSA", "DBMS"], "score": [90, 80, 70, 60]})

pd.merge(students, scores, on="id", how="inner")    # only matching ids (1, 2)
pd.merge(students, scores, on="id", how="left")     # all students; Meena gets NaN
pd.concat([df_jan, df_feb], ignore_index=True)      # stack rows

how: inner, left, right, outer, cross.

Reshape:

long = df.melt(id_vars="name", value_vars=["dsa", "os"], var_name="subject", value_name="score")  # wide → long
wide = long.pivot(index="name", columns="subject", values="score")                                # long → wide

Time series:

sales["date"] = pd.to_datetime(sales["date"])
sales = sales.set_index("date")
sales["amount"].resample("ME").sum()       # monthly totals ("M" in older pandas)
sales["amount"].rolling(7).mean()          # 7-day moving average
sales.loc["2026-01":"2026-03"]             # date range
sales.index.dayofweek; sales["amount"].pct_change()

10. Visualisation, Performance & Advanced Concepts

Quick charts (Matplotlib under the hood):

import matplotlib.pyplot as plt
df.groupby("dept")["marks"].mean().plot(kind="bar", title="Average marks by department")
df["marks"].plot(kind="hist", bins=10)
plt.tight_layout(); plt.show()

Performance tips:

  • Vectorise — avoid for loops and iterrows(); use column operations, np.where, np.select, .str and .dt accessors.
  • Use the right dtypes: category for repeated strings, smaller ints (int32), pd.to_datetime once.
  • Read only needed columns: pd.read_csv(..., usecols=[...], dtype={...}); process huge files in chunksize pieces.
  • df.memory_usage(deep=True) to find heavy columns; Parquet (to_parquet) is faster and smaller than CSV.
  • For very large data: Polars, DuckDB, or Dask/Spark.

Other useful features: MultiIndex, df.explode() (list → rows), df.pipe() for readable chains, pd.get_dummies() (one-hot encoding for ML), df.corr() (correlations), df.sample(frac=0.2, random_state=1).

From pandas to machine learning:

from sklearn.model_selection import train_test_split
X = pd.get_dummies(df[["dept", "year"]], drop_first=True)
y = df["passed"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

11. Examples with Output

Example 1 — NumPy statistics

marks = np.array([45, 82, 67, 91, 38])
print(marks.mean(), marks.max(), (marks >= 60).sum())
64.6 91 3

Example 2 — Group by

df = pd.DataFrame({"dept": ["CSE", "ECE", "CSE", "ECE"], "marks": [91, 78, 85, 66]})
print(df.groupby("dept")["marks"].mean())
dept
CSE    88.0
ECE    72.0
Name: marks, dtype: float64

Example 3 — Filling missing values

s = pd.Series([10, None, 30, None])
print(s.isna().sum(), s.fillna(s.mean()).tolist())
2 [10.0, 20.0, 30.0, 20.0]

12. Practice Problems

  1. Create a 5×5 NumPy matrix of random integers; find the row with the largest sum and normalise each column to 0–1.
  2. Load a CSV of students; report the shape, missing values per column and the 5 top scorers.
  3. Clean a messy dataset: trim names, fix city spellings, convert dates and numbers, and drop duplicates.
  4. Find the average, maximum and count of marks per department, sorted by average.
  5. Merge an orders table with a customers table and find the top 3 customers by total spend.
  6. Build a pivot table of monthly sales per product and plot it as a line chart.
  7. Compute a 7-day moving average of daily website visits and mark days 50% above average.
  8. Prepare a dataset for scikit-learn: handle missing values, one-hot encode categories and split train/test.

13. Interview Questions

1. NumPy array vs Python list? Arrays hold one data type in contiguous memory and support vectorised operations — much faster and smaller than lists.

2. What is broadcasting? NumPy's rule for operating on arrays of different shapes by virtually stretching dimensions of size 1 to match.

3. Series vs DataFrame? A Series is a labelled 1-D array (one column); a DataFrame is a 2-D table of Series sharing an index.

4. loc vs iloc? loc selects by labels (end inclusive); iloc selects by integer positions (end exclusive).

5. How do you handle missing data? Find with isna(), then drop (dropna) or fill (fillna with a constant, mean/median, forward-fill) depending on the context.

6. merge vs concat vs join? merge joins on columns like SQL; concat stacks along an axis; join merges on the index.

7. What does groupby do? Splits data into groups, applies an aggregation or transformation to each, and combines the results.

8. apply vs vectorised operations? apply runs a Python function per row/element (slow); vectorised methods run in optimised C and should be preferred.

9. What is a view vs a copy? A view shares data with the original (changes propagate); a copy is independent. NumPy slices are views; use .copy() to be safe.

10. pivot vs pivot_table? pivot reshapes without aggregating (needs unique pairs); pivot_table aggregates duplicates with a function like mean or sum.

14. Cheat Sheet & References

NumPy

np.array([1, 2, 3]); np.zeros((2, 3)); np.arange(0, 10, 2); np.linspace(0, 1, 5)
a.shape; a.reshape(3, 4); a.T; a.flatten()
a[a > 5]; np.where(a > 5, 1, 0)
a.sum(axis=0); a.mean(); a.std(); a.argmax()
a @ b; np.linalg.inv(m)

Load and inspect

df = pd.read_csv("file.csv")
df.head(); df.info(); df.describe(); df.shape
df["col"].value_counts(); df.isna().sum()
df.to_csv("out.csv", index=False)

Select and filter

df["a"]; df[["a", "b"]]
df.loc[rows, cols]; df.iloc[0:5, 0:2]
df[(df.a > 1) & (df.b == "x")]
df.query("a > 1"); df[df.c.isin(["x", "y"])]
df.sort_values("a", ascending=False)

Clean and transform

df.dropna(); df.fillna(0); df.drop_duplicates()
df.rename(columns={"a": "b"}); df.drop(columns=["c"])
df["d"] = pd.to_datetime(df["d"])
df["s"] = df["s"].str.strip().str.lower()
df["n"] = pd.to_numeric(df["n"], errors="coerce")

Group, merge, reshape

df.groupby("k")["v"].agg(["mean", "sum", "count"])
pd.pivot_table(df, index="a", columns="b", values="v", aggfunc="sum")
pd.merge(a, b, on="id", how="left")
pd.concat([a, b], ignore_index=True)
df.melt(id_vars="id"); df.set_index("date").resample("ME").sum()

Official documentation