Python for Data Science

NumPy, pandas, cleaning, reshaping, time series, plotting, scikit-learn pipelines, and performance habits.

Workflow and Imports

Load
Inspect
Clean
Transform
Aggregate
Model or plot

Keep each step a small, testable function

Python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

Notebooks: Restart and run all before trusting a result; hidden state causes bugs.
Environments: Pin versions in requirements.txt or pyproject.toml.

NumPy Arrays

Python

a = np.array([[1, 2, 3], [4, 5, 6]])
a.shape, a.ndim, a.dtype     # (2, 3), 2, int64
np.zeros((3, 4)); np.ones(5); np.arange(0, 10, 2)
np.linspace(0, 1, 5)         # 5 evenly spaced
a.reshape(3, 2); a.T; a.ravel()

Views vs copies: Slices are views; use .copy() before mutating a subset.
Broadcasting: Shapes align from the right; size-1 axes stretch to match.
Boolean masks: a[a > 2] selects elements without a loop.

NumPy Math

Python

a.sum(axis=0)    # column sums
a.mean(axis=1)   # row means
a @ b            # matrix product
np.where(a > 2, 1, 0)
np.clip(a, 0, 5)
np.percentile(a, [25, 50, 75])

NaN-aware: np.nanmean, np.nansum skip missing values.
Random: rng = np.random.default_rng(42) for reproducible draws.

Think in axes

axis=0 collapses rows (one result per column); axis=1 collapses columns (one result per row).

pandas Basics

Python

df = pd.read_csv("orders.csv", parse_dates=["ts"])
df.head(); df.info(); df.describe()
df["revenue"].value_counts(dropna=False)
df.loc[df["country"] == "IN", ["user_id", "revenue"]]
df.iloc[:5, :3]
df.assign(gross=lambda d: d["revenue"] * 1.18)

loc vs iloc: loc uses labels and boolean masks; iloc uses integer positions.
Chaining: Prefer .assign() and .pipe() to avoid SettingWithCopyWarning.

Missing Data

Python

df.isna().mean().sort_values(ascending=False)
df.dropna(subset=["user_id"])
df["age"] = df["age"].fillna(df["age"].median())
df["city"] = df["city"].fillna("unknown")
df["age_missing"] = df["age"].isna()

Sentinels: Convert placeholders like -1, "N/A", or "" to NaN first.
Indicator column: Keep "was missing" when absence itself carries signal.

Impute after splitting

Compute medians and modes on the training data only, then apply them to validation and test.

Filter, Sort, Deduplicate

Python

df.query("revenue > 100 and status == 'paid'")
df[df["country"].isin(["IN", "US"])]
df.sort_values(["user_id", "ts"], ascending=[True, False])
df.drop_duplicates(subset=["order_id"], keep="last")
df.nlargest(10, "revenue")

query: Readable filters; reference variables with @name.
keep: "first", "last", or False to drop every duplicate.
Stable sort: Pass kind="stable" when ties must keep their order.

GroupBy and Aggregation

Python

(df.groupby("country", as_index=False)
   .agg(users=("user_id", "nunique"),
        revenue=("revenue", "sum"),
        aov=("revenue", "mean")))
df["share"] = df["revenue"] / df.groupby(
    "country")["revenue"].transform("sum")

agg: One row per group.
transform: Same shape as the input; good for group-wise features.
filter: Keep whole groups that satisfy a condition.

Merge and Join

Python

orders.merge(users, on="user_id", how="left",
             validate="many_to_one", indicator=True)
pd.concat([df_2025, df_2026], ignore_index=True)

how	Keeps
inner	Keys present in both
left	All left rows
outer	All keys from both
cross	Every combination

Validate join cardinality

validate="many_to_one" raises if the right key is duplicated, catching silent row explosions.

Reshaping

Python

wide = df.pivot_table(index="user_id",
                      columns="month",
                      values="revenue",
                      aggfunc="sum", fill_value=0)
long = wide.reset_index().melt(id_vars="user_id",
                               var_name="month")
df.explode("tags")

pivot_table: Long → wide with aggregation; handles duplicate keys.
melt: Wide → long; the tidy shape most plotting and modeling tools expect.
explode: One row per list element.

Dates and Time Series

Python

df["ts"] = pd.to_datetime(df["ts"], utc=True)
df["dow"] = df["ts"].dt.dayofweek
daily = df.set_index("ts").resample("D")["revenue"].sum()
daily.rolling(7).mean()
df["prev"] = df.groupby("user_id")["revenue"].shift(1)

Time zones: Store UTC; convert with .dt.tz_convert() for reporting.
Gaps: resample creates missing periods; decide between fill with 0 and NaN.

Vectorize, Don't Loop

Python

# slow: row-wise Python
df["band"] = df["revenue"].apply(
    lambda r: "hi" if r > 100 else "lo")
# fast: vectorized
df["band"] = np.where(df["revenue"] > 100, "hi", "lo")
df["band"] = pd.cut(df["revenue"], [0, 100, np.inf],
                    labels=["lo", "hi"])

map: Element-wise lookup from a dict or Series.
Categoricals: astype("category") saves memory for repeated strings.

Files and Formats

CSV: Human-readable; slow and loses dtypes. Pass dtype and parse_dates.
Parquet: Columnar, typed, compressed; the default for analytics.
Feather: Fast local interchange between Python and R.
Chunks: pd.read_csv(path, chunksize=100_000) streams large files.

Python

df.to_parquet("orders.parquet", index=False)
df = pd.read_parquet("orders.parquet",
                     columns=["user_id", "revenue"])

Plotting

Python

fig, ax = plt.subplots(figsize=(8, 4))
daily.plot(ax=ax, label="daily")
daily.rolling(7).mean().plot(ax=ax, label="7-day")
ax.set(title="Revenue", xlabel="", ylabel="USD")
ax.legend(); fig.tight_layout()
fig.savefig("revenue.png", dpi=200)

Question	Chart
Trend over time	Line
Compare categories	Bar
Distribution	Histogram or box plot
Relationship	Scatter

Data Quality Checks

Python

assert df["order_id"].is_unique
assert df["revenue"].ge(0).all()
assert df["ts"].between("2020-01-01", "2030-01-01").all()
df.duplicated().sum()
df.select_dtypes("number").describe().T

Keys: Unique ids where expected; no orphan foreign keys after joins.
Ranges: Non-negative amounts, plausible dates, valid categories.
Freshness: The latest timestamp is as recent as the pipeline promises.

scikit-learn Pipeline

Python

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LogisticRegression

pre = ColumnTransformer([
    ("num", StandardScaler(), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"),
     cat_cols),
])
model = Pipeline([("pre", pre),
                  ("clf", LogisticRegression())])
model.fit(X_train, y_train)

Pipelines prevent leakage

Fitting the whole pipeline inside cross-validation keeps scalers and encoders train-only.

handle_unknown="ignore": New categories at inference become all-zero instead of an error.

Evaluation Snippets

Python

from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report

scores = cross_val_score(model, X, y, cv=5,
                         scoring="roc_auc")
print(scores.mean(), scores.std())
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

Report the spread: Mean ± std across folds, not one lucky split.
Probabilities: Use predict_proba for ROC-AUC, PR-AUC, and threshold tuning.

Performance Tips

Dtypes: Downcast numbers and use categories for repeated strings.
Read less: Load only the needed columns and rows.
Avoid apply: Vectorized NumPy/pandas operations are 10–100× faster.
Profile: %timeit and df.memory_usage(deep=True) find the real hot spots.
Scale out: Polars or DuckDB handle larger-than-pandas workloads on one machine.

Debug Checklist

Check df.shape after every merge and filter.
Look at dtypes; numbers stored as strings break math silently.
Count nulls before and after each transform.
Verify keys are unique where you expect them to be.
Set seeds (random_state) for reproducible splits and models.