Algorithmic Bias
Algorithmic bias occurs when an ML system systematically produces outputs that are unfair or discriminatory — arising from biased training data, flawed problem framing, or feedback loops that amplify initial disparities.
Why Does This Exist?
ML systems learn patterns from data generated by humans in societies with existing inequalities. If historical hiring data shows that a company hired mostly men for engineering roles, a model trained on that data will learn to prefer male candidates — not because the model was explicitly programmed to discriminate, but because it faithfully replicated historical patterns. "Learning from data" is not neutral when the data encodes historical injustice.
Algorithmic bias has been documented across facial recognition (higher error rates for darker-skinned women), natural language models (associating "doctor" with male pronouns), and healthcare risk scoring (recommending fewer resources for Black patients).
Think of It Like This
A hiring manager who only learned from the past
Imagine training a new hiring manager by having them read thousands of historical hiring decisions made when diversity was not a priority. They would absorb the implicit preferences in those decisions and apply them going forward — not maliciously, but as a faithful student of the patterns they were taught. An ML model trained on biased historical data does exactly this, systematically and at scale.
How It Actually Works
Sources of bias
Historical bias: Existing societal inequities encoded in the data. Even if the data is an accurate reflection of the past, using it to make decisions perpetuates that past rather than enabling a different future.
Representation bias: Some groups are underrepresented in the training data. A face recognition system trained mostly on lighter-skinned faces from Western contexts will have higher error rates on darker-skinned faces because it saw fewer examples during training.
Measurement bias: The labels themselves are biased. "Criminal recidivism" labels depend on who gets arrested, which depends on policing patterns, which are themselves biased. A model predicting "recidivism" is actually predicting "likelihood of re-arrest," which is a different and biased quantity.
Aggregation bias: Using a single model for all subgroups when the relationship between features and outcome differs across groups. Medical models trained predominantly on male subjects can perform poorly on female patients because physiological differences mean the feature-label relationships differ.
Feedback loops / automation bias: A predictive policing algorithm sends more police to certain neighborhoods → more arrests in those neighborhoods → "confirms" the algorithm's predictions → more policing. The algorithm creates the pattern it predicts.
Bias propagation through the ML pipeline
Problem framing ↓ (choice of objective can encode bias)Data collection ↓ (sampling and labeling can be biased)Feature engineering ↓ (proxy features carry protected-attribute signal)Model training ↓ (objective function can weight groups unequally)Evaluation ↓ (aggregate metrics hide subgroup disparities)Deployment ↓ (feedback loops amplify initial biases)Bias can enter at every stage. Mitigation at one stage doesn't fix bias introduced at another.
Mitigation strategies
Pre-processing: Reweigh or resample training data to reduce disparities before training.
In-processing: Add fairness constraints to the training objective — penalize disparate impact directly during optimization.
Post-processing: Adjust prediction thresholds separately per group after training to equalize error rates.
Code
import numpy as npimport pandas as pdfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import confusion_matrix
# ── Simulate biased hiring data ───────────────────────────────────────────────np.random.seed(42)n = 3000gender = np.random.binomial(1, 0.3, n) # 1 = male (majority), 0 = female
# Qualifications (equally distributed across genders)qualification = np.random.normal(0, 1, n)
# Historical hiring: equally qualified women hired less often (historical bias)hire_prob = 1 / (1 + np.exp(-(qualification + 0.8 * gender)))was_hired = np.random.binomial(1, hire_prob)
X = pd.DataFrame({"qualification": qualification, "gender": gender})y = was_hired
from sklearn.model_selection import train_test_splitX_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
# ── Train model (with gender feature) ────────────────────────────────────────clf_biased = LogisticRegression()clf_biased.fit(X_tr, y_tr)pred_biased = clf_biased.predict(X_te)
# ── Measure disparate impact ──────────────────────────────────────────────────def selection_rate(y_pred, gender_col): return { "male": y_pred[gender_col == 1].mean(), "female": y_pred[gender_col == 0].mean(), }
rates_biased = selection_rate(pred_biased, X_te["gender"].values)print("Selection rates (biased model):")print(f" Male: {rates_biased['male']:.3f}")print(f" Female: {rates_biased['female']:.3f}")disparate_impact = rates_biased["female"] / rates_biased["male"]print(f" Disparate impact ratio: {disparate_impact:.3f} (< 0.8 = discriminatory by 4/5ths rule)")
# ── Post-processing: equalize thresholds ─────────────────────────────────────from sklearn.metrics import roc_curve
proba = clf_biased.predict_proba(X_te)[:, 1]male_mask = X_te["gender"].values == 1female_mask = X_te["gender"].values == 0
# Find threshold for each group that achieves equal selection rate (target: 0.5)target_rate = 0.5def find_threshold(proba, mask, target): for thresh in np.arange(0.1, 0.9, 0.01): if (proba[mask] >= thresh).mean() <= target: return thresh return 0.5
thresh_male = find_threshold(proba, male_mask, target_rate)thresh_female = find_threshold(proba, female_mask, target_rate)
pred_fair = np.where( male_mask, (proba >= thresh_male).astype(int), (proba >= thresh_female).astype(int))
rates_fair = selection_rate(pred_fair, X_te["gender"].values)print("\nSelection rates (post-processing, equalized):")print(f" Male: {rates_fair['male']:.3f}")print(f" Female: {rates_fair['female']:.3f}")print(f" Disparate impact ratio: {rates_fair['female'] / rates_fair['male']:.3f}")Watch Out For
Believing 'fairness through unawareness' works
Removing the protected attribute from the feature set does not prevent the model from discriminating — it will simply use correlated proxy features to infer the protected attribute. Studies have shown that gender and race can be predicted from seemingly neutral features like zip code, first name, and browsing behavior. Formal fairness constraints are necessary, not just attribute removal.
Fixing bias only at the model layer
If the problem framing is wrong (optimizing for a biased proxy), the labels are biased (criminal records vs. actual criminality), or the deployment context creates feedback loops, fixing the model alone is insufficient. Bias audits must cover the entire socio-technical system — data collection, labeling, objectives, deployment, and feedback — not just the model weights.
The Quick Version
- Algorithmic bias arises from biased data (historical, representation, measurement), biased problem framing, or feedback loops in deployment.
- Bias can enter at every stage of the ML pipeline — mitigation at one stage doesn't fix bias introduced at another.
- "Fairness through unawareness" (dropping protected attributes) is ineffective due to proxy features.
- Mitigation strategies: pre-processing (reweighting data), in-processing (fairness constraints), post-processing (threshold adjustment).
- Bias is a socio-technical problem; model-only fixes are necessary but not sufficient.