Fraud & Anomaly Detection System

Advanced

Overview

A Fraud Detection System operates in the high-stakes environment of financial transactions, account takeovers, and payment processing (e.g., Stripe, PayPal, banking). The system must analyze a transaction in milliseconds, compare it against the user's historical behavior and global fraud patterns, and definitively approve or decline the action. It requires a blend of real-time stream processing, Graph Databases to detect fraud rings, and Machine Learning classifiers.

Real-time Fraud Detection System combining Graph networks, streaming ML, and Rules Engines

Functional Requirements

Evaluate incoming financial transactions (or login attempts) and assign a fraud probability score.
Instantly Block, Allow, or route the transaction for Step-Up Authentication (e.g., 2FA/OTP).
Detect coordinated fraud rings using linked entities (shared IP, device ID, shipping address).
Allow human analysts to review flagged transactions in a case management queue.

Non-Functional Requirements

Ultra-low latency: the entire decision pipeline must complete in < 50-100ms so the user isn't stuck at checkout.
High availability (99.999%): if the system goes down, payments fail, causing catastrophic revenue loss.
Explainability: financial regulations often require the system to explain *why* a transaction was blocked.
Adaptive: fraud patterns shift rapidly; the system must ingest new features and rules in minutes.

Capacity Estimation

Assume a global payment processor handling 10,000 transactions per second (TPS) at peak.

Latency Budget: 100ms total. 20ms for network overhead, 30ms for Feature Store lookup, 30ms for Graph DB traversal, 20ms for ML inference.
Feature Store: Extracting sliding window aggregations (e.g., "total amount spent by this card in the last 1 hour") requires 100k+ reads/sec from an in-memory database like Redis.
Graph Database: Storing billions of edges (User -> Device, Device -> IP, IP -> Transaction) requires a distributed graph database (e.g., TigerGraph or Neo4j) heavily optimized for multi-hop queries.

High-Level Architecture

When a transaction arrives at the API Gateway, it routes to the Fraud Orchestrator. The Orchestrator performs parallel data gathering: it fetches historical user features from a Redis Feature Store and queries a Graph Database to check if the device or IP is linked to known bad actors. These features are fed into a fast Machine Learning Model (like XGBoost, due to its speed and tabular data performance). The ML model outputs a fraud score. Finally, a Rules Engine takes the score and business logic (e.g., "If score > 80 AND amount > $5000 -> Block") to make the final decision. The result is returned, and all telemetry is logged asynchronously to a Data Lake via Kafka.

Data Model

Entity	Fields / Schema	Storage Choice
real_time_features	user_id (PK), velocity_1h, velocity_24h, distinct_ips_24h	In-Memory Store (Redis / Aerospike)
entity_graph	node_id, node_type (User/Device/IP), edges (used_by, logged_in_from)	Graph Database (TigerGraph / Neo4j)
transaction_ledger	tx_id, user_id, amount, status, fraud_score, decision_reason	Relational DB (PostgreSQL / CockroachDB)

Detailed Design

The Role of the Graph Database

Traditional ML struggles to detect "Fraud Rings" (organized groups creating synthetic identities). A Graph Database excels here. When a transaction occurs, the system runs a fast 2-hop or 3-hop query: "Has the device ID associated with this transaction ever been used by an account that was previously marked as fraudulent?" If the graph traversal finds a link, a high-risk boolean feature is passed to the ML model.

Feature Store & Sliding Windows

Fraud models rely heavily on "Velocity Features" (e.g., number of password resets in 10 minutes, amount of money transferred to new payees in 24 hours). A stream processor (Apache Flink) reads raw events from Kafka, continuously calculates these sliding window aggregates, and writes them to the Redis Feature Store. During a transaction, the Orchestrator does an O(1) lookup in Redis to get these pre-computed aggregates.

Explainability vs. Deep Learning

While Deep Learning is popular, fraud systems often prefer Gradient Boosted Trees (XGBoost/LightGBM) or Random Forests. These models handle tabular data exceptionally well, infer very fast on CPUs (no GPU needed), and most importantly, support explainability tools like SHAP values. If regulators ask why a legitimate user's account was frozen, the company can point to the specific features (e.g., "Foreign IP + High Velocity") that triggered the block.

Bottlenecks & Solutions

The primary bottleneck is Stateful Stream Processing. Calculating sliding windows over millions of users requires massive amounts of RAM in Flink to maintain state. If a node fails, state must be restored from checkpoints (RocksDB to S3), which can temporarily halt feature updates. Furthermore, querying the Graph DB under a strict 30ms budget requires carefully restricting traversal depth (e.g., max 3 hops, capped at 100 edges per node) to prevent "super-node" queries from timing out.

Interview Follow-up Questions

Q: How do you handle 'Friendly Fraud' (a user making a purchase and falsely claiming their card was stolen)?

Friendly fraud is hard to catch in real-time because the behavioral biometrics match the legitimate user. We combat this using historical dispute data. If a user has a high chargeback ratio, we flag them. We also capture rich device telemetry (typing speed, gyro data on mobile) to prove it was the actual user holding the phone.

Q: What happens if the ML model goes down or the Feature Store times out?

The Fraud Orchestrator enforces strict timeouts (e.g., 50ms). If the ML model or Feature Store times out, the system falls back to the deterministic Rules Engine. The Rules Engine uses basic payload data (e.g., 'Is the amount > $10,000?') to make a safe, degraded decision. In payments, availability often trumps advanced fraud detection to avoid losing legitimate revenue.