Content Moderation System (Toxicity & Policy)
IntermediateOverview
A Content Moderation System automatically reviews user-generated content (text, images, video) to detect toxicity, hate speech, spam, and policy violations. It acts as the protective shield for platforms like Facebook, Discord, and Reddit. Because AI models aren't perfect, the system must balance aggressive automated filtering with human-in-the-loop review for ambiguous cases.
Functional Requirements
- Scan incoming text, images, or video for multiple violation types (e.g., hate speech, CSAM, spam).
- Action content instantly (Block, Allow, or Flag for Review).
- Provide a dashboard and queue for human moderators to review flagged content.
- Allow users to appeal moderation decisions.
Non-Functional Requirements
- Low latency: text moderation must happen synchronously (< 100ms) before the post is published.
- High throughput: handle millions of posts and messages per second.
- High precision: minimize false positives so legitimate users aren't unfairly banned.
- Adaptability: the system must quickly update to handle new slang, dog whistles, or spam campaigns.
Capacity Estimation
Assume a social messaging app processing 500,000 messages per second.
- Synchronous Path: 500k QPS of text must be scanned in < 100ms. This requires highly optimized, lightweight models (like DistilBERT or SVMs) rather than massive LLMs.
- Asynchronous Path: Images/Videos take longer to process. If 10% of messages contain media (50k QPS), they are routed to a background Kafka queue for heavy Computer Vision processing (e.g., ResNet, CLIP).
- Human Review: If the AI flags 0.1% of traffic as "ambiguous", that's 500 items per second entering the manual review queue, requiring a massive global team of human moderators.
High-Level Architecture
The architecture uses a Multi-Stage Filter Pipeline. When a user posts content, it hits the Moderation Gateway. Stage 1 is a deterministic filter (Regex, Blocklists, perceptual hashes for known CSAM). This takes < 5ms. Stage 2 is the ML Classifier (e.g., RoBERTa for text toxicity), which outputs probability scores. A Decision Engine evaluates these scores against business rules. If the score is > 0.95, it's auto-blocked. If < 0.50, it's auto-allowed. If between 0.50 and 0.95, it's published but flagged, sending an event to Kafka. The Manual Review Service pulls from Kafka and populates a UI for human moderators to make the final call.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| moderation_rules | rule_id, category (spam, hate), threshold_allow, threshold_block | In-Memory Cache (Redis) |
| content_hash_blocklist | hash_id (PK), hash_type (PhotoDNA, MD5), added_by | Key-Value Store (DynamoDB / Redis) |
| moderation_tickets | ticket_id, content_id, user_id, AI_scores (JSON), status (pending, resolved), human_decision | Relational DB (PostgreSQL) |
Detailed Design
Deterministic vs. Probabilistic Filtering
Machine Learning is expensive. Before running neural networks, content passes through a Deterministic layer. Text is checked against Regex and exact-match blocklists. Images are hashed using Perceptual Hashing (PhotoDNA or pHash), which detects images even if they are slightly cropped or resized. If a perceptual hash matches a known illegal image database (like NCMEC), the content is instantly blocked without any ML inference.
The Decision Engine (Rules Engine)
The outputs of the ML models are just raw probabilities (e.g., Toxicity=0.88, Spam=0.12). A lightweight Rules Engine (often written in Go or Rust) evaluates these scores. This decouples the Data Science team (who update models) from the Trust & Safety team (who update thresholds). During a high-profile event (e.g., an election), Trust & Safety can lower the auto-block threshold from 0.95 to 0.85 instantly via the Rules Engine.
Human-in-the-Loop Feedback
When a human moderator reviews a ticket, their decision (Block/Allow) is saved. This data is the most valuable asset in the system. It is fed back into a Data Lake where it is used as high-quality labeled training data to retrain and fine-tune the ML classifiers, continuously improving the AI's accuracy.
Bottlenecks & Solutions
The biggest operational bottleneck is the Human Review Backlog. If a new spam attack occurs, the AI might flag 10x the normal volume, overwhelming human moderators. Systemic solutions include 'Auto-Actioning' bursts of highly similar flagged content, throttling user posting rates based on account age during anomalies, or dynamically increasing the flag threshold to shed load.