Notification System

Intermediate

Overview

A notification system accepts requests from many internal services and delivers messages to users across multiple channels - push, SMS, and email - while honoring user preferences and avoiding duplicate sends. It must fan a single event out to the right channels reliably and at high throughput.

High-level architecture for Notification System

Functional Requirements

Accept notification requests from many producer services and deliver them over push, SMS, and email channels.
Honor per-user, per-channel preferences, including opt-out and unsubscribe.
Deduplicate delivery so a user is not notified more than once for the same underlying event.
Support reusable templates so producers send structured data rather than fully-rendered text.
Track and expose the delivery status of each notification (queued, sent, failed).

Non-Functional Requirements

Reliability: no notification is silently lost; delivery is at-least-once with retries.
High throughput: absorb bursts from many producers and fan out to third-party gateways without back-pressuring callers.
Low latency for priority traffic: time-sensitive notifications (for example, security alerts) are delivered ahead of bulk traffic.

Capacity Estimation

Assume 100 million notifications per day with an average record size of 500 bytes and a 30-day status-retention window.

QPS: 100,000,000 / 86,400 s ~ ~1,160 notifications/sec average; a 5x peak gives ~5,000/sec.
Bandwidth: ingest is small payloads - 1,160/sec x 1 KB ~ ~1.2 MB/s ~ 10 Mbps; outbound to third-party gateways is of the same order per channel.
Storage: 100e6 records/day x 500 bytes = ~50 GB/day; retained 30 days for status and auditing ~ ~1.5 TB.

High-Level Architecture

The core of the system is an event-driven message queue architecture. API servers receive notification requests, perform validation, and push them to an inbound queue. Workers process this queue, fetch user preferences, determine the required channels (e.g., Email vs SMS), and push the localized, formatted message to channel-specific outbound queues. Finally, channel-specific sender workers pull from these queues and interact with third-party APIs (like Twilio or SendGrid) to actually deliver the messages.

Data Model

Entity	Fields / Schema	Storage Choice
Notification	id, user_id, channel, template_id, payload, status, dedup_key, created_at	Wide-column / NoSQL store (Cassandra or DynamoDB)
Preference	user_id, channel, enabled, opt_out_at	Relational or key-value store (cached heavily in Redis)
Template	template_id, channel, locale, body	Key-value / document store
Dedup marker	dedup_key, ttl	In-memory key-value store with TTL (Redis)

Detailed Design

Message Brokers and Fan-out

The pipeline is built around a message broker (Kafka or RabbitMQ) using publish-subscribe. The API validates a request, checks preferences and deduplication, then publishes to a per-channel topic. Each channel has its own pool of workers subscribed to its topic, so push, SMS, and email scale independently and a slow email provider never stalls push delivery. This fan-out is the essence of event-driven design: producers emit one logical notification and the system routes it to whatever channels the user has enabled.

Deduplication & Idempotency

Because delivery is at-least-once, the same message can be processed more than once after a retry or a broker redelivery. Each notification carries a dedup_key (typically derived from the event id and channel). Before sending, a worker records that key in a short-lived Redis store; if the key already exists, the duplicate is dropped instead of re-sent.

Retry and Dead-Letter Queues

Third-party APIs (like Twilio) fail frequently due to rate limits or outages. Failed sends are retried with exponential backoff. Messages that repeatedly fail are routed to a dead-letter queue (DLQ) for inspection, ensuring they don't block the main processing queue forever.

Bottlenecks & Solutions

The main bottleneck is usually third-party API rate limits. If you try to send 10,000 SMS messages per second, Twilio might throttle you. The system must implement rate limiting on the outbound sender workers to respect these limits, which can cause queues to back up during massive burst events (like breaking news).

Interview Follow-up Questions

Q: How do you handle a scenario where a marketing team accidentally sends 100 million emails, clogging the queue for password resets?

We must implement Priority Queuing. Instead of one queue per channel, we have multiple: `email_high_priority` (password resets), `email_normal`, and `email_bulk` (marketing). Workers prioritize draining the high-priority queue before touching the bulk queue.

Q: Why not use a relational database to store the notification history?

At 100M writes per day, a relational DB would require significant tuning, sharding, and maintenance. Since we only need to write sequentially and read by `user_id` or `notification_id` (no complex JOINs), a wide-column store like Cassandra is vastly cheaper and scales writes horizontally much better.