Social Media News Feed
IntermediateOverview
A news feed service builds each user a personalized, reverse-chronological (or ranked) stream of posts from the accounts they follow. It must ingest a high write volume of posts while serving feed reads with low latency to a much larger read audience.
Functional Requirements
- Publish a post (text, image, or video reference) that reaches a user's followers.
- Generate and return a user's feed, ordered by recency or a ranking signal.
- Follow and unfollow other users, changing which posts appear in a feed.
- Support pagination so a client can scroll through older feed entries.
Non-Functional Requirements
- Low read latency: feed retrieval should return in under ~200 ms at p99.
- High availability for reads, favoring freshness-later over downtime.
- Eventual consistency is acceptable: a post may take seconds to appear in all feeds.
- Scale to hundreds of millions of users with highly skewed follower counts.
Capacity Estimation
Assume 500M daily active users (DAU), each reading the feed 10 times/day, and 100M new posts/day.
- QPS:
- Feed reads: 500M x 10 = 5B/day -> 5B / 86,400 s ~ 58,000 reads/sec (peak ~3x ~ 175,000/sec).
- Post writes: 100M / 86,400 s ~ 1,160 writes/sec.
- Storage: a post record (text + metadata, media stored separately) ~ 1 KB. Per day: 100M x 1 KB = 100 GB/day -> ~36.5 TB/year for post metadata, plus separate blob storage for media.
- Bandwidth:
- Post ingress: 1,160/s x 1 KB ~ 1.2 MB/s (metadata only).
- Feed egress: assume a page of 20 posts ~ 20 KB; 58,000/s x 20 KB ~ 1.16 GB/s served largely from cache.
High-Level Architecture
The architecture relies on a "Fan-Out" strategy. Because reading a feed is vastly more common than writing a post, the system pre-computes feeds. When a user posts, a Fan-Out Worker grabs their followers list and pushes the Post ID into a Redis cache for each follower (their materialized timeline). When a follower opens the app, the Feed Service just fetches their pre-computed list of Post IDs from Redis, hydrates them with post data from the DB, and returns them instantly.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| post | post_id (PK), author_id, text, media_url, created_at | Wide-column store sharded by post_id (Cassandra) |
| follow | follower_id, followee_id, created_at | Wide-column / graph store, indexed both directions |
| feed | user_id, ordered list of post_id + score | In-memory cache (Redis) (per-user materialized timeline) |
| media | media_id, bytes | Blob/object store (S3), referenced by URL |
Detailed Design
Fan-Out on Write vs. Fan-Out on Read
- Fan-Out on Write (Push): When User A posts, the system finds all their followers and 'pushes' the post into their pre-computed feed caches. Pros: Feed reads are O(1) and lightning fast. Cons: If User A has 50 million followers, pushing to 50 million caches takes minutes and wastes massive compute (the 'Hot Key' problem).
- Fan-Out on Read (Pull): When User B opens their app, the system looks up everyone they follow, queries the DB for their recent posts, merges, sorts, and returns them. Pros: No wasted writes. Cons: Tremendous read latency, especially if User B follows 5,000 people.
Hybrid Model for Celebrities
The standard industry solution is a Hybrid approach. Pure fan-out-on-write is used for normal users (e.g., < 100k followers). For celebrities (e.g., Justin Bieber), their posts are NOT pushed to followers' caches. Instead, when a user opens their feed, the system grabs their pre-computed cache (from normal friends) AND separately pulls the latest posts from any celebrities they follow (Fan-out on Read), merging them at request time.
Pagination
Standard offset/limit pagination fails as the feed constantly shifts with new posts. The API must use cursor-based pagination. The client sends max_id=12345 (the oldest post currently visible on screen), and the server returns the next 20 posts older than that ID.
Bottlenecks & Solutions
Pre-computing feeds for all users wastes massive amounts of memory in Redis, especially for 'inactive' users who rarely open the app. To save costs, the system should only keep feeds in Redis for Active Users (e.g., logged in within the last 14 days). If an inactive user logs in, their feed is rebuilt on the fly.