System Design Case Study: A News Feed

Open a social app and you see a feed: recent posts from the accounts you follow, newest or "best" first, loading instantly and scrolling forever. Behind it is one central question: when do you assemble each person's feed — when a post is written, or when the feed is read?

Everything else in the design — the celebrity problem, the hybrid, the storage layout — follows from how you answer that.

1. Requirements

Functional: post text and images; follow and unfollow accounts; see a feed of followed accounts' posts, newest first (ranking later); infinite scroll; like and reply.

Non-functional: the feed loads in under ~200 ms; a new post appears in followers' feeds within seconds; reads hugely outnumber writes; the feed may be slightly stale or incomplete, but must never show a post the viewer isn't allowed to see.

Out of scope: search, direct messages, ads (mention that ads are usually just another candidate source merged in at ranking time).

2. The numbers

200 million daily users, each opening the feed 10 times a day, and about 50 million posts a day. Average 200 followers per account, with a long tail up to tens of millions.

3. APIs

POST /posts                       { text, mediaIds[] }  → { postId }
POST /users/{id}/follow
DELETE /users/{id}/follow
GET  /feed?cursor=...&limit=20    → { posts: [...], nextCursor }
POST /posts/{id}/likes

The feed endpoint returns hydrated posts — author name, avatar, text, media URLs, counts — so the client renders in one round trip.

4. Option 1: fan-out on read (pull)

Build the feed when it's requested:

SELECT * FROM posts
WHERE author_id IN (SELECT followee FROM follows WHERE follower = :me)
ORDER BY created_at DESC LIMIT 20;

Writes are trivial — one row. Reads are expensive: for someone who follows 500 accounts, every feed load is a merge across 500 authors' recent posts. At 23,000 reads a second, that's a lot of repeated work, and latency grows with the number of accounts you follow.

Once posts are sharded by author, that query becomes a scatter-gather across many shards for every page view — and the slowest shard sets the response time.

5. Option 2: fan-out on write (push)

Do the work once per post instead. When someone posts, push the post's ID onto a precomputed feed list for each follower. Reading a feed is then one lookup of a list that already exists.

The numbers favour push: 600 posts a second × 200 followers = 120,000 small list writes a second, done asynchronously by workers. That buys 23,000 cheap reads a second with predictable latency. Feed lists hold only post IDs (8 bytes each), capped at a few hundred, so 200 million feeds × 800 IDs × 8 bytes ≈ 1.3 TB — a large but ordinary Redis cluster.

The fan-out worker, simplified:

// consumes "post created" events from a log
async function fanOut({ postId, authorId }) {
  if (await isCelebrity(authorId)) return;          // handled at read time

  for await (const batch of followers.inBatches(authorId, 1000)) {
    const active = batch.filter((f) => f.lastSeenDays < 30);
    const pipe = redis.pipeline();
    for (const f of active) {
      pipe.zadd(`feed:${f.id}`, postId, postId);     // score = time-sortable ID
      pipe.zremrangebyrank(`feed:${f.id}`, 0, -801); // keep newest 800
    }
    await pipe.exec();
  }
}

A sorted set keyed by a time-sortable post ID keeps each feed ordered and makes the trim a single command. Pipelining sends a thousand followers' writes in one round trip. And skipping inactive followers cuts the write volume dramatically — in many apps, most accounts haven't opened the app this month.

Because the post-created event comes from a log, a worker crash just means the batch is replayed. ZADD of the same ID twice is harmless — the write is idempotent by construction.

6. The celebrity problem

Push breaks on the long tail. An account with 40 million followers turns one post into 40 million writes: the fan-out queue backs up, and followers see the post minutes late.

The hybrid most large feeds use:

• Accounts below a follower threshold (say 10,000): fan-out on write.
• Accounts above it: not fanned out. At read time, the feed service fetches recent posts from the handful of big accounts the reader follows and merges them with the precomputed list.

Most readers follow only a few huge accounts, so the read-time merge is small. Inactive users are another cheap win: don't maintain feeds for accounts that haven't logged in for months, and rebuild one with a pull query when they come back.

async function readFeed(userId, cursor, limit = 20) {
  const [pushed, celebs] = await Promise.all([
    redis.zrevrangebyscore(`feed:${userId}`, `(${cursor}`, '-inf', 'LIMIT', 0, limit),
    followedCelebrities(userId),                      // usually a handful
  ]);
  const pulled = await Promise.all(
    celebs.map((c) => recentPostIds(c, cursor, limit)) // cached per celebrity
  );
  const ids = mergeDescending([pushed, ...pulled]).slice(0, limit);
  return hydrate(ids);
}

The celebrity side is cheap because recentPostIds(celebrity) is the same for every reader — one cache entry per celebrity, read millions of times. The expensive part of pull (a different query per reader) only applies to the push side, which is already precomputed.

7. Storing posts and media

• Posts: a table keyed by post ID. Use time-sortable IDs so the IDs in feed lists also encode order.
• Media: images and video go to object storage and are served through a CDN. The post stores only URLs.
• Hydration: a feed read gets 20 IDs, then fetches those posts in one batch from a post cache. Counts that change constantly — likes, replies — are fetched separately, so updating them doesn't invalidate the post.

The social graph — who follows whom — is its own store, queried two ways: "who does X follow" (feed reads, celebrity merge) and "who follows X" (fan-out). Keep both directions as separate tables keyed by the first user, so each query is a single partition read.

Counters

A like is a write to a counter that a million people may be incrementing at once. Don't UPDATE posts SET likes = likes + 1 on a hot row. Buffer increments and flush them in batches, or spread one logical counter over several sharded counters and sum them on read. A like count that's a few seconds behind is fine; a write hotspot that slows every other query on that partition is not.

8. Pagination

Offset pagination breaks here, because new posts shift every offset while the user scrolls. Use a cursor — "posts older than ID X" — which stays correct however many posts arrive. See cursor vs. offset pagination.

With time-sortable IDs, the cursor is just the last post ID the client saw. When the user scrolls past the ~800 IDs kept in their precomputed list, fall back to a pull query for older pages — rare, and acceptable to be slower.

9. Ranking

A chronological feed is the baseline. Ranked feeds add a second stage: fetch a few hundred candidates (the precomputed list plus celebrity posts, plus perhaps recommended posts), score them with a model using features such as author affinity, recency and predicted engagement, and return the top 20. The candidate stage is the architecture above; ranking is a service that runs on top of it.

Two consequences follow. A ranked feed can't use "older than ID X" as its cursor, since order no longer follows IDs — store the ranked list for the session and page through that. And ranking makes the 200 ms budget tight: candidate fetch, feature lookup and model scoring must all fit, so features are precomputed and cached, and the model is small enough to score hundreds of candidates in a few milliseconds.

10. Privacy and deletes

A post's audience can change after fan-out: the author deletes it, blocks a follower, or makes their account private. Chasing every copy through millions of feed lists is expensive and racy. Instead, filter at read time: hydration checks that the post still exists and that the viewer may see it, and drops anything that fails. Feed lists are a cache of candidates, never a source of truth for permissions.

Follow and unfollow

Changing the graph changes feeds that already exist:

• Follow: the new followee's recent posts aren't in your list, because fan-out happened before you followed. Backfill asynchronously — fetch their last few dozen posts and ZADD them into your feed. Until that finishes, the next read can merge them in pull-style, so the follow feels instant.
• Unfollow: their posts are still in your list. Remove them in the background (a scan of 800 IDs is cheap), and filter by the current follow set at read time meanwhile.

Both operations are rare compared with reads, so doing a little extra work here to keep the read path simple is the right trade.

Bootstrapping a new user

A brand-new account follows nobody, so the push path gives them an empty feed. New users get candidates from elsewhere — popular posts in their region, suggested accounts — mixed in at the ranking stage, until their own graph is big enough to fill a page. It's the same candidate-plus-ranking architecture, with a different candidate source.

Multiple regions

Users in different regions follow each other, so a post written in one region has followers everywhere. Replicate the post-created log across regions and run fan-out workers in each region for the followers who live there; each region keeps feed lists only for its own users. Reads stay local. The cost is a cross-region replication delay before a post appears for distant followers — usually under a second, and invisible next to everything else in the feed.

11. What breaks first

• Fan-out lag during spikes: queue depth grows. Workers scale horizontally; prioritise fan-out for active followers.
• Unfollows and deletes: a deleted post may still be in a million feed lists. Filter at read time (hydration finds no post) rather than chasing every copy.
• Hot posts: one viral post read millions of times. It's in every cache, so replicate hot keys across cache nodes.
• Cache loss: if a feed cache node dies, its users' feeds rebuild with a pull query on next read. Throttle rebuilds so a node failure doesn't become a database stampede — see cache stampedes.

12. Trade-offs to state out loud

13. Follow-up questions to expect

• "Where's the celebrity threshold?" Where fan-out cost exceeds the read-time merge cost — measure it. Some systems also push a celebrity's posts to their most engaged followers only.
• "New post doesn't show up for the author." The author's own posts are merged in at read time from their recent posts, so they never wait on fan-out — read-your-writes again.
• "Real-time updates while the feed is open." Don't push full posts; show a "new posts" pill from a lightweight notification over an existing connection, and load them on tap.
• "Edits." Feed lists hold IDs, so an edited post is just a new version in the post store and cache. Nothing else changes.
• "Mute a word or an account." Another read-time filter in hydration, alongside privacy checks. Muting never touches the precomputed lists, so un-muting is instant too.

Practise it

"News feed" is a prompt in the system design practice tool — draw the fan-out workers and feed cache, and have Claude check whether your notes justify every box.