Design a Notification System

Deliver push, email, SMS and in-app notifications to hundreds of millions of users: the right channel, once, on time, and never at 3 a.m.

Difficulty: medium. Patterns: fan-out, multi-channel, idempotency, rate-limiting, priority-queues. Reported at Meta, Amazon, Uber, LinkedIn, Airbnb, Twilio, Microsoft.

Study shows every answer; Practice hides them until you have produced your own.

Functional requirements

• Send a notification to a user or a set of users. Internal services call one API with a template id and data; the system decides channels, renders, and delivers. Callers never talk to APNs, FCM or an email provider directly.
• Multiple channels. Mobile push (iOS, Android), web push, email, SMS, and in-app inbox. A notification may go to several channels with different content per channel.
• User preferences and opt-outs. Per category and per channel, plus legal opt-outs (marketing email unsubscribe, SMS STOP) that override everything. Quiet hours per user timezone.
• Priority. Transactional (a login code, a ride arriving) is delivered within seconds and bypasses batching and quiet hours; marketing is low priority, batched, and rate limited.
• Deduplication and rate limiting. A user never gets the same notification twice, and never more than N marketing messages a day. Digest similar events ("5 people liked your post").
• Scheduling. Send at a given time, in the user's local time, or within a window; cancel a scheduled send.
• Delivery tracking. Sent, delivered, opened, clicked, bounced per notification; feed back into templates and preferences (auto-unsubscribe hard bounces).
• Out of scope. Composing marketing campaigns (audience selection UI), the in-app inbox UI, and content personalisation models. We deliver what we are given.

Non-functional requirements

• Scale (500 M users · 10 B notifications/day). Around 115 k/s average, with marketing blasts adding millions in minutes. The design must absorb bursts without delaying transactional traffic.
• Latency for transactional (p99 < 5 s end to end). A one-time code that arrives after the user gave up is a failed login. Priority isolation is the hardest tradeoff: bursts of low-priority traffic must not touch this path.
• At-least-once delivery, effectively once (duplicates < 0.01 %). Providers and retries produce duplicates; idempotency keys at every hop keep them from reaching the user.
• Availability (99.99 % for accept · providers may be down). The API must accept even when APNs or the email provider is failing; delivery is retried later.
• Compliance (opt-outs honoured within seconds). CAN-SPAM, GDPR, TCPA: sending marketing after an unsubscribe is a legal problem, not a bug.
• Observability (every notification traceable). Support must answer "did user X get message Y and why not" from a single lookup.
• Cost (SMS is 1000× email). Channel selection and dedupe are cost controls as much as UX.

Back-of-envelope estimates

• Notifications per second: ~115 k avg · 1 M+ peak. 10 B/day ÷ 86 400 s ≈ 116 k/s. A marketing campaign to 100 M users released over 10 minutes adds ~170 k/s; several at once, plus a breaking-news style event, reach 1 M/s. Design queues for the peak, delivery for a smoothed rate.
• Channel split: ~70 % push · 25 % email · 4 % in-app only · 1 % SMS. Typical consumer mix. 1 % SMS of 10 B is 100 M SMS/day; at ~$0.01 each that is $1 M/day, which is why SMS is reserved for auth codes and critical alerts.
• Device tokens: ~1.5 B. 500 M users × ~3 devices/tokens (phone, tablet, browser) = 1.5 B tokens at ~200 B each ≈ 300 GB. Fits a sharded key-value store keyed by user; tokens churn constantly (reinstalls), so the store must handle high update rates and invalidation feedback.
• Provider calls per second: ~100 k/s to APNs/FCM. 70 % of 116 k/s is ~80 k/s push sends on average, each an HTTP/2 request to APNs or FCM (batched for FCM). Providers accept this from one account; the constraint is per-connection throughput, so hundreds of persistent HTTP/2 connections.
• Delivery-state storage: ~2 TB/day. One record per notification with status transitions: ~200 B × 10 B = 2 TB/day. Keep 30 days hot for support lookups (60 TB), then aggregate. A wide-column store keyed by user id with time-ordered notification ids.
• Dedupe keys: ~10 B/day · ~250 GB. One idempotency key per (user, notification key) with a 24 h TTL: 10 B × ~25 B = 250 GB live in a Redis cluster, or in the same wide-column store with TTL. Bloom filters are not appropriate: a false positive drops a real notification.
• Email throughput: ~30 k/s. 25 % of 116 k/s ≈ 29 k/s. Large ESPs and self-hosted MTAs deliver this with a pool of sending IPs; reputation warm-up means the pool must be built up over weeks, which is an operational constraint worth mentioning.

Components

• Calling services: Internal producers: order service, social graph, auth, marketing tools. They send a notification request with a template id, recipient(s), data, priority and an idempotency key. They never know about channels or providers.
• Notification API: Validates the request, checks the idempotency key, enriches with the template category and priority, and writes it to the appropriate priority topic. Returns 202 with a notification id in milliseconds; nothing downstream happens synchronously.
• Priority topics (Kafka · critical / high / bulk): Separate topics per priority with separate consumer fleets, so a marketing blast in the bulk topic cannot delay a login code in the critical topic. Keyed by user id so one user's notifications are processed in order. Durable buffer for provider outages.
• Routing service (preferences + channel selection): For each notification: load user preferences, opt-outs, quiet hours, timezone, and devices; apply category and channel rules; apply per-user rate limits and digest rules; decide the channel set; render templates per channel; enqueue one delivery task per channel. The brain of the system.
• Preferences store (KV by user · cached): Per-user preferences by category and channel, legal opt-outs, quiet hours and timezone. Read on every notification, so it is cached aggressively (a few hundred bytes per user) with invalidation on change. Opt-outs are written synchronously and are the last check before any send.
• Device registry (user → tokens · endpoints): Push tokens per user and platform, email addresses with verification and bounce state, phone numbers with consent. Updated by clients on login and token refresh, and by provider feedback (invalid token, hard bounce).
• Template service (versioned · per channel · localised): Renders content per channel and locale from a template id and data. Versioned so a running campaign is not changed mid-flight. Also holds category metadata (transactional vs marketing, default channels, TTL).
• Rate limit + dedupe (Redis · per user counters · idempotency keys): Idempotency keys for the API and for provider sends (24 h TTL), per-user daily counters per category, and the digest buffer that collects similar events for a short window before sending one summary.
• Delivery queues (per channel · per priority): One queue per (channel, priority): push-critical, push-bulk, email-bulk, sms-critical and so on. Each is consumed by a worker fleet sized for that provider's throughput and with its own retry and backoff policy. Isolation means an email provider outage backs up only email.
• Channel workers (APNs / FCM / SMTP / SMS adapters): Stateless senders per channel with persistent connections to providers, provider-side idempotency where available, circuit breakers, and exponential backoff. They write delivery state and forward provider feedback (invalid tokens, bounces) to the device registry.
• Providers (APNs · FCM · SES · Twilio): Third parties that actually reach the device or inbox. Each has its own limits, error semantics and feedback channel. They fail independently and sometimes silently.
• Delivery state (Cassandra · key=user id, time-ordered): Every notification's lifecycle: created, routed, sent per channel, delivered, opened, failed with reason. Written by the router and workers, read by the in-app inbox and by support tooling. Time-ordered ids give cursor pagination per user.
• In-app inbox: Serves the notification list and unread count for the app from the delivery-state store, and marks read. The in-app channel is "free" and is where everything goes even if push is off.
• Scheduler (time-bucketed sends): Holds notifications with a future send time in time-bucketed storage and releases them into the priority topics when due, converting local-time windows into UTC per user timezone. Cancellation deletes the row before release.
• Analytics + feedback (stream: opens, clicks, bounces): Consumes delivery and engagement events for dashboards (delivery rate per channel and provider, open rates per template) and feeds back into preferences: auto-reduce frequency for users who never open, auto-unsubscribe hard bounces.

User flows

1. Send a login code: the critical path. One user, one SMS or push, under five seconds, no matter what else is happening. Every hop is isolated from bulk traffic and idempotent.
2. Marketing blast to 100 M users. The bulk path. It must be fast enough to finish in a reasonable window and must never touch the critical path's capacity, preferences, or provider quota.
3. APNs is down for 40 minutes. Providers fail. The system must keep accepting, avoid a retry storm, deliver critical messages by another channel, and drain the backlog in order of priority when the provider returns.
4. Twelve likes in a minute become one notification. Social events arrive in bursts. Without digesting, users get spammed and providers get unnecessary load. The router buffers similar low-priority events per user for a short window and sends a summary.
5. User unsubscribes while a campaign is mid-flight. Opt-outs are legal requirements with a deadline. The check must happen at the last possible moment, and the write must propagate faster than the queues drain.

Deep dives

1. Priority isolation. Why separate topics and fleets per priority instead of a priority field in one queue?
2. Effectively-once delivery. Kafka is at-least-once, workers crash, providers time out. How does a user never get the same push twice?
3. Channel selection and preferences. Who decides that a notification is a push, an email, or both, and how do preferences, opt-outs and cost fit in?
4. Delivery state and the inbox. Ten billion lifecycle records a day. What stores them, how are they keyed, and how does the in-app inbox read them?
5. Scheduled and local-time sends. How do you send 100 M messages "at 10 a.m. local time" without scanning a table every minute?