Design a Notification System

• Send a notification to a user or a set of users — Internal services call one API with a template id and data; the system decides channels, renders, and delivers. Callers never talk to APNs, FCM or an email provider directly.
• Multiple channels — Mobile push (iOS, Android), web push, email, SMS, and in-app inbox. A notification may go to several channels with different content per channel.
• User preferences and opt-outs — Per category and per channel, plus legal opt-outs (marketing email unsubscribe, SMS STOP) that override everything. Quiet hours per user timezone.
• Priority — Transactional (a login code, a ride arriving) is delivered within seconds and bypasses batching and quiet hours; marketing is low priority, batched, and rate limited.
• Deduplication and rate limiting — A user never gets the same notification twice, and never more than N marketing messages a day. Digest similar events ("5 people liked your post").
• Scheduling — Send at a given time, in the user's local time, or within a window; cancel a scheduled send.
• Delivery tracking — Sent, delivered, opened, clicked, bounced per notification; feed back into templates and preferences (auto-unsubscribe hard bounces).
• Out of scope — Composing marketing campaigns (audience selection UI), the in-app inbox UI, and content personalisation models. We deliver what we are given.

• Scale (500 M users · 10 B notifications/day) — Around 115 k/s average, with marketing blasts adding millions in minutes. The design must absorb bursts without delaying transactional traffic.
• Latency for transactional (p99 < 5 s end to end) — A one-time code that arrives after the user gave up is a failed login. Priority isolation is the hardest tradeoff: bursts of low-priority traffic must not touch this path.
• At-least-once delivery, effectively once (duplicates < 0.01 %) — Providers and retries produce duplicates; idempotency keys at every hop keep them from reaching the user.
• Availability (99.99 % for accept · providers may be down) — The API must accept even when APNs or the email provider is failing; delivery is retried later.
• Compliance (opt-outs honoured within seconds) — CAN-SPAM, GDPR, TCPA: sending marketing after an unsubscribe is a legal problem, not a bug.
• Observability (every notification traceable) — Support must answer "did user X get message Y and why not" from a single lookup.
• Cost (SMS is 1000× email) — Channel selection and dedupe are cost controls as much as UX.

• Notifications per second: ~115 k avg · 1 M+ peak — 10 B/day ÷ 86 400 s ≈ 116 k/s. A marketing campaign to 100 M users released over 10 minutes adds ~170 k/s; several at once, plus a breaking-news style event, reach 1 M/s. Design queues for the peak, delivery for a smoothed rate.
• Channel split: ~70 % push · 25 % email · 4 % in-app only · 1 % SMS — Typical consumer mix. 1 % SMS of 10 B is 100 M SMS/day; at ~$0.01 each that is $1 M/day, which is why SMS is reserved for auth codes and critical alerts.
• Device tokens: ~1.5 B — 500 M users × ~3 devices/tokens (phone, tablet, browser) = 1.5 B tokens at ~200 B each ≈ 300 GB. Fits a sharded key-value store keyed by user; tokens churn constantly (reinstalls), so the store must handle high update rates and invalidation feedback.
• Provider calls per second: ~100 k/s to APNs/FCM — 70 % of 116 k/s is ~80 k/s push sends on average, each an HTTP/2 request to APNs or FCM (batched for FCM). Providers accept this from one account; the constraint is per-connection throughput, so hundreds of persistent HTTP/2 connections.
• Delivery-state storage: ~2 TB/day — One record per notification with status transitions: ~200 B × 10 B = 2 TB/day. Keep 30 days hot for support lookups (60 TB), then aggregate. A wide-column store keyed by user id with time-ordered notification ids.
• Dedupe keys: ~10 B/day · ~250 GB — One idempotency key per (user, notification key) with a 24 h TTL: 10 B × ~25 B = 250 GB live in a Redis cluster, or in the same wide-column store with TTL. Bloom filters are not appropriate: a false positive drops a real notification.
• Email throughput: ~30 k/s — 25 % of 116 k/s ≈ 29 k/s. Large ESPs and self-hosted MTAs deliver this with a pool of sending IPs; reputation warm-up means the pool must be built up over weeks, which is an operational constraint worth mentioning.

• Calling services — Internal producers: order service, social graph, auth, marketing tools. They send a notification request with a template id, recipient(s), data, priority and an idempotency key. They never know about channels or providers.
• Notification API — Validates the request, checks the idempotency key, enriches with the template category and priority, and writes it to the appropriate priority topic. Returns 202 with a notification id in milliseconds; nothing downstream happens synchronously.
• Priority topics (Kafka · critical / high / bulk) — Separate topics per priority with separate consumer fleets, so a marketing blast in the bulk topic cannot delay a login code in the critical topic. Keyed by user id so one user's notifications are processed in order. Durable buffer for provider outages.
• Routing service (preferences + channel selection) — For each notification: load user preferences, opt-outs, quiet hours, timezone, and devices; apply category and channel rules; apply per-user rate limits and digest rules; decide the channel set; render templates per channel; enqueue one delivery task per channel. The brain of the system.
• Preferences store (KV by user · cached) — Per-user preferences by category and channel, legal opt-outs, quiet hours and timezone. Read on every notification, so it is cached aggressively (a few hundred bytes per user) with invalidation on change. Opt-outs are written synchronously and are the last check before any send.
• Device registry (user → tokens · endpoints) — Push tokens per user and platform, email addresses with verification and bounce state, phone numbers with consent. Updated by clients on login and token refresh, and by provider feedback (invalid token, hard bounce).
• Template service (versioned · per channel · localised) — Renders content per channel and locale from a template id and data. Versioned so a running campaign is not changed mid-flight. Also holds category metadata (transactional vs marketing, default channels, TTL).
• Rate limit + dedupe (Redis · per user counters · idempotency keys) — Idempotency keys for the API and for provider sends (24 h TTL), per-user daily counters per category, and the digest buffer that collects similar events for a short window before sending one summary.
• Delivery queues (per channel · per priority) — One queue per (channel, priority): push-critical, push-bulk, email-bulk, sms-critical and so on. Each is consumed by a worker fleet sized for that provider's throughput and with its own retry and backoff policy. Isolation means an email provider outage backs up only email.
• Channel workers (APNs / FCM / SMTP / SMS adapters) — Stateless senders per channel with persistent connections to providers, provider-side idempotency where available, circuit breakers, and exponential backoff. They write delivery state and forward provider feedback (invalid tokens, bounces) to the device registry.
• Providers (APNs · FCM · SES · Twilio) — Third parties that actually reach the device or inbox. Each has its own limits, error semantics and feedback channel. They fail independently and sometimes silently.
• Delivery state (Cassandra · key=user id, time-ordered) — Every notification's lifecycle: created, routed, sent per channel, delivered, opened, failed with reason. Written by the router and workers, read by the in-app inbox and by support tooling. Time-ordered ids give cursor pagination per user.
• In-app inbox — Serves the notification list and unread count for the app from the delivery-state store, and marks read. The in-app channel is "free" and is where everything goes even if push is off.
• Scheduler (time-bucketed sends) — Holds notifications with a future send time in time-bucketed storage and releases them into the priority topics when due, converting local-time windows into UTC per user timezone. Cancellation deletes the row before release.
• Analytics + feedback (stream: opens, clicks, bounces) — Consumes delivery and engagement events for dashboards (delivery rate per channel and provider, open rates per template) and feeds back into preferences: auto-reduce frequency for users who never open, auto-unsubscribe hard bounces.

1. Send a login code: the critical path — One user, one SMS or push, under five seconds, no matter what else is happening. Every hop is isolated from bulk traffic and idempotent.
   1. Auth service calls the notification API with a template, recipient, data, priority and idempotency key.
   2. API checks the idempotency key and writes the request to the critical topic.
   3. Router loads preferences and devices, and applies the transactional rules.
   4. Router renders the channel-specific content and enqueues delivery tasks on the critical delivery queues.
   5. SMS worker sends via the provider with a send-level idempotency key and records the result.
   6. Provider delivery receipts arrive by webhook and update the state.
2. Marketing blast to 100 M users — The bulk path. It must be fast enough to finish in a reasonable window and must never touch the critical path's capacity, preferences, or provider quota.
   1. Campaign tool submits one request with an audience reference, not 100 M user ids.
   2. Expander streams user ids into the bulk topic, bucketed by the user's timezone window.
   3. Bulk routers apply marketing preferences, opt-outs, frequency caps and quiet hours strictly.
   4. Bulk delivery queues drain into channel workers at provider-safe rates.
   5. Engagement and bounce feedback flow back into preferences and the device registry.
3. APNs is down for 40 minutes — Providers fail. The system must keep accepting, avoid a retry storm, deliver critical messages by another channel, and drain the backlog in order of priority when the provider returns.
   1. Push workers see timeouts and 5xx from APNs; the circuit breaker opens.
   2. Push delivery queues grow; the API keeps accepting normally.
   3. Critical notifications whose push attempt fails fall back to SMS or email within their TTL.
   4. When APNs recovers, the backlog drains critical-first with per-user coalescing.
   5. Alerts fire on breaker state and queue age, not just on error rate.
4. Twelve likes in a minute become one notification — Social events arrive in bursts. Without digesting, users get spammed and providers get unnecessary load. The router buffers similar low-priority events per user for a short window and sends a summary.
   1. Twelve "like" events for one post arrive at the API within a minute.
   2. Router sees a digest key and adds the event to the user's digest buffer instead of sending.
   3. When the window closes, the router renders one summary from the buffered events.
   4. The summary goes through the normal delivery path with one dedupe key for the whole group.
   5. The in-app inbox shows the same grouped item, updated in place as more events arrive.
5. User unsubscribes while a campaign is mid-flight — Opt-outs are legal requirements with a deadline. The check must happen at the last possible moment, and the write must propagate faster than the queues drain.
   1. User clicks unsubscribe in an email; the link writes the opt-out synchronously.
   2. A campaign task for this user is already sitting in the email bulk delivery queue.
   3. The email worker re-checks opt-outs immediately before sending.
   4. Scheduled sends for the user are filtered at release time as well.
   5. The opt-out is logged with its source and timestamp for audit.

Ask: Why separate topics and fleets per priority instead of a priority field in one queue?

Good answers name: Separate topics, consumer fleets, delivery queues and provider pools per priority tier, One Kafka topic with a priority field, A priority queue (RabbitMQ priorities, Redis sorted set), Rate-limit bulk producers so they never saturate.

Our pick: Three tiers: critical (auth, safety, payments; seconds), high (social, transactional updates; a minute), bulk (marketing, digests; hours). Each has its own Kafka topic keyed by user id, its own router consumer group, its own delivery queues per channel, its own worker fleet with its own provider connections and quota share, and its own SLO and alerts. The critical tier is over-provisioned (it is small) and never autoscaled down below a floor. Bulk producers are additionally admitted at a controlled rate by the expander. Cross-tier sharing is limited to read-only caches (preferences, devices), which are sized for the bulk rate so critical reads never miss. In an incident, bulk is the tier you pause.

1. Bulk is backed up by two hours and someone submits a "high" campaign. Where does it go?
   The tier is set by the template category, not by the caller; a marketing template cannot claim high. If the product genuinely needs a faster marketing lane, add a fourth tier with its own capacity rather than letting callers pick. Priority you can request is priority everyone requests.
2. Critical volume is 1 % of traffic. Is a whole fleet for it wasteful?
   It is a few instances and a few partitions; the cost is negligible next to the incident it prevents. Keep the critical fleet's minimum size at 2× its peak so a deploy or a zone loss does not degrade it.
3. How do you keep per-user ordering if tiers are separate?
   You do not, across tiers, and that is fine: a login code and a marketing email have no ordering relationship. Within a tier, keying by user id gives order. If two notifications in different tiers must be ordered (rare), put them in the same tier.
4. What is the SLO per tier and how do you measure it?
   Critical: 99.9 % delivered-to-provider within 5 s of accept. High: within 60 s. Bulk: campaign completes within its window. Measured from the state store timestamps (accepted → sent) and from provider receipts for end-to-end. Alert on burn rate for critical; on queue age for the others.

Ask: Kafka is at-least-once, workers crash, providers time out. How does a user never get the same push twice?

Good answers name: Idempotency keys at API, router and per-channel send, with a reconciliation path for ambiguous provider results, Kafka transactions end to end, At-most-once: never retry, Rely on provider-side dedupe (APNs collapse id, email Message-ID).

Our pick: API: SET NX on the caller's idempotency key (24 h) returns the existing notification id on repeats. Router: processing is idempotent by notification id; the routed state is written with a conditional put, and delivery tasks carry a deterministic send key (notification id + channel). Worker: SET NX on the send key before the provider call; if present, skip. Provider timeout: leave the key set, mark state "unknown", and let a reconciler query the provider by client reference (most support it) to settle sent or failed; only a confirmed failure clears the key and re-enqueues. Push uses APNs collapse-id and FCM collapse keys so even a rare duplicate replaces rather than stacks. Redis unavailability fails open for bulk (a rare duplicate is cheaper than a stall) and fails closed for critical channels like SMS (retry later rather than double-charge and double-text).

1. The worker sets the send key, then crashes before calling the provider. Now the message is never sent.
   The key has a short "in-progress" TTL (say 60 s) distinct from the 24-hour "sent" state: SET NX with EX 60, then after a successful send, extend to 24 h. If the worker dies, the key expires and the redelivered task sends. The window of a duplicate is a worker that took over 60 s to call the provider, which timeouts prevent.
2. Redis loses the dedupe keys (failover, memory pressure). What happens?
   Some in-flight redeliveries can duplicate. Bound it: keys are also written to the delivery state store (the durable record of "sent"), and workers check state on a Redis miss for critical channels. Run Redis with replication and disable eviction on the dedupe keyspace; a full Redis is an alert, not a silent drop.
3. Caller sends the same logical event with different idempotency keys.
   Then it is two notifications, by contract. Protect the user anyway with a content-level dedupe: hash of (user, template, key fields) with a short TTL (e.g. 10 minutes) in the router, which collapses accidental re-sends from buggy producers. Log and alert on the collapse rate per producer.
4. How do you test this?
   Chaos in staging: kill workers mid-send, inject provider timeouts and 5xx, replay Kafka offsets, and assert zero duplicates at a fake provider that records every call. Run the same fault injection in production against a canary user pool with a fake provider endpoint. Duplicate rate per channel is a tracked metric from provider receipts.

Ask: Who decides that a notification is a push, an email, or both, and how do preferences, opt-outs and cost fit in?

Good answers name: Central router with layered policy: legal → category → user prefs → device → cost → frequency, Caller chooses channels, Rules engine evaluated at send time by workers, ML model chooses the channel per user.

Our pick: Templates declare a category (transactional, social, marketing), default channels, fallback order, TTL and whether quiet hours apply. The router evaluates in order: legal opt-outs per channel (absolute); category rules (transactional ignores marketing opt-outs and quiet hours); the user's per-category channel preferences; device availability (a push token active in 30 days beats email; no token means email; SMS only where the template allows and the number is verified); cost preference (push over email over SMS when equivalent); frequency caps and digest windows; quiet hours with deferral. The output is the channel set and per-channel rendering. Preferences and device records are cached in Redis with write-through invalidation and a 5-minute TTL; the critical tier reads through on a miss. Every decision is written to the state record with a reason code, so "why did I not get this" is one lookup.

1. The preference cache is stale by up to 5 minutes. Is that acceptable?
   For preferences, yes; for legal opt-outs, no. Opt-outs invalidate the cache synchronously on write and are re-checked by the worker at send time from a source that was invalidated, so the effective window is seconds. Preferences that only affect channel choice can lag.
2. A user has three devices. Push to all, or one?
   All active tokens for informational pushes (APNs collapse id keeps the display tidy), because you do not know which device is in hand. For high-cost or interactive actions (approve this login), the most recently active device first with a short fallback to the others. Delete tokens the provider reports invalid, and expire tokens unseen for 90 days.
3. How do you handle a user with no channels available?
   The in-app inbox always receives it, so nothing is lost. Record state "no_channel" with the reason so the product can prompt for a verified email or push permission. For critical auth flows, the caller gets a synchronous answer (from the state or a lookup endpoint) that no out-of-band channel exists, and can offer a different factor.
4. Where does localisation happen?
   In the template service at render time, using the user's locale from the preferences record and the template's translated variants. Rendering is per channel because email needs HTML and a subject, push needs a 100-character body, SMS needs plain text with the sender id. Templates are versioned; a campaign pins a version at submission.

Ask: Ten billion lifecycle records a day. What stores them, how are they keyed, and how does the in-app inbox read them?

Good answers name: Wide-column store partitioned by user id, clustered by time-ordered notification id, with TTL; unread counter in Redis, Relational table per shard, Only an event log (Kafka) and derived analytics, Store the inbox in the app (push everything, no server inbox).

Our pick: A Cassandra or Scylla table notifications_by_user with partition key (user_id, month bucket) and clustering key notification_id descending (a time-ordered ULID or Snowflake id), columns for template, category, channels, per-channel status and timestamps, digest group, and reason codes, with a 30-day TTL on the hot table and a nightly job that writes aggregates (per template, per channel, per day) to the analytics warehouse. A second table notifications_by_id maps id → (user_id, bucket) for tracking lookups. The unread count is a Redis counter per user incremented on in-app write and reset on read, with a periodic reconciliation. The inbox API reads the newest page with a cursor of the last notification id and groups digests on read. Support tooling reads by user or by id and gets every state transition with reasons.

1. The router writes "routed", then two workers write "sent" for two channels concurrently. Any conflict?
   Each writes different columns (push_status, email_status) of the same row; Cassandra merges by column with timestamps, so there is no lost update. Timestamps come from the writers, so keep clocks synced; or use lightweight transactions for the rare state that must be exclusive (such as the fallback decision).
2. A user has 50 000 notifications in a month (a heavy social user). Is the partition too big?
   At 200 bytes each that is 10 MB, within Cassandra's comfortable partition size (under 100 MB). The month bucket bounds it. For outliers, bucket by week. Reads always page from the newest, so partition size mostly affects compaction, not latency.
3. How does the unread count stay correct?
   It does not need to be exactly correct; it needs to be plausible and to hit zero when the inbox is read. Increment on write of an in-app notification, set to zero on open, and reconcile from a count query occasionally or when a client reports a mismatch. Badges are the one place where a stale count is very visible, so the reset path must be reliable.
4. Retention: legal wants 7 years of "we sent this" for some categories.
   Keep the hot store at 30 days and archive the events (not the rows) to object storage in partitioned Parquet by day and category; 7 years of 10 B events a day is about 5 PB compressed, which is affordable in cold storage and queryable with a batch engine when a regulator asks.

Ask: How do you send 100 M messages "at 10 a.m. local time" without scanning a table every minute?

Good answers name: Time-bucketed partitions (per minute) in a wide-column store, released by a leader-elected sweeper into the priority topics, Redis sorted set by due time, Kafka with per-delay topics, Cron-style scanning of a send_at index.

Our pick: A scheduled table partitioned by (due_minute_utc, shard) where shard = hash(user) mod 64, clustered by notification id, with the full request payload. Submitting a local-time send converts to UTC per user timezone at submission and writes one row. A sweeper (leader-elected per shard range) reads the current minute's partitions, publishes each row to the appropriate priority topic, and marks the partition released; the router applies preferences and quiet hours at release time, so late changes are honoured. Cancellation deletes the row or, for a whole campaign, marks the campaign cancelled and the sweeper skips its rows. Clock skew is handled by sweeping a minute a few seconds after it ends, and a missed minute is caught by a catch-up scan of recent unreleased partitions. Digest windows (under a few minutes) use a Redis sorted set instead, because they are short-lived and high-churn.

1. The sweeper crashes after publishing half of a minute's partition. What happens on restart?
   The partition is not marked released, so the new sweeper republishes it; the duplicates are absorbed by the idempotency key at the API/router hop, which is exactly what it is for. Marking per row instead of per partition avoids even that, at the cost of 100 M extra writes; per partition with idempotency is the better trade.
2. A user changes timezone after a send was scheduled in local time.
   The row stores the UTC time computed at submission; the change is not reflected unless you re-derive. For campaigns, store the local target alongside and have the router at release time check whether the user's current local time is within the window; if not, re-schedule once. Rare enough that this approximation is fine.
3. Everyone in the largest timezone is due at 10:00. How hot is that minute?
   Tens of millions of rows in 64 shard partitions, a few hundred thousand each; readable in parallel by the sweeper fleet in a minute or two. Smooth it deliberately: the expander jitters the due minute within the send window (10:00 to 10:20) so release, routing and providers see a steady rate rather than a spike.
4. How do you support "remind me in 3 days unless I complete the task"?
   Schedule it with a cancellation key (user, task); the task-completion event calls DELETE on that key, which maps to the row through a small index. Alternatively the router re-checks a predicate at release ("task still incomplete?") via a callback URL supplied by the caller, which also handles cases the caller forgot to cancel.