System design interview question

Design a Notification System

Q: Why separate topics and fleets per priority instead of a priority field in one queue?

Three tiers: critical (auth, safety, payments; seconds), high (social, transactional updates; a minute), bulk (marketing, digests; hours). Each has its own Kafka topic keyed by user id, its own router consumer group, its own delivery queues per channel, its own worker fleet with its own provider connections and quota share, and its own SLO and alerts. The critical tier is over-provisioned (it is small) and never autoscaled down below a floor. Bulk producers are additionally admitted at a controlled rate by the expander. Cross-tier sharing is limited to read-only caches (preferences, devices), which are sized for the bulk rate so critical reads never miss. In an incident, bulk is the tier you pause.

Q: Kafka is at-least-once, workers crash, providers time out. How does a user never get the same push twice?

API: SET NX on the caller's idempotency key (24 h) returns the existing notification id on repeats. Router: processing is idempotent by notification id; the routed state is written with a conditional put, and delivery tasks carry a deterministic send key (notification id + channel). Worker: SET NX on the send key before the provider call; if present, skip. Provider timeout: leave the key set, mark state "unknown", and let a reconciler query the provider by client reference (most support it) to settle sent or failed; only a confirmed failure clears the key and re-enqueues. Push uses APNs collapse-id and FCM collapse keys so even a rare duplicate replaces rather than stacks. Redis unavailability fails open for bulk (a rare duplicate is cheaper than a stall) and fails closed for critical channels like SMS (retry later rather than double-charge and double-text).

Q: Who decides that a notification is a push, an email, or both, and how do preferences, opt-outs and cost fit in?

Templates declare a category (transactional, social, marketing), default channels, fallback order, TTL and whether quiet hours apply. The router evaluates in order: legal opt-outs per channel (absolute); category rules (transactional ignores marketing opt-outs and quiet hours); the user's per-category channel preferences; device availability (a push token active in 30 days beats email; no token means email; SMS only where the template allows and the number is verified); cost preference (push over email over SMS when equivalent); frequency caps and digest windows; quiet hours with deferral. The output is the channel set and per-channel rendering. Preferences and device records are cached in Redis with write-through invalidation and a 5-minute TTL; the critical tier reads through on a miss. Every decision is written to the state record with a reason code, so "why did I not get this" is one lookup.

Q: Ten billion lifecycle records a day. What stores them, how are they keyed, and how does the in-app inbox read them?

A Cassandra or Scylla table notifications_by_user with partition key (user_id, month bucket) and clustering key notification_id descending (a time-ordered ULID or Snowflake id), columns for template, category, channels, per-channel status and timestamps, digest group, and reason codes, with a 30-day TTL on the hot table and a nightly job that writes aggregates (per template, per channel, per day) to the analytics warehouse. A second table notifications_by_id maps id → (user_id, bucket) for tracking lookups. The unread count is a Redis counter per user incremented on in-app write and reset on read, with a periodic reconciliation. The inbox API reads the newest page with a cursor of the last notification id and groups digests on read. Support tooling reads by user or by id and gets every state transition with reasons.

Q: How do you send 100 M messages "at 10 a.m. local time" without scanning a table every minute?

A scheduled table partitioned by (due_minute_utc, shard) where shard = hash(user) mod 64, clustered by notification id, with the full request payload. Submitting a local-time send converts to UTC per user timezone at submission and writes one row. A sweeper (leader-elected per shard range) reads the current minute's partitions, publishes each row to the appropriate priority topic, and marks the partition released; the router applies preferences and quiet hours at release time, so late changes are honoured. Cancellation deletes the row or, for a whole campaign, marks the campaign cancelled and the sweeper skips its rows. Clock skew is handled by sweeping a minute a few seconds after it ends, and a missed minute is caught by a catch-up scan of recent unreleased partitions. Digest windows (under a few minutes) use a Redis sorted set instead, because they are short-lived and high-churn.

Deliver push, email, SMS and in-app notifications to hundreds of millions of users: the right channel, once, on time, and never at 3 a.m.

Last updated 2026-09-21. Difficulty: medium. Patterns: fan-out, multi-channel, idempotency, rate-limiting, priority-queues. Reported at Meta and 6 more with Pro.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Pause to answer yourself at the key decisions, and ask the coach anything along the way.

Walk through it Take the mock interview

Functional requirements

Send a notification to a user or a set of users. Internal services call one API with a template id and data; the system decides channels, renders, and delivers. Callers never talk to APNs, FCM or an email provider directly.
Multiple channels. Mobile push (iOS, Android), web push, email, SMS, and in-app inbox. A notification may go to several channels with different content per channel.
User preferences and opt-outs. Per category and per channel, plus legal opt-outs (marketing email unsubscribe, SMS STOP) that override everything. Quiet hours per user timezone.
Priority. Transactional (a login code, a ride arriving) is delivered within seconds and bypasses batching and quiet hours; marketing is low priority, batched, and rate limited.
Deduplication and rate limiting. A user never gets the same notification twice, and never more than N marketing messages a day. Digest similar events ("5 people liked your post").
Scheduling. Send at a given time, in the user's local time, or within a window; cancel a scheduled send.
Delivery tracking. Sent, delivered, opened, clicked, bounced per notification; feed back into templates and preferences (auto-unsubscribe hard bounces).
Out of scope. Composing marketing campaigns (audience selection UI), the in-app inbox UI, and content personalisation models. We deliver what we are given.

Non-functional requirements

Scale (500 M users · 10 B notifications/day). Around 115 k/s average, with marketing blasts adding millions in minutes. The design must absorb bursts without delaying transactional traffic.
Latency for transactional (p99 < 5 s end to end). A one-time code that arrives after the user gave up is a failed login. Priority isolation is the hardest tradeoff: bursts of low-priority traffic must not touch this path.
At-least-once delivery, effectively once (duplicates < 0.01 %). Providers and retries produce duplicates; idempotency keys at every hop keep them from reaching the user.
Availability (99.99 % for accept · providers may be down). The API must accept even when APNs or the email provider is failing; delivery is retried later.
Compliance (opt-outs honoured within seconds). CAN-SPAM, GDPR, TCPA: sending marketing after an unsubscribe is a legal problem, not a bug.
Observability (every notification traceable). Support must answer "did user X get message Y and why not" from a single lookup.
Cost (SMS is 1000× email). Channel selection and dedupe are cost controls as much as UX.

Back-of-envelope estimates

Notifications per second: ~115 k avg · 1 M+ peak. 10 B/day ÷ 86 400 s ≈ 116 k/s. A marketing campaign to 100 M users released over 10 minutes adds ~170 k/s; several at once, plus a breaking-news style event, reach 1 M/s. Design queues for the peak, delivery for a smoothed rate.
Channel split: ~70 % push · 25 % email · 4 % in-app only · 1 % SMS. Typical consumer mix. 1 % SMS of 10 B is 100 M SMS/day; at ~$0.01 each that is $1 M/day, which is why SMS is reserved for auth codes and critical alerts.
Device tokens: ~1.5 B. 500 M users × ~3 devices/tokens (phone, tablet, browser) = 1.5 B tokens at ~200 B each ≈ 300 GB. Fits a sharded key-value store keyed by user; tokens churn constantly (reinstalls), so the store must handle high update rates and invalidation feedback.
Provider calls per second: ~100 k/s to APNs/FCM. 70 % of 116 k/s is ~80 k/s push sends on average, each an HTTP/2 request to APNs or FCM (batched for FCM). Providers accept this from one account; the constraint is per-connection throughput, so hundreds of persistent HTTP/2 connections.
Delivery-state storage: ~2 TB/day. One record per notification with status transitions: ~200 B × 10 B = 2 TB/day. Keep 30 days hot for support lookups (60 TB), then aggregate. A wide-column store keyed by user id with time-ordered notification ids.
Dedupe keys: ~10 B/day · ~250 GB. One idempotency key per (user, notification key) with a 24 h TTL: 10 B × ~25 B = 250 GB live in a Redis cluster, or in the same wide-column store with TTL. Bloom filters are not appropriate: a false positive drops a real notification.
Email throughput: ~30 k/s. 25 % of 116 k/s ≈ 29 k/s. Large ESPs and self-hosted MTAs deliver this with a pool of sending IPs; reputation warm-up means the pool must be built up over weeks, which is an operational constraint worth mentioning.

Components

Calling services: Internal producers: order service, social graph, auth, marketing tools. They send a notification request with a template id, recipient(s), data, priority and an idempotency key. They never know about channels or providers.
Notification API: Validates the request, checks the idempotency key, enriches with the template category and priority, and writes it to the appropriate priority topic. Returns 202 with a notification id in milliseconds; nothing downstream happens synchronously.
Priority topics (Kafka · critical / high / bulk): Separate topics per priority with separate consumer fleets, so a marketing blast in the bulk topic cannot delay a login code in the critical topic. Keyed by user id so one user's notifications are processed in order. Durable buffer for provider outages.
Routing service (preferences + channel selection): For each notification: load user preferences, opt-outs, quiet hours, timezone, and devices; apply category and channel rules; apply per-user rate limits and digest rules; decide the channel set; render templates per channel; enqueue one delivery task per channel. The brain of the system.
Preferences store (KV by user · cached): Per-user preferences by category and channel, legal opt-outs, quiet hours and timezone. Read on every notification, so it is cached aggressively (a few hundred bytes per user) with invalidation on change. Opt-outs are written synchronously and are the last check before any send.
Device registry (user → tokens · endpoints): Push tokens per user and platform, email addresses with verification and bounce state, phone numbers with consent. Updated by clients on login and token refresh, and by provider feedback (invalid token, hard bounce).
Template service (versioned · per channel · localised): Renders content per channel and locale from a template id and data. Versioned so a running campaign is not changed mid-flight. Also holds category metadata (transactional vs marketing, default channels, TTL).
Rate limit + dedupe (Redis · per user counters · idempotency keys): Idempotency keys for the API and for provider sends (24 h TTL), per-user daily counters per category, and the digest buffer that collects similar events for a short window before sending one summary.
Delivery queues (per channel · per priority): One queue per (channel, priority): push-critical, push-bulk, email-bulk, sms-critical and so on. Each is consumed by a worker fleet sized for that provider's throughput and with its own retry and backoff policy. Isolation means an email provider outage backs up only email.
Channel workers (APNs / FCM / SMTP / SMS adapters): Stateless senders per channel with persistent connections to providers, provider-side idempotency where available, circuit breakers, and exponential backoff. They write delivery state and forward provider feedback (invalid tokens, bounces) to the device registry.
Providers (APNs · FCM · SES · Twilio): Third parties that actually reach the device or inbox. Each has its own limits, error semantics and feedback channel. They fail independently and sometimes silently.
Delivery state (Cassandra · key=user id, time-ordered): Every notification's lifecycle: created, routed, sent per channel, delivered, opened, failed with reason. Written by the router and workers, read by the in-app inbox and by support tooling. Time-ordered ids give cursor pagination per user.
In-app inbox: Serves the notification list and unread count for the app from the delivery-state store, and marks read. The in-app channel is "free" and is where everything goes even if push is off.
Scheduler (time-bucketed sends): Holds notifications with a future send time in time-bucketed storage and releases them into the priority topics when due, converting local-time windows into UTC per user timezone. Cancellation deletes the row before release.
Analytics + feedback (stream: opens, clicks, bounces): Consumes delivery and engagement events for dashboards (delivery rate per channel and provider, open rates per template) and feeds back into preferences: auto-reduce frequency for users who never open, auto-unsubscribe hard bounces.

User flows

Send a login code: the critical path. One user, one SMS or push, under five seconds, no matter what else is happening. Every hop is isolated from bulk traffic and idempotent.
1. Auth service calls the notification API with a template, recipient, data, priority and idempotency key.
2. API checks the idempotency key and writes the request to the critical topic. SET NX on the key with a 24-hour TTL; a repeat returns the original notification id without a second enqueue. The critical topic has its own partitions and consumers; nothing in the bulk path shares them. Accept latency is a few milliseconds.
3. Router loads preferences and devices, and applies the transactional rules. Transactional categories ignore quiet hours and marketing opt-outs but still respect legal channel opt-outs (SMS STOP means no SMS, so fall back to push or email). Preferences are cached per user; a miss reads the store. Device lookup returns live push tokens and the verified phone number.
4. Router renders the channel-specific content and enqueues delivery tasks on the critical delivery queues. For a login code the policy is: push if a device was active in the last 30 days, else SMS; both if the template says so. Render per channel and locale. Write the "routed" state with the chosen channels, then enqueue. If the state write fails, the message is retried from Kafka; the idempotency at the worker level prevents double sends.
5. SMS worker sends via the provider with a send-level idempotency key and records the result. Before calling the provider, SET NX the send key; if it exists, another worker already sent (a redelivery after a crash), so skip. Call the provider with a short timeout; a timeout is ambiguous, so the send key stays set and the message is not retried blindly: a reconciliation query to the provider decides. Write "sent" with the provider message id. p99 across all of this is under 2 s; the carrier adds the rest.
6. Provider delivery receipts arrive by webhook and update the state. Delivered, failed (invalid number, carrier rejection) or undelivered. A failure for a critical message triggers the fallback channel if the TTL has not expired. Receipts are the only way to know the carrier actually delivered.
Marketing blast to 100 M users. The bulk path. It must be fast enough to finish in a reasonable window and must never touch the critical path's capacity, preferences, or provider quota.
1. Campaign tool submits one request with an audience reference, not 100 M user ids. The audience is a pre-computed segment stored as a file or a query; the API records the campaign and hands expansion to a bulk expander that streams user ids into the bulk topic at a controlled rate (say 200 k/s). Submitting 100 M individual requests would take the API down and give no way to cancel.
2. Expander streams user ids into the bulk topic, bucketed by the user's timezone window. Users whose local window is open now go straight to the bulk topic; others are written to the scheduler's time buckets and released when their window opens. The rate is capped so downstream fleets see a smooth load; the campaign can be paused or cancelled by stopping the expander and deleting scheduled buckets.
3. Bulk routers apply marketing preferences, opt-outs, frequency caps and quiet hours strictly. Marketing opt-out means drop. Frequency cap: at most N marketing messages per user per day across all campaigns, enforced with a Redis counter keyed by user and day. Quiet hours defer to the next window rather than drop. Around 30 to 50 % of the audience is typically filtered here, which is why filtering happens before rendering and delivery.
4. Bulk delivery queues drain into channel workers at provider-safe rates. Push bulk at 100 k/s across hundreds of HTTP/2 connections; email at the ESP's contracted rate with sending-IP reputation in mind; SMS marketing (rare) at carrier limits. Each fleet autoscales on its own queue depth, and each has a circuit breaker per provider so a failing provider parks its queue instead of burning retries.
5. Engagement and bounce feedback flow back into preferences and the device registry. Hard bounces disable the email address; unsubscribes (one-click list-unsubscribe header) write an opt-out within seconds; invalid push tokens are deleted. Users who never open marketing get their frequency reduced automatically, which improves deliverability for everyone.
APNs is down for 40 minutes. Providers fail. The system must keep accepting, avoid a retry storm, deliver critical messages by another channel, and drain the backlog in order of priority when the provider returns.
1. Push workers see timeouts and 5xx from APNs; the circuit breaker opens. After the failure rate crosses 50 % over 30 s, the breaker opens and workers stop calling APNs, leaving tasks in the queue with a lease returned. Half-open probes every 30 s test recovery. No exponential retry storms against a struggling provider.
2. Push delivery queues grow; the API keeps accepting normally. Accept latency is unchanged because acceptance only writes to Kafka. The push queues hold 40 minutes × 80 k/s ≈ 200 M tasks, which is fine for a durable queue. Bulk push tasks past their TTL will be dropped on drain.
3. Critical notifications whose push attempt fails fall back to SMS or email within their TTL. A critical task that cannot be sent within a fallback deadline (say 20 s) is re-routed by the router to the next channel in the template's fallback list. This is the only case where SMS volume spikes, and it is bounded by the critical rate (a few hundred per second).
4. When APNs recovers, the backlog drains critical-first with per-user coalescing. Separate queues per priority mean critical drains first without any special logic. Bulk tasks older than their TTL are dropped (state: expired). For a user with 15 queued social notifications, the digest rule collapses them into one, so the recovery does not produce a wall of notifications.
5. Alerts fire on breaker state and queue age, not just on error rate. The SLI is time from accept to delivered for critical messages; the leading indicator is the age of the oldest task per delivery queue. Provider status is a dashboard panel with the breaker state per region, so on-call knows in seconds whether the problem is ours or theirs.
Twelve likes in a minute become one notification. Social events arrive in bursts. Without digesting, users get spammed and providers get unnecessary load. The router buffers similar low-priority events per user for a short window and sends a summary.
1. Twelve "like" events for one post arrive at the API within a minute.
2. Router sees a digest key and adds the event to the user's digest buffer instead of sending. The buffer is a Redis list keyed by (user, digest_key) with a 60-second window; the first event sets a timer (a scheduled release) and the rest append. Since the topic is keyed by user id, all of a user's events are handled by one consumer and the buffer sees them in order.
3. When the window closes, the router renders one summary from the buffered events. The template gets the list: "Ana and 11 others liked your post". The window trades latency (one minute) for coherence; for a single event the summary is the plain notification. Windows are longer for lower-priority categories.
4. The summary goes through the normal delivery path with one dedupe key for the whole group. The 12 original notifications are recorded in state as "digested into X" so support can trace them, and the frequency counter increments once, not twelve times.
5. The in-app inbox shows the same grouped item, updated in place as more events arrive. The inbox reads the state store by user with cursor pagination and groups by digest key on read, so an item can keep updating (count 12 → 40) without new pushes.
User unsubscribes while a campaign is mid-flight. Opt-outs are legal requirements with a deadline. The check must happen at the last possible moment, and the write must propagate faster than the queues drain.
1. User clicks unsubscribe in an email; the link writes the opt-out synchronously. The link carries a signed token identifying user and category; the handler writes the opt-out to the preferences store and invalidates the cache before returning the confirmation page. One-click unsubscribe (RFC 8058) via the List-Unsubscribe header does the same from the mail client.
2. A campaign task for this user is already sitting in the email bulk delivery queue. The router made its decision minutes ago. If the worker sends blindly, the user receives marketing after unsubscribing.
3. The email worker re-checks opt-outs immediately before sending. A cheap cached read per send (with the cache invalidated on write, and a short TTL as a backstop) catches opt-outs made after routing. The worker records state "suppressed: opted_out" instead of sending. This last-moment check is the compliance guarantee; the router's earlier check is just an optimisation.
4. Scheduled sends for the user are filtered at release time as well. Anything in the scheduler passes through the router again when released, so a preference change made days before a scheduled send is honoured without scanning the schedule.
5. The opt-out is logged with its source and timestamp for audit. Regulators and ESPs ask for proof. The preferences store keeps an append-only history; the analytics pipeline reports unsubscribe rates per template, which is an early warning that content is off.

Deep dives

Priority isolation

Why separate topics and fleets per priority instead of a priority field in one queue?

The failure this prevents is the most common real incident in notification systems: a marketing campaign of 100 M messages is released, every worker is busy for an hour, and login codes take 40 minutes. Kafka topics are FIFO per partition; a priority field does not let consumers skip ahead. Even a priority queue shares the consumer fleet, the preference cache, and the provider connections.

Isolation must be end to end: separate queues, separate consumers, separate provider connection pools and quota, and separate rate limits. Then the bulk path can be saturated and the critical path never notices.

Separate topics, consumer fleets, delivery queues and provider pools per priority tier chosen
One Kafka topic with a priority field rejected
A priority queue (RabbitMQ priorities, Redis sorted set) situational: a small system where a single queue suffices and provider capacity is not the constraint
Rate-limit bulk producers so they never saturate situational: as a complement: bulk producers are rate limited anyway, but isolation is the guarantee

The answer: Three tiers: critical (auth, safety, payments; seconds), high (social, transactional updates; a minute), bulk (marketing, digests; hours). Each has its own Kafka topic keyed by user id, its own router consumer group, its own delivery queues per channel, its own worker fleet with its own provider connections and quota share, and its own SLO and alerts. The critical tier is over-provisioned (it is small) and never autoscaled down below a floor. Bulk producers are additionally admitted at a controlled rate by the expander. Cross-tier sharing is limited to read-only caches (preferences, devices), which are sized for the bulk rate so critical reads never miss. In an incident, bulk is the tier you pause.

Bulk is backed up by two hours and someone submits a "high" campaign. Where does it go?

The tier is set by the template category, not by the caller; a marketing template cannot claim high. If the product genuinely needs a faster marketing lane, add a fourth tier with its own capacity rather than letting callers pick. Priority you can request is priority everyone requests.

Critical volume is 1 % of traffic. Is a whole fleet for it wasteful?

It is a few instances and a few partitions; the cost is negligible next to the incident it prevents. Keep the critical fleet's minimum size at 2× its peak so a deploy or a zone loss does not degrade it.

How do you keep per-user ordering if tiers are separate?

You do not, across tiers, and that is fine: a login code and a marketing email have no ordering relationship. Within a tier, keying by user id gives order. If two notifications in different tiers must be ordered (rare), put them in the same tier.

What is the SLO per tier and how do you measure it?

Critical: 99.9 % delivered-to-provider within 5 s of accept. High: within 60 s. Bulk: campaign completes within its window. Measured from the state store timestamps (accepted → sent) and from provider receipts for end-to-end. Alert on burn rate for critical; on queue age for the others.

Effectively-once delivery

Kafka is at-least-once, workers crash, providers time out. How does a user never get the same push twice?

Duplicates enter at three points: the caller retries the API, Kafka redelivers to a router after a crash, and a worker retries a provider call that actually succeeded (a timeout is ambiguous). A duplicate login code is annoying; a duplicate "your order shipped" push is a support ticket; a duplicate marketing email is an unsubscribe.

There is no end-to-end exactly-once across third-party providers. What exists is an idempotency key at each hop, checked atomically before the side effect, and a policy for the ambiguous case.

Idempotency keys at API, router and per-channel send, with a reconciliation path for ambiguous provider results chosen
Kafka transactions end to end rejected: useful between the router and delivery queues if both are Kafka, but does not solve the provider hop
At-most-once: never retry rejected: for some bulk marketing, dropping on failure is acceptable and cheaper than reconciliation
Rely on provider-side dedupe (APNs collapse id, email Message-ID) situational: always set them as defence in depth; do not rely on them

The answer: API: SET NX on the caller's idempotency key (24 h) returns the existing notification id on repeats. Router: processing is idempotent by notification id; the routed state is written with a conditional put, and delivery tasks carry a deterministic send key (notification id + channel). Worker: SET NX on the send key before the provider call; if present, skip. Provider timeout: leave the key set, mark state "unknown", and let a reconciler query the provider by client reference (most support it) to settle sent or failed; only a confirmed failure clears the key and re-enqueues. Push uses APNs collapse-id and FCM collapse keys so even a rare duplicate replaces rather than stacks. Redis unavailability fails open for bulk (a rare duplicate is cheaper than a stall) and fails closed for critical channels like SMS (retry later rather than double-charge and double-text).

The worker sets the send key, then crashes before calling the provider. Now the message is never sent.

The key has a short "in-progress" TTL (say 60 s) distinct from the 24-hour "sent" state: SET NX with EX 60, then after a successful send, extend to 24 h. If the worker dies, the key expires and the redelivered task sends. The window of a duplicate is a worker that took over 60 s to call the provider, which timeouts prevent.

Redis loses the dedupe keys (failover, memory pressure). What happens?

Some in-flight redeliveries can duplicate. Bound it: keys are also written to the delivery state store (the durable record of "sent"), and workers check state on a Redis miss for critical channels. Run Redis with replication and disable eviction on the dedupe keyspace; a full Redis is an alert, not a silent drop.

Caller sends the same logical event with different idempotency keys.

Then it is two notifications, by contract. Protect the user anyway with a content-level dedupe: hash of (user, template, key fields) with a short TTL (e.g. 10 minutes) in the router, which collapses accidental re-sends from buggy producers. Log and alert on the collapse rate per producer.

How do you test this?

Chaos in staging: kill workers mid-send, inject provider timeouts and 5xx, replay Kafka offsets, and assert zero duplicates at a fake provider that records every call. Run the same fault injection in production against a canary user pool with a fake provider endpoint. Duplicate rate per channel is a tracked metric from provider receipts.

Channel selection and preferences

Who decides that a notification is a push, an email, or both, and how do preferences, opt-outs and cost fit in?

Callers should not decide channels: they do not know the user's devices, preferences, or that SMS costs a cent. Centralising the decision makes preferences consistent, compliance enforceable, and cost controllable. The decision also has to be fast (it runs 100 k times a second) and correct under stale caches, because a wrong decision is either a missed critical message or an illegal one.

The rule set is layered: legal opt-outs first, then category defaults from the template, then user preferences, then device availability and cost, then frequency and quiet hours.

Central router with layered policy: legal → category → user prefs → device → cost → frequency chosen
Caller chooses channels rejected: allow a caller to restrict channels (never widen) for cases like "SMS only for this code"
Rules engine evaluated at send time by workers rejected: workers do re-check opt-outs at the last moment, but the decision lives in the router
ML model chooses the channel per user situational: for marketing and social tiers, after the rule layers, once you have engagement data

The answer: Templates declare a category (transactional, social, marketing), default channels, fallback order, TTL and whether quiet hours apply. The router evaluates in order: legal opt-outs per channel (absolute); category rules (transactional ignores marketing opt-outs and quiet hours); the user's per-category channel preferences; device availability (a push token active in 30 days beats email; no token means email; SMS only where the template allows and the number is verified); cost preference (push over email over SMS when equivalent); frequency caps and digest windows; quiet hours with deferral. The output is the channel set and per-channel rendering. Preferences and device records are cached in Redis with write-through invalidation and a 5-minute TTL; the critical tier reads through on a miss. Every decision is written to the state record with a reason code, so "why did I not get this" is one lookup.

The preference cache is stale by up to 5 minutes. Is that acceptable?

For preferences, yes; for legal opt-outs, no. Opt-outs invalidate the cache synchronously on write and are re-checked by the worker at send time from a source that was invalidated, so the effective window is seconds. Preferences that only affect channel choice can lag.

A user has three devices. Push to all, or one?

All active tokens for informational pushes (APNs collapse id keeps the display tidy), because you do not know which device is in hand. For high-cost or interactive actions (approve this login), the most recently active device first with a short fallback to the others. Delete tokens the provider reports invalid, and expire tokens unseen for 90 days.

How do you handle a user with no channels available?

The in-app inbox always receives it, so nothing is lost. Record state "no_channel" with the reason so the product can prompt for a verified email or push permission. For critical auth flows, the caller gets a synchronous answer (from the state or a lookup endpoint) that no out-of-band channel exists, and can offer a different factor.

Where does localisation happen?

In the template service at render time, using the user's locale from the preferences record and the template's translated variants. Rendering is per channel because email needs HTML and a subject, push needs a 100-character body, SMS needs plain text with the sender id. Templates are versioned; a campaign pins a version at submission.

Delivery state and the inbox

Ten billion lifecycle records a day. What stores them, how are they keyed, and how does the in-app inbox read them?

Every notification transitions through several states, written by different components at different times, and must be readable per user (inbox, support) and per notification (tracking). Volume is 2 TB a day; retention is 30 days hot plus aggregates. Writes are append-heavy; reads are recent-first per user with pagination and an unread count.

This is the same shape as a message history store: partition by user, cluster by time-ordered id, and keep aggregates (unread count) as separate small counters rather than scanning.

Wide-column store partitioned by user id, clustered by time-ordered notification id, with TTL; unread counter in Redis chosen
Relational table per shard situational: a much smaller system, or for the aggregated reporting tables
Only an event log (Kafka) and derived analytics rejected: the log exists anyway as the feed into analytics; it is not the read store
Store the inbox in the app (push everything, no server inbox) rejected: never at this scale

The answer: A Cassandra or Scylla table notifications_by_user with partition key (user_id, month bucket) and clustering key notification_id descending (a time-ordered ULID or Snowflake id), columns for template, category, channels, per-channel status and timestamps, digest group, and reason codes, with a 30-day TTL on the hot table and a nightly job that writes aggregates (per template, per channel, per day) to the analytics warehouse. A second table notifications_by_id maps id → (user_id, bucket) for tracking lookups. The unread count is a Redis counter per user incremented on in-app write and reset on read, with a periodic reconciliation. The inbox API reads the newest page with a cursor of the last notification id and groups digests on read. Support tooling reads by user or by id and gets every state transition with reasons.

The router writes "routed", then two workers write "sent" for two channels concurrently. Any conflict?

Each writes different columns (push_status, email_status) of the same row; Cassandra merges by column with timestamps, so there is no lost update. Timestamps come from the writers, so keep clocks synced; or use lightweight transactions for the rare state that must be exclusive (such as the fallback decision).

A user has 50 000 notifications in a month (a heavy social user). Is the partition too big?

At 200 bytes each that is 10 MB, within Cassandra's comfortable partition size (under 100 MB). The month bucket bounds it. For outliers, bucket by week. Reads always page from the newest, so partition size mostly affects compaction, not latency.

How does the unread count stay correct?

It does not need to be exactly correct; it needs to be plausible and to hit zero when the inbox is read. Increment on write of an in-app notification, set to zero on open, and reconcile from a count query occasionally or when a client reports a mismatch. Badges are the one place where a stale count is very visible, so the reset path must be reliable.

Retention: legal wants 7 years of "we sent this" for some categories.

Keep the hot store at 30 days and archive the events (not the rows) to object storage in partitioned Parquet by day and category; 7 years of 10 B events a day is about 5 PB compressed, which is affordable in cold storage and queryable with a batch engine when a regulator asks.

Scheduled and local-time sends

How do you send 100 M messages "at 10 a.m. local time" without scanning a table every minute?

Scheduled sends are a large share of marketing and reminders. Naively, a scheduler polls a table for rows with send_at ≤ now, which at hundreds of millions of rows is a full index scan every tick and a hot spot when many rows share a time (10 a.m. in a big timezone). Local time means a single campaign has dozens of distinct release times.

The efficient structure is time-bucketed storage: partition scheduled items by their due minute, so releasing a minute is reading one partition, and cancellation is a delete in that partition.

Time-bucketed partitions (per minute) in a wide-column store, released by a leader-elected sweeper into the priority topics chosen
Redis sorted set by due time situational: short horizons (digest windows, minutes-away releases); we use it for digests
Kafka with per-delay topics rejected: fixed short delays (retry in 30 s)
Cron-style scanning of a send_at index rejected: thousands of rows, not billions

The answer: A scheduled table partitioned by (due_minute_utc, shard) where shard = hash(user) mod 64, clustered by notification id, with the full request payload. Submitting a local-time send converts to UTC per user timezone at submission and writes one row. A sweeper (leader-elected per shard range) reads the current minute's partitions, publishes each row to the appropriate priority topic, and marks the partition released; the router applies preferences and quiet hours at release time, so late changes are honoured. Cancellation deletes the row or, for a whole campaign, marks the campaign cancelled and the sweeper skips its rows. Clock skew is handled by sweeping a minute a few seconds after it ends, and a missed minute is caught by a catch-up scan of recent unreleased partitions. Digest windows (under a few minutes) use a Redis sorted set instead, because they are short-lived and high-churn.

The sweeper crashes after publishing half of a minute's partition. What happens on restart?

The partition is not marked released, so the new sweeper republishes it; the duplicates are absorbed by the idempotency key at the API/router hop, which is exactly what it is for. Marking per row instead of per partition avoids even that, at the cost of 100 M extra writes; per partition with idempotency is the better trade.

A user changes timezone after a send was scheduled in local time.

The row stores the UTC time computed at submission; the change is not reflected unless you re-derive. For campaigns, store the local target alongside and have the router at release time check whether the user's current local time is within the window; if not, re-schedule once. Rare enough that this approximation is fine.

Everyone in the largest timezone is due at 10:00. How hot is that minute?

Tens of millions of rows in 64 shard partitions, a few hundred thousand each; readable in parallel by the sweeper fleet in a minute or two. Smooth it deliberately: the expander jitters the due minute within the send window (10:00 to 10:20) so release, routing and providers see a steady rate rather than a spike.

How do you support "remind me in 3 days unless I complete the task"?

Schedule it with a cancellation key (user, task); the task-completion event calls DELETE on that key, which maps to the row through a small index. Alternatively the router re-checks a predicate at release ("task still incomplete?") via a callback URL supplied by the caller, which also handles cases the caller forgot to cancel.