Design Google Calendar

• Create, edit and delete events: Timed events carry a time zone. All-day events are dates with no zone, so a birthday on 5 March is 5 March everywhere.
• Recurring events with exceptions: "Every weekday", "monthly on the last Friday until June". Edits apply to this event, this and following, or all events, and each of those is a different operation.
• Invite guests and track responses: Inside the organisation and outside it by email. The organiser owns the event; each guest has a response (needs action, yes, no, maybe) and their own reminders.
• Free/busy and find a time: See when colleagues and rooms are busy, without titles, and get suggested slots that fit everyone’s working hours in their own zones.
• Reminders: A default lead time per user (say 10 minutes), overridable per event, as push or email. They must follow the event when it moves and never fire for one that was cancelled.
• Day, week and month views on every device: Web, mobile and desktop, readable offline on mobile, with a change on one device or by the organiser showing up everywhere within seconds.
• Works with other calendars: External guests on Outlook or iCloud get standard iCalendar invitations by email and can reply; third-party apps can sync over CalDAV.
• Out of scope: Video conferencing, public booking pages, tasks, search, and the internals of push and email delivery: we call the existing notification system and reuse the job scheduler for timers.

• Scale (500 M users · ~4 k writes/s · ~120 k reads/s): Reads run 20 to 50 times writes, but most are answered from the copy each device keeps. What reaches the server is incremental syncs, web views and free/busy.
• Time correctness (right wall-clock time, always): A 9:00 meeting stays at 9:00 in its own zone across daylight saving changes and after governments change zone rules. This is the requirement that drives the hardest trade-off: store rules and local times, and pay for expanding and re-expanding them, instead of simple UTC rows.
• Freshness across devices and guests (p95 < 5 s): An edit reaches the editor’s other devices and every guest’s calendar within seconds, and the editor sees their own write at once. It drives asynchronous fan-out plus push hints.
• Read latency (week view and free/busy p99 < 300 ms): Web week views, and free/busy for up to about 50 calendars. Each is one partition read per calendar, done in parallel.
• Reminders on time (p99 within 30 s of due): Including the bursts at :50 and :20 when tens of millions of reminders fall due in the same minute, and never for a meeting that moved or was declined.
• Availability (99.99 %): People join meetings from their calendar at 8:59. Apps keep working offline from their local copy, so a server problem costs freshness before it costs reads.
• Privacy (free/busy shows blocks, not titles): Colleagues see busy intervals only unless the owner shares details, private events show as busy even to people who can see details, and guests of a large event need not see each other.

• Event writes per second: ~4 k avg · ~12 k peak. 500 M users × 5 events a week = 2.5 B/week; ÷ 604 800 s ≈ 4.1 k/s. Weekday mornings, when Europe and the Americas overlap, run about 3× that, ~12 k/s. Each is a single-partition transaction, so writes are not the hard part.
• Calendar writes with guest copies: ~16 k/s. An average event sits on 4 calendars (the organiser plus 3 guests), so 4.1 k × 4 ≈ 16 k/s calendar-entry writes, each with its change-log row. Copies multiply writes by 4 to make every view a single-partition read, which pays when reads are 30× writes.
• View reads per second: ~120 k avg · ~360 k peak. At 30 reads per write, 4 k × 30 ≈ 120 k/s. Most never reach the server as reads: apps keep a copy and ask only what changed, and most of those syncs return nothing. The server’s read load is syncs, web week views and free/busy.
• Event storage per year: ~520 TB. 2.5 B events a week × 52 ≈ 130 B events a year. At ~1 KB per calendar entry on 4 calendars, 130 B × 4 KB ≈ 520 TB/year before replication, ~1.5 PB with three replicas. Partitioned by calendar, with old years on cheaper storage.
• Instance index (12-month horizon): ~50 TB. One-off meetings are booked only weeks ahead, so the horizon is mostly series instances: about 1 000 per calendar (a dozen weekly or daily series plus the next few weeks of one-offs). 500 M × 1 000 = 500 B rows × ~100 B ≈ 50 TB. Materialising "forever" has no bound at all, which is why the horizon exists.
• One all-hands invitation: 5 k copies · 260 k instances. 5 000 guests on a weekly series: 5 000 copies plus 5 000 × 52 = 260 k instance rows and a timer per guest. At ~20 k writes/s on a separate large-event lane that is ~13 s of work after the organiser’s save has returned. This case, not the average 4-calendar meeting, sets the fan-out design.
• Reminders and their burst: ~12 k/s avg · ~670 k/s at :50. 500 M calendars × ~2 reminded instances a day = 1 B/day ≈ 12 k/s. The busiest hour holds ~8 % (80 M) and half of those meetings start on the hour, so 40 M reminders fall due in one minute; spread across that minute it is ~670 k/s, about 55× the average. The burst, not the average, sizes this path.
• One find-a-time query: ~70 bucket reads. Five guests over two weeks: 5 calendars × 14 day buckets = 70 parallel reads of ~6 busy intervals each, so ~420 intervals to merge. The merge takes microseconds; the parallel reads set the latency.

• Calendar apps (web · iOS · Android · desktop): Mobile and desktop apps keep a local copy of each calendar, expand recurring series themselves with the device’s time zone data, and schedule popup alarms locally, so most of the 20 to 50 reads per write never reach the server. They sync with an opaque per-calendar token and treat a push hint only as a prompt to sync. Browser tabs keep no full copy and ask the server for an expanded window.
• API gateway (auth · routing · rate limits): Authenticates users and third-party apps (OAuth), routes to the event and free/busy endpoints, and rate-limits per user and per app. Stateless. Calendar permissions are enforced behind it, by the services that own the data, because they depend on each calendar’s sharing settings.
• Event service (writes · sync · window reads): Owns every write: create, edit with a scope of this, this and following, or all, respond, delete. Each write is one transaction in one calendar’s partition that updates the entry, appends to that calendar’s change log and adds an outbox row, so it never waits on guests. It also answers incremental sync and expands series at read time for browser views, applying sharing rules (private events read as busy).
• View cache (Redis · key includes head seq): Expanded instances per (calendar, month) for browser views, keyed by the calendar’s current change sequence. A write bumps the sequence, so an old entry is simply never asked for again and no invalidation is needed. Absorbs week-flipping and popular shared calendars such as company holidays.
• Other providers (Outlook · iCloud · any mailbox): Exchange, iCloud, Fastmail or a plain mailbox. They keep their own copy of the event and speak only standards: iCalendar objects by email, or CalDAV. Delivery is slow, and their clients interpret recurrence and time zones their own way, so we never assume an update was applied.
• Free/busy service (interval merge · find a time): Answers "when are these calendars busy" and "find a slot for these people". Reads each calendar’s day buckets in parallel, keeps instances that count as busy (opaque, not declined, not cancelled) and returns blocks without titles unless the caller may see details. Finding a slot is a sweep over the merged intervals plus each person’s working hours, computed in their own zone.
• Instance index (wide-column · (calendar, day)): Materialised instances from 30 days back to 12 months ahead, partitioned by (calendar, day) and sorted by start: UTC start and end, event and instance id, busy or free, the owner’s response and the series version. Derived, so any part of it can be deleted and rebuilt from the event store. A multi-day event is written into every day it covers, so a day read is complete without looking backwards.
• Event store (sharded SQL / Spanner · by calendar): Partitioned by calendar id: entries (the organiser’s master or a guest’s copy), overrides keyed by RECURRENCE-ID, the master’s attendee table, the calendar’s change log and the outbox. Single-partition transactions give each calendar a gap-free change sequence, which is what makes sync tokens simple. Start times are stored as local wall-clock time plus an IANA zone; UTC is derived.
• Interop gateway (iCalendar · iTIP/iMIP · CalDAV): The boundary with every other calendar system. Sends invitations, updates and cancellations to external guests as iTIP messages over email (iMIP, RFC 6047) and accepts their REPLY messages, checking the sender and SEQUENCE before changing that one attendee’s response. Also serves CalDAV (RFC 4791) and .ics feeds, and maps Windows zone names from Exchange to IANA ids.
• Notification system (existing · push · email): The company’s existing notification platform: priority tiers, preferences, quiet hours, idempotency keys and connections to APNs, FCM and email providers. Calendar is one more caller, sending reminders at high priority with a TTL and data-only pushes for change hints. Nothing about delivery is rebuilt here.
• Reminder timers (job scheduler · due-time index): Durable one-shot timers for reminders due in the next 7 days, keyed by (calendar, instance, lead time), stored and fired by the job scheduler: partitioned by key hash, indexed by due time, owned by leased dispatchers. Each timer carries the instance version, and one whose instance moved, was cancelled or was declined is dropped at fire time. Due times are jittered by ±30 s so the :50 burst spreads over a minute.
• Instance expander (RRULE engine + tzdata): Expands each changed series into instances for the horizon: iterate the RRULE in the event’s own zone, convert each local start to UTC with current tzdata, drop EXDATEs and apply overrides. Runs a daily roll per calendar that extends the horizon and creates timers entering the 7-day window, and re-expands future instances in a zone when a tzdata release changes its rules.
• Change stream (Kafka · key = calendar_id): Committed calendar changes, relayed from the outbox so a change is published if and only if it committed. Keyed by calendar id, which keeps each calendar’s changes, and therefore each master’s versions, in order. Consumers: invite fan-out, the instance expander and change hints.
• Invite fan-out (async · batched by shard): Turns a change to an organiser’s master into guest copies with one conditional upsert per guest calendar, applied only if the copy’s version is older, so replays and reordering are harmless. Large guest lists run on a separate lane in batches grouped by destination shard, checkpointing after each batch. Hands external guests to the interop gateway and carries responses back to the master’s attendee table.
• Change hints (WebSocket · data-only push): Tells devices that a calendar changed, never what changed. Keeps a registry of which devices watch which calendars, coalesces bursts to one hint per calendar every two seconds, and sends it over an open WebSocket or as a collapsible data-only push. A lost hint costs latency, not correctness, because the next sync uses the token.

1. Schedule a weekly meeting with guests: The core write. The organiser picks a slot from free/busy and saves one series row; guest copies, instances, reminders and every device follow from the change stream within seconds.
   1. The organiser’s app asks for the busy times of five guests over the next two weeks.
   2. The organiser saves a weekly series at Monday 9:00 New York time.
   3. The change reaches the change stream and fan-out writes a copy into each internal guest’s calendar.
   4. The external guest gets an email invitation that any calendar app can read.
   5. The expander materialises the series into each calendar’s instances for the next 12 months.
   6. Reminder timers are set for instances whose reminders fall due in the next 7 days.
   7. Every guest’s devices get a hint and sync the new event.
   8. The external guest accepts in Outlook and the reply updates their response on the master.
2. A guest’s phone and laptop catch up: The main read path. Reads outnumber writes 20 to 50 times, so apps keep a local copy and ask only for what changed; a lost hint or an expired token makes catching up slower, never wrong.
   1. The phone calls incremental sync with its token for each calendar it shows.
   2. The app expands the series locally for the weeks on screen and schedules its own alarm.
   3. The laptop opens next week in a browser tab, and the server expands the window.
   4. A hint is lost; the next app open or background sync picks up the change anyway.
   5. A laptop that slept for six weeks presents a token older than the log keeps, and does a full sync.
   6. A third-party CalDAV client syncs the same calendar with a sync-collection report.
3. Invite 5,000 people to a weekly all-hands: The scale case. One save becomes 5,000 calendar copies, 260,000 instances and thousands of replies, without making the organiser wait or delaying everyone else’s invitations.
   1. The organiser invites a 5,000-person group to a weekly all-hands.
   2. Fan-out writes the copies on a separate lane for large events, batched by destination shard.
   3. A fan-out worker dies after 3,000 copies, and another resumes from the checkpoint.
   4. The expander writes 52 instances for each guest, about 260,000 rows, at a capped rate.
   5. Thousands of people reply within an hour; each reply changes their copy and one row on the master.
   6. The organiser moves one week’s all-hands an hour later, and that one instance changes for 5,000 people.
4. Move this and all following events: The edit most designs get wrong. Changing a series from one week onwards splits it into two series, and overrides, instances, timers, guests’ copies and external calendars all have to follow.
   1. The organiser moves the weekly meeting from Monday 9:00 to Tuesday 10:00, starting with the 11 January occurrence.
   2. In one transaction, the old series ends just before that occurrence and a new series starts on 12 January.
   3. Fan-out updates both series in every guest’s calendar and sends external guests two iTIP messages.
   4. The expander deletes the old series’ instances from 11 January on and writes the new series’ instances.
   5. Timers for removed instances are deleted and new ones created; a stale timer that fires anyway is dropped by its version check.
   6. Every guest’s devices sync two changed series and redraw.
5. Reminders for the 09:00 wave: The background path and the burst. Meetings start on the hour, so tens of millions of reminders fall due in the same minute, and some of their meetings moved or were declined after the timer was set.
   1. A daily roll per calendar creates timers whose due time enters the next 7 days.
   2. Just before 08:50, the dispatchers that hold leases load the timers due in the next minute.
   3. A timer for a meeting that moved after it was set fires, and its version check drops it.
   4. Valid reminders go to the notification system, spread across the minute, each with an idempotency key.
   5. The notification system sends email and pushes, skipping devices that ring locally.
   6. Lateness is measured per minute and per partition.

Ask: Why not just write every occurrence of a recurring event as its own row?

Good answers name: Store the rule (RRULE, local DTSTART + TZID, EXDATE, overrides), expand per window, materialise a bounded horizon, Materialise every occurrence as an independent row, Store the rule and expand on every read, with nothing materialised.

Our pick: A series is one entry: DTSTART as local wall-clock time with an IANA TZID, a DURATION, the RRULE (plus any RDATEs), EXDATEs for cancelled occurrences, and override rows keyed by RECURRENCE-ID, the original start of the occurrence they replace. Expansion iterates the rule in the event’s zone, converts each local start to UTC with current tzdata, removes EXDATEs and swaps in overrides; RFC 5545 already says that generated dates which do not exist, such as 30 February, are skipped. Views expand the visible window at read time, while the expander materialises instances from 30 days back to 12 months ahead into the instance index for free/busy and reminders, rolls it forward daily, and anything beyond it is expanded on the fly. "This" writes an override, "all" updates the series and re-keys overrides by date, and "this and following" ends the series with an UNTIL just before the chosen occurrence and starts a new series there. Google’s public Calendar API exposes the same shape: recurring events carry RRULE lines, and each instance carries recurringEventId and originalStartTime.

1. A daily standup has no end date. How many rows does it cost?
   One series row, plus an override or EXDATE for each day that was changed or cancelled. In the instance index it is about 260 weekday rows per guest for the 12-month horizon, and the daily roll adds one as the horizon moves. A view of 2030 expands the rule for that window on the fly, so "no end" never means unbounded storage.
2. 14 March has an override with a different room. Then the organiser moves the whole series from 9:00 to 10:00. What happens to it?
   Its RECURRENCE-ID was 9:00 on 14 March, which the new rule no longer generates, so a naive implementation orphans it. We re-key overrides by occurrence date: it now replaces the 10:00 instance on 14 March and keeps its different room. An override that had moved its own time keeps that explicit time, and cancelled dates stay cancelled. The UI tells the organiser which exceptions will be kept before they confirm.
3. Why split the series instead of using RANGE=THISANDFUTURE?
   RFC 5545 does let an override apply to "this and all later instances" with RANGE=THISANDFUTURE on its RECURRENCE-ID, but client support is uneven, and each use piles more meaning onto one series. Ending the old series with UNTIL and starting a new one is understood by every client, keeps each series simple, and leaves past instances exactly as they were. A split_from link lets the UI still offer "edit all events" across both halves.
4. What does "monthly on the 31st" do in a 30-day month?
   RFC 5545 says a generated date that does not exist is skipped and not counted, so the rule fires only in the seven months with 31 days. If the user meant the last day of the month, the rule is BYMONTHDAY=-1, and the editor should offer that choice explicitly. COUNT-limited rules count only real occurrences, which is why skipping has to happen inside the expander, not after it.
5. How would you know the instance index has drifted from the rules?
   A verifier samples calendars continuously, re-expands their series from the event store and diffs the result against the instance index, alerting on any mismatch beyond the seconds of expander lag. Every instance row carries the series version it was derived from, so a stale row is detectable without re-expanding. Because the index is derived, the fix for a bad shard is to rebuild it, not to patch rows.

Ask: Everything else in the system is stored in UTC. Why not events?

Good answers name: Local wall-clock time + IANA zone id as the truth, UTC derived and recomputed, UTC instants only, Local time with a fixed UTC offset (-05:00), Floating times and dates with no zone.

Our pick: Timed events store their start as a local date-time plus an IANA zone id and a duration; all-day events store a date and no zone. Single events and the instance index also carry the derived UTC start and end, which is what queries, free/busy and timers use. When a tzdata release changes a zone, a re-expansion job finds every series and future one-off event in that zone, recomputes UTC, rewrites instances, moves reminder timers and bumps each calendar’s change sequence so devices resync. Ambiguous local times follow RFC 5545: a time in the spring-forward gap uses the offset from before the gap (02:30 becomes 03:30), and a time that happens twice means the first one. The viewer’s zone matters only for display and working hours, never for storage.

1. The weekly sync is 9:00 New York. What does a guest in London see in March?
   14:00 most of the year, but 13:00 for two or three weeks in March and for one week in late October or early November. The US changes clocks on the second Sunday of March and the first Sunday of November, the UK on the last Sundays of March and October. That is correct, because the meeting is anchored to New York; if it confuses people, the fix is a note in the UI that the time differs this week, not a change to storage.
2. A new tzdata release changes a zone’s rules from next month. Walk me through it.
   The release rolls out to every service and app through the normal release process, and the expander runs a job for the changed zones only: find series and future single events with that TZID, recompute UTC for instances after the change date, rewrite those instance rows and timers, and append a change to each affected calendar so devices resync. A meeting between two zones moves for whoever is not in the changed zone, which is the right outcome. Apps that expand locally with older tzdata could disagree, so sync reports the device’s tzdata version and the server sends pre-expanded instances when they differ.
3. What happens to an event at 02:30 on the night the clocks spring forward, and one at 01:30 on the night they fall back?
   RFC 5545 defines both. In New York 02:30 does not exist on the spring-forward night, so it takes the offset from before the gap and lands at 03:30 EDT; 01:30 happens twice on the fall-back night and means the first one, 01:30 EDT. Because instances and timers are UTC instants after expansion, each occurrence gets exactly one instant and one reminder, so nothing fires twice. Both nights are tested explicitly with an injected clock.
4. The organiser moves from New York to London. Should their weekly 1:1 move?
   No. The zone belongs to the event, not the person, so the 1:1 stays at 9:00 New York and the organiser now sees it at 14:00. If they want it at 9:00 London from now on, that is a "this and following" edit with a new TZID, which the app can offer when it notices the device zone changed. Working hours and the default zone for new events do follow the user.
5. An invitation arrives from Exchange with TZID "Eastern Standard Time". What do you store?
   Exchange uses Windows zone names and sends a VTIMEZONE with the rules. The interop gateway maps Windows names to IANA ids with the CLDR windowsZones table, so "Eastern Standard Time" becomes America/New_York and later rule changes apply automatically. If a name cannot be mapped, we keep the sender’s VTIMEZONE rules as a custom zone for that event, which RFC 5545 allows, and flag it so a later update can replace it.

Ask: Why copy a meeting into every guest’s calendar instead of storing it once and pointing at it?

Good answers name: Organiser-owned master, a copy in each guest’s calendar, fanned out asynchronously, One shared row that guests reference, Synchronous fan-out inside the organiser’s request.

Our pick: The organiser’s entry is the master: it holds the event fields, the attendee table and the version. Saving it is one transaction in the organiser’s partition with an outbox row; fan-out then upserts a copy into each internal guest’s calendar, conditional on the copy’s version being older, and appends to that calendar’s change log, so guests converge within seconds and replays are harmless. Guests change only their own state on a copy (response, reminders, colour); anything else is an edit request to the master. Responses travel the other way: the guest’s copy changes at once and the response reaches the master’s attendee row through the outbox. Invitations above a few hundred guests run on a separate fan-out lane in batches grouped by destination shard, and a company-wide event for 300,000 people is a shared calendar that people subscribe to, read by reference like a holiday calendar. Exchange and CalDAV scheduling (RFC 6638) also give each attendee their own copy, which is why external guests fit the same model.

1. The fan-out worker dies after writing 3,000 of 5,000 copies. What now?
   The task is redelivered from the change stream with its checkpoint, and a new worker resumes from the last committed batch. Writes are conditional upserts on (calendar, event) with the master version, so the repeated batch changes nothing. The organiser saw "sent" the moment the master committed; the first 3,000 guests are already correct and the rest follow a few seconds late.
2. The organiser edits twice within a second. Can a guest end up with the older version?
   Not durably. Changes are keyed by the organiser’s calendar in the change stream, so one event’s versions are consumed in order, and every copy write is conditional on the version being newer. If a retry delivers version 2 after version 3, the condition fails and the copy stays at 3. Clients compare versions too, so a late hint cannot roll a view back.
3. How does the organiser see "3,100 yes, 400 no" without reading 5,000 rows?
   Responses update rows in the attendee table keyed by (event, attendee), so there is no single hot row, and a consumer keeps per-event counters from the same response changes. The organiser’s view shows the counters and pages through attendees on demand. Above a small guest count we also stop emailing the organiser for each reply, because 5,000 reply emails is its own incident.
4. Why not let guests edit their copy freely, say to fix a typo?
   Because there would be two truths, and the next organiser edit would either overwrite the guest’s change or have to merge with it. When the organiser allows guests to modify the event, the guest’s edit goes to the master with the version they saw, is applied with the usual check and fans out to everyone. Purely personal changes such as reminders, colour or private notes stay on the copy and fan-out never overwrites them.
5. A guest is removed from the meeting. What has to happen?
   The master drops the attendee, and fan-out deletes their copy and leaves a tombstone in their change log so their devices remove it. The expander deletes their instances and the timers go with them. An external guest gets an iTIP CANCEL addressed to them by email, since we cannot reach into their calendar.

Ask: Find a 30-minute slot next week for eight people in three time zones, without telling anyone what is on the others’ calendars. How?

Good answers name: Busy intervals from the instance index per (calendar, day), merged with a sweep at query time, Expand each attendee’s rules at query time, Per-day slot bitmaps, one bit per 5 minutes, An interval tree per calendar.

Our pick: The free/busy service checks the caller’s permission on each calendar, reads the day buckets for the window from the instance index in parallel, and keeps instances that count as busy: opaque, not declined, not cancelled, with tentative counted as busy. Multi-day events were written into every day they cover, so no query has to look backwards for a long event that started earlier. Callers with free/busy permission get blocks with no title, location or guests, and private events appear only as busy even to people who can see details. Finding a slot merges everyone’s busy lists and their working hours (computed per person for each local date in their own zone, then converted to UTC) with a sweep over sorted endpoints, then ranks free gaps of at least the meeting length: inside everyone’s working hours first, then fewest optional guests in conflict, then earliest, preferring :00 and :30 starts. With no perfect slot it returns the least-conflicted ones and names who is busy. A query is capped at about 50 calendars, and rooms are simply more calendars with capacity and features.

1. Two people book the last free half hour in the same room a second apart. Who wins?
   A room is a calendar that accepts invitations automatically, and acceptance is a check-and-insert inside the room calendar’s own partition: no accepted instance may overlap the new one. Writes to one partition are serialised, so the first commit wins and the second invitation is declined automatically within a second or two. Free/busy only suggested the room; the room’s calendar is what enforces it, the same way a booking system re-checks at booking time.
2. Forty required people and no slot works. What do you return?
   The slots with the fewest people in conflict, each listing who is busy, ranked by how many required versus optional guests it breaks. The sweep already counts overlaps at every point, so this costs nothing extra. The UI can then suggest making someone optional or splitting the meeting.
3. One executive’s calendar is queried thousands of times a day by people scheduling with them. Does that hurt?
   Each query reads a few day buckets from one partition, so the load is real but small. If one calendar becomes hot, the free/busy service caches its day buckets for a few seconds keyed by the calendar’s change sequence, the same versioned-key trick as the view cache, which absorbs bursts without serving anything stale after an edit. In practice an assistant with delegate access books most of those meetings anyway.
4. How do you include someone at another company?
   We only know what their provider publishes. If it exposes free/busy, for example through a CalDAV free-busy query or a published free/busy feed, the interop gateway fetches it with a short cache; otherwise they show as unknown and the organiser sends a normal invitation. We never infer their availability from the invitations we have sent them.
5. How would you know free/busy is telling people someone is free when they are not?
   Sample answers and re-derive them: for a random set of free/busy responses, recompute busy intervals from the event store and compare; the mismatch rate should be zero apart from seconds of expander lag. Also watch room auto-declines and conflict warnings on accepted invitations, because a rise means free/busy said free when the calendar was not.

Ask: 40 million reminders are due at 08:50. How do they go out on time, and what stops a reminder for a meeting that has since moved?

Good answers name: Durable timers for the next 7 days, keyed by (calendar, instance, lead), version-checked at fire time, Scan for instances starting in lead-time minutes, every minute, A timer for every instance in the 12-month horizon, Device-local alarms only.

Our pick: The expander creates a timer when it materialises an instance whose reminder falls due in the next 7 days, and a daily roll per calendar, spread over the day by hash, adds timers as they enter that window. A timer’s key is (calendar, instance, lead time), its due time is the instance’s UTC start minus the lead plus a jitter of up to ±30 seconds taken from a hash of the key, and it carries the instance version; when the instance moves, the expander upserts the same key with the new due time and version. Timers live in the job scheduler, partitioned by key hash and indexed by due time, and each partition’s leased dispatcher loads the next minute into a timer wheel. At fire time a timer whose instance changed version, was cancelled or was declined is dropped; valid ones go to the notification system with an idempotency key of (timer, version) and a TTL of the lead time, and the jitter spreads the 40 million to roughly 670 thousand a second. Apps that schedule popup alarms locally register that per device, so the server sends email and pushes only to devices that do not, which keeps most popups out of the burst.

1. A dispatcher dies halfway through the 08:50 timers. What happens?
   Its lease on that partition expires in seconds, and another dispatcher takes the partition, reads its due-time index and reloads the minute. Timers already handed to the notification system are sent again with the same idempotency key, which the notification system drops as duplicates, so users get one reminder. Lateness for that partition is a few seconds, inside the 30-second target.
2. A user changes their default reminder from 10 to 30 minutes. What changes?
   Their next 7 days of instances get new timers: upsert keys with the 30-minute lead and delete the 10-minute ones, around 20 timers for a busy calendar. Instances beyond the window pick up the new lead when the daily roll reaches them. Per-event overrides keep their own lead, since they are separate keys.
3. Why 7 days, and not 48 hours or the whole horizon?
   It trades churn against safety margin. A shorter window means fewer timers to rewrite on edits but less slack if the daily roll fails; 7 days lets the roll be broken for days without a missed reminder. The whole horizon would be hundreds of billions of timers rewritten on every series edit, and leads longer than the window, such as a reminder a week before, are created when their due time enters it.
4. Ten times the users. What breaks first?
   The burst minute: 400 million reminders due at once, about 7 million a second even with the jitter. The timer store shards linearly by partition, so the real ceiling is push and email provider capacity. The levers are local alarms on devices, which remove most push popups, and sending email reminders a minute or two early, which nobody notices.
5. How do you know reminders are on time?
   Every send records due and fire time, and the SLI is the share fired within 30 seconds of due, by minute, which shows whether the :50 burst is the problem. Next to it: the oldest unfired timer per partition, the rate dropped by version checks (a spike means a bulk edit or an expander bug), and a canary calendar per region with a reminder every minute that alerts when one does not arrive.

Ask: A phone was offline for a week. How does it catch up without downloading everything, and how do changes appear within seconds the rest of the time?

Good answers name: Per-calendar change log with a sequence number, opaque sync tokens, tombstones and data-free push hints, "Changed since this timestamp" queries, Full download on every change, Push the changed data itself.

Our pick: Each calendar partition holds a change log of (seq, event_id, kind), written in the same transaction as the change, so a calendar’s sequence is gap-free in commit order. The sync token is a signed (calendar, seq, epoch), opaque to clients. A sync returns the current state of every event changed after seq, collapsed to one item per event with deletions as tombstones, plus a new token. Log entries older than a day are compacted to the latest per event and tombstones are kept 30 days; an older token, or one from an earlier epoch after a restore or re-shard, gets 410 Gone and the client does a full sync, the same contract Google’s public Calendar API uses. After commit, the change stream feeds the hints service, which coalesces to one hint per calendar every two seconds and sends it over open WebSockets or as a collapsible data-only push; apps also sync on open and in the background, because mobile operating systems throttle background pushes.

1. The phone edited an event offline, and the organiser changed it meanwhile. What happens on reconnect?
   The phone sends its edit with the version it saw (If-Match), and the server finds a newer version and returns 412 with the current event. The app rebases field by field: if the organiser changed the time and the user changed the description, both apply; if both changed the time, the server’s version wins and the user is told. Responses and personal reminders never conflict, because only the guest writes them.
2. You restore a shard from backup. What happens to every device syncing against it?
   Their tokens may point past the restored state, so the restore bumps the partition’s epoch and every old token gets 410. That triggers full syncs for every calendar on the shard at once, so the 410 carries a jittered Retry-After and full syncs are paged, which spreads the herd over minutes. Correctness first, then smoothing.
3. A shared room calendar is watched by 10,000 devices. Does each booking send 10,000 hints?
   Coalescing helps first: one hint per calendar per two seconds, however many changes. For calendars with many watchers, the hints service pushes only to devices currently showing that calendar and lets the rest find out on their next sync. Push is a latency optimisation, so it is the first thing to shed.
4. Why can the token not just be a timestamp the client remembers?
   Because two writes can commit in a different order from their timestamps, and a change committed with an earlier timestamp after the client synced would be skipped forever. The sequence is assigned inside the commit, in the partition that serialises writes, so it is exactly commit order. Keeping the token opaque also lets us change what is inside it, such as adding the epoch, without breaking clients.
5. How do you know sync is fast and correct?
   Canary devices in each region edit and watch a test calendar and report commit-to-device latency, and the share of syncs answered with 410 should be tiny outside restores. For correctness, apps periodically send a checksum of their local window and the server compares it with its own; any mismatch rate above noise is a sync bug, caught before users report missing meetings.