Design Zapier

• Build a Zap: One trigger (“new row in this sheet”) and a sequence of actions (“post to Slack, then create a CRM contact”), with field mapping from earlier steps into later ones.
• Connect accounts: OAuth or API keys for each app, stored once and reused across Zaps, refreshed automatically when they expire.
• Detect new events: Instantly by webhook when the app supports it, otherwise by polling the app’s API every 1 to 15 minutes depending on plan. Each new item fires the Zap exactly once.
• Run the steps: Each action calls the app’s API with mapped data; filters and paths decide which steps run. Each successful action counts as a billable task.
• Recover from failures: Retry transient errors, pause on expired credentials and tell the user, and let failed runs be replayed from the step that failed.
• Partners build integrations: Thousands of app integrations are written by the apps’ own developers on our platform. Their code runs on our infrastructure.
• Out of scope: The visual editor’s UX, the template marketplace, billing itself, and AI features that write Zaps.

• Scale (10 M active Zaps · ~23 k polls/s): Most Zaps are idle most of the time, so the system spends far more work checking for events than running them.
• Each new item fires once (no missed and no duplicate triggers): The requirement that drives the hardest trade-off: third-party APIs rarely offer reliable cursors, so “what is new” must be worked out by us, per Zap, from overlapping pages of results, without missing items or firing twice.
• Good citizen to third-party APIs (stay under every app’s rate limits): We share each partner’s API limits across all our users on that app. Exceeding them gets our whole integration throttled or banned, which breaks every customer on it.
• Tenant isolation (one account cannot delay another): A customer importing 50 k rows creates 50 k runs in minutes. Everyone else’s Zaps must keep running on time.
• Latency (instant triggers start < 10 s): For webhooks, users notice delay. Polling latency is set by the plan’s interval, which is a product decision more than a technical one.
• Security of credentials and code (partner code fully isolated): We hold OAuth tokens to millions of accounts and run code written by thousands of third parties. One integration must never read another’s data or tokens.

• Polls per second: ~23 k/s. 10 M active Zaps, 70 % with polling triggers = 7 M. At an average interval of 5 minutes: 7 M ÷ 300 s ≈ 23 k polls/s, around the clock. About 95 % return nothing new.
• Zap runs per second: ~1.2 k avg · ~3.5 k peak. 100 M runs a day ÷ 86 400 ≈ 1.2 k/s; business hours in the US and Europe push peaks to about 3×.
• Outbound API calls: ~27 k/s. Polls (23 k/s) plus action steps: 100 M runs × ~3 actions = 300 M a day ≈ 3.5 k/s. Total ≈ 27 k/s calls to partner APIs, each through partner-written code in a sandbox. Polling, not running Zaps, dominates.
• Webhook events per second: ~600/s. 3 M Zaps with instant triggers receiving 50 M events a day ≈ 580/s average, with bursts many times higher when a partner replays a backlog.
• Seen-item store: ~56 GB. Per polling Zap, keep hashes of the last ~1 000 item ids at 8 B each = 8 KB; 7 M Zaps × 8 KB ≈ 56 GB. Small enough for a key-value store, read on every one of the 23 k polls a second.
• Task history per day: ~1.5 TB. 300 M action steps × ~5 KB of input and output each = 1.5 TB/day, about 45 TB at 30 days’ retention. It is customer data from other apps, so retention is short and purge must be real.

• Zap editor: Builds Zaps, connects accounts through OAuth, tests steps with sample data, and shows run history. Turning a Zap on is the moment subscriptions are created or polling is scheduled.
• Zap API: Creates and versions Zaps, starts OAuth flows, turns Zaps on and off, and serves history. Turning on a webhook trigger calls the integration’s subscribe code; turning on a polling trigger primes its seen-items set.
• Webhook ingress (persist, then 200): Receives partner webhooks on per-Zap URLs, verifies signatures where offered, writes the event durably and answers 200 in milliseconds. Never runs a Zap inline, because senders time out and retry.
• Poll scheduler (time buckets + jitter): Holds every polling Zap’s next due time, spread across the interval with a stable per-Zap offset, and releases each second’s bucket of due polls to the pollers. Keeps 23 k polls a second smooth instead of a spike at every whole minute.
• Zap store (versioned definitions): Zap definitions (versioned, so a run uses the version it started with), connected account references, hook ids and trigger state. Read on every run and every poll, so heavily cached.
• Run queues (per-account partitions): Runs waiting to execute, partitioned by account and drained with weighted fairness, so one account’s 50 k-run import waits behind its own share rather than in front of everyone else.
• Pollers: For each due poll, invoke the trigger’s code in the sandbox, compare returned item ids with the Zap’s seen set, and enqueue a run for each genuinely new item.
• Seen-item store (per-Zap id hashes): For each polling Zap, hashes of the last ~1 000 item ids it has fired on. The answer to “is this new?” when the app offers no reliable cursor. Primed when the Zap is turned on, so existing items never fire.
• Step runner: Executes a run step by step: evaluates filters and paths, maps fields from earlier outputs, invokes each action in the sandbox, persists each step’s result before moving on, and schedules retries or halts.
• Task history (30-day retention): Per run and per step: input, output, status and error. What the user reads to debug, what replay resumes from, and what billing counts. Deleted on schedule, because it is other companies’ customer data.
• Credential vault (encrypted, per connection): OAuth access and refresh tokens and API keys, encrypted with per-tenant keys. Released to the sandbox only for the one invocation that needs them. Refreshes are single-flight per connection.
• Integration sandbox (microVM per invocation): Runs partner-written integration code (trigger, search and action functions) in isolated, short-lived sandboxes with a timeout, no shared state and controlled egress. About 27 k invocations a second, so warm pools matter.
• Rate limiter (per app and connection): Token buckets per app, per connected account and per endpoint, adjusted from Retry-After and 429 responses. Every outbound call takes a token first, so all users of an app share its limits fairly.
• App APIs: Thousands of third-party APIs: inconsistent pagination, rate limits, error codes and auth. They send webhooks to us and receive our polls and actions. We control none of them.

1. A new spreadsheet row runs a Zap: The core path for most Zaps: a scheduled poll, partner code in a sandbox, working out what is new, and running the actions.
   1. The poll scheduler releases this second’s bucket of due polling Zaps.
   2. A poller invokes the trigger’s code in the sandbox with the Zap’s connection.
   3. The integration calls the app’s API for its most recent items.
   4. The poller compares the returned ids with the Zap’s seen set and keeps only new ones.
   5. A run is enqueued in the account’s partition with the item as trigger output.
   6. The step runner executes each action, persisting each step’s result before the next.
2. An instant trigger by webhook: Seconds instead of minutes, and no polling cost, for apps that can push events.
   1. The user turns on a Zap with an instant trigger; the platform subscribes to the app.
   2. A contact is created in the CRM, and the CRM posts the event to the hook URL.
   3. The ingress resolves the hook to its Zap, writes the event durably and answers 200.
   4. The run executes like any other, starting a few seconds after the event.
3. One account imports fifty thousand rows: The scale case: a burst from one tenant, spread over shared partner limits, without delaying everyone else.
   1. A customer pastes 50 k rows into a sheet; the next polls and a webhook backlog produce 50 k runs in minutes.
   2. Runs land in that account’s partition; the runners drain partitions with weighted fairness.
   3. Every action takes a token from the app’s and the connection’s rate-limit buckets.
   4. Runs without a token are pushed back with a delay instead of hammering the API.
   5. The account reaches its plan’s task limit, and further runs are held, not dropped.
4. A token expires and the app says 429: The failure path: refreshing credentials without racing, backing off politely, and resuming a run from the step that failed.
   1. An action gets 401: the OAuth access token has expired.
   2. One invocation refreshes the token under a per-connection lock; the others wait and reuse it.
   3. The retried call now gets 429 with Retry-After: 120.
   4. The runner records the step as waiting and requeues the run with a delay.
   5. If it still fails after the retry budget, the run is marked errored for later replay.
5. A user disconnects an app: The cleanup and privacy path: stop watching, stop calling, and actually forget.
   1. The user disconnects their CRM account; every Zap using it is turned off.
   2. Webhook subscriptions are removed at the app, and polling stops.
   3. Tokens are revoked at the provider and deleted from the vault.
   4. The Zaps’ seen-item sets are dropped and their history is purged on schedule.

Ask: Webhooks are better. Why poll at all, and how do you poll seven million Zaps without melting yourself or the apps?

Good answers name: Webhooks (subscribe on turn-on) wherever the app supports them; polling otherwise, spread into per-second buckets by a stable per-Zap offset, Cron-style: every Zap polled at the top of its interval, One shared poll per account and app, fanned out to all Zaps that watch the same resource, Long-lived streaming connections to each app.

Our pick: For each trigger, prefer the integration’s subscribe/unsubscribe hooks: turning the Zap on registers a per-Zap URL, turning it off removes it. Otherwise, poll at the plan’s interval (1 to 15 minutes), scheduling each Zap at offset = hash(zap_id) mod interval so the fleet’s polls fill every second evenly. The scheduler is a set of time buckets (one per second) holding due Zap ids; pollers drain each bucket and reschedule. Polls that cannot get a rate-limit token are skipped rather than queued, since the next poll sees the same data. Zaps that error repeatedly back off and eventually pause with a notification instead of polling a dead endpoint forever.

1. Could you poll less often when a Zap rarely sees new items?
   Yes, an adaptive interval (back off when nothing changes, return to the plan interval after a hit) can cut polls substantially. The catch is latency after a quiet period, which some users notice. A safe version caps the back-off at a few times the plan interval and never exceeds what the plan promises without saying so.
2. A partner’s webhook delivery breaks silently for a day. How would you notice?
   Track webhook volume per integration against its normal pattern and alert on a sudden drop to near zero. For critical apps, a low-frequency reconciliation poll can catch events missed while hooks were broken, deduplicated against what webhooks already delivered.
3. The poll scheduler loses a node. Are polls lost?
   Due times are stored durably with each Zap; buckets are a cache of the next few minutes. A replacement rebuilds its buckets from the store and polls anything overdue. A poll that runs twice is harmless because the seen set decides what is new.
4. Why skip rather than queue a poll that hits the rate limit?
   Because polls are idempotent snapshots: the next poll returns the same new items plus any newer ones. Queuing skipped polls just builds a backlog of redundant calls that will hit the same limit when they run.

Ask: A poll returns the latest 100 rows. How do you know which of them you have not fired on yet?

Good answers name: Per-Zap set of seen item ids (hashed, last ~1 000), primed when the Zap is turned on, Cursor or “updated since” timestamp kept per Zap, Hash of the whole item, to fire on new and changed items.

Our pick: When a polling Zap is turned on, run the trigger once and store the hashes of every returned id without firing: that is the baseline. On each later poll, compute hash(id) for each returned item, insert into the Zap’s set with an insert-if-absent operation, and enqueue a run only for ids that were absent. The set is bounded (last ~1 000 ids, or by age), sized well above the page size so an id cannot fall out while still being returned. Runs derive their id from (zap_id, item id), so even a race between two polls is caught at enqueue. Where an integration declares a reliable cursor, the trigger uses it instead and the set becomes a safety net. Zapier documents essentially this: triggers must return items newest first with a unique id, and new items are those whose id has not been seen.

1. A user deletes 50 rows and pastes them back. What fires?
   If the app assigns new ids, all 50 fire, which is usually right from the user’s point of view. If it reuses ids still in the seen set, none fire. The behaviour follows the app’s ids, which is why the integration, not the platform, decides which field is the id.
2. The newest 100 items all arrive in one second during an import, but the import added 500. What happens?
   The poll sees only the newest 100; the other 400 are never returned and never fire, unless the trigger paginates. Integrations for bulk-prone apps should page backwards until they reach a seen id, with a cap per poll so one import cannot run one poll for minutes.
3. Why hash the ids instead of storing them?
   Ids can be long strings (email message ids are often 100 bytes). An 8-byte hash keeps the set small and fixed-size; at 1 000 ids per Zap the chance of two ids colliding is negligible.
4. How do you test that a trigger’s dedupe works before thousands of users rely on it?
   The developer platform’s validation runs the trigger twice and checks that every item has an id, that ids are stable across calls, and that items come newest first. Integrations that fail are not publishable.

Ask: Thousands of third-party developers write integration code that runs on your servers with your users’ tokens. How do you contain it?

Good answers name: Each invocation in an isolated microVM or function sandbox, with only that connection’s credentials, a timeout and restricted egress, Integrations as plugins loaded into a shared runner process, Declarative integrations only (HTTP request templates, no code).

Our pick: Run integration code in function sandboxes (Zapier has said it runs partner apps on AWS Lambda), one invocation per trigger poll, search or action, with a 30-second timeout and memory cap. The platform, not the integration, fetches the connection’s decrypted credentials and injects them into that invocation only; the integration never sees the vault. Egress goes through a proxy that logs and can block destinations, and outputs are size-limited. Keep warm pools for popular integrations to hide cold starts. Offer a declarative mode for simple REST APIs, where the platform builds the HTTP requests itself and there is no third-party code to run at all.

1. An integration update starts timing out for every user. How do you limit the damage?
   Version integrations and roll new versions out gradually, with automatic rollback on error-rate jumps. Timeouts cap the cost of each failure, and the error shows on that integration’s dashboard for its developer rather than as a platform outage.
2. Could an integration exfiltrate the token it is given?
   It could send it to its own server, which is why egress is proxied and logged and why integrations are reviewed before publishing. Short-lived access tokens limit the window, and refresh tokens never leave the platform: the sandbox gets access tokens only.
3. How do you keep 27 k invocations a second affordable?
   Warm pools sized per integration by traffic, batching where an integration supports bulk endpoints, and the declarative mode, which runs requests in a shared, trusted HTTP client with no sandbox. Polling is most of the volume, so cutting polls (webhooks, adaptive intervals) cuts sandbox cost most.
4. Where do integration logs go, and who can see them?
   Per-invocation logs go to a log store scoped to the integration, with tokens and secrets scrubbed by the platform before writing. Developers see their own integration’s logs; users see their own runs’ inputs and outputs in task history.

Ask: Every user of the Slack integration shares Slack’s limits on your app. How do you stay under them, and what about token refreshes?

Good answers name: Token buckets per app, per connection and per endpoint, tuned from Retry-After; single-flight refresh per connection with a version check, Call freely and retry on 429, Each run refreshes its own token when it sees a 401.

Our pick: Before every outbound call, the sandbox takes a token from three buckets: the app’s global budget, the connection’s, and the endpoint’s where it has its own limit. Buckets are configured from each app’s documented limits and adjusted at runtime: a 429 with Retry-After drains the relevant bucket for that long, and sustained 429s lower its rate. Calls that cannot get a token are delayed (runs requeued with a delay, polls skipped). For OAuth, refresh is single-flight per connection: take a short lock, re-read the stored token, refresh only if it is still the expired one, and store the new pair with compare-and-swap on a token version, then release; waiters reuse the new token. Refresh proactively a few minutes before expiry to keep the race rare.

1. The limiter itself is down. Fail open or closed?
   Fail open with a conservative local limit per runner host, because failing closed stops every Zap. Local limits are blunt, but a short window of slightly uneven pacing is better than a total outage; alert loudly so it is short.
2. How do you find out an app’s real limits when they are undocumented?
   Learn them: start conservative, increase while calls succeed, and back off on 429 or 503, like TCP congestion control. Record the learned rate per app so restarts do not relearn from zero.
3. A refresh fails because the user revoked access in the app. What then?
   The connection is marked broken, every Zap using it is paused, and the owner is emailed with a reconnect link. Retrying would only generate errors, and pausing makes the cause obvious.
4. Should a big customer get a bigger share of a partner’s app-wide limit?
   Weighted by plan, yes, but with a floor for everyone, so small accounts are never starved. Per-connection limits come from the partner and apply regardless: no plan buys more than Slack allows on one token.

Ask: One customer creates fifty thousand runs in two minutes. Why doesn’t everyone else’s Zap wait behind them?

Good answers name: Per-account partitions of the run queue, drained with weighted fair scheduling by plan, One global FIFO queue, Separate runner clusters per plan tier.

Our pick: Partition runs by account (logically, for example account-keyed sub-queues in a queue that supports many keys, or a sharded queue with an index of non-empty accounts). Runners pick the next account by weighted round robin over accounts with pending work, weights by plan, and take a small batch from that account. Runs blocked on rate limits are requeued with a delay instead of occupying a runner. Per-account concurrency caps stop one account holding every runner, and task quotas hold runs beyond the plan’s limit visibly. Measure fairness directly: queueing delay per account, at p50 and p99, should be flat across accounts that are not bursting.

1. Should runs within one account stay in order?
   Within one Zap, often yes: a “new row” Zap that creates invoices should process rows in order. Ordering per Zap (not per account) is cheap to keep by sequencing runs per Zap; ordering across all of an account’s Zaps is not worth the lost parallelism.
2. A huge account pays for priority. How do you give it more without starving others?
   Higher weight in the round robin and a higher concurrency cap, but never strict priority. Strict priority lets one large backlog starve everyone below it indefinitely; weights guarantee every account progress.
3. How do you find the noisy accounts?
   Queue depth and runs per minute per account, top-N, with alerts on sudden jumps. Often the cause is a Zap in a loop (it writes to the sheet it triggers on), which deserves automatic detection and a pause.
4. A Zap triggers itself in a loop. How do you stop it?
   Detect it: runs of the same Zap whose action output becomes its own trigger input, or a run rate far above the Zap’s history. Pause the Zap and notify the owner. Without this, a loop burns the user’s task quota and the partner’s rate limits in minutes.

Ask: Step three of a five-step Zap fails. When you retry, how do you avoid redoing steps one and two?

Good answers name: Persist each step’s result before starting the next; retries and replays resume at the first incomplete step, Retry the whole run from the trigger, Pass idempotency keys to every action.

Our pick: The runner executes steps in order and writes each step’s status and output to history before starting the next. Transient errors (timeouts, 5xx, 429) requeue the run with exponential backoff, resuming at the failed step with its stored inputs. After the retry budget, the run is marked errored, and the user can replay it from that step once the cause is fixed, with optional automatic replay for transient failures. The step that was in flight during a crash has an unknown outcome and may repeat; integrations reduce that by passing a key derived from (run_id, step) where the app supports idempotency, or by upserting on a natural key. Say plainly that delivery is at-least-once per step, with completed steps never repeated.

1. The runner crashes right after the CRM call succeeded but before writing the result. What happens?
   The run is picked up again at step 2, whose status is still “started”, and the CRM call is made again. If the integration upserts by email or passes an idempotency key, the second call finds the first contact; otherwise there are two contacts. This window is small, and integration guidelines push towards safe actions for exactly this reason.
2. A user edits the Zap while a run is waiting to retry. Which version does the retry use?
   The version the run started with, recorded on the run. Changing field mappings under a half-finished run would mix two definitions and produce outputs that neither version would.
3. Should every errored run be replayed automatically?
   Only transient failures, with limits. Replaying a run that failed on bad data or a missing field just fails again and burns tasks. Classify errors (retryable or not, from status codes and integration hints) and replay only the retryable ones.
4. How do you count billable tasks when steps can repeat?
   Count a task when a step’s success is recorded in history, once per (run_id, step). A repeated attempt after a crash records success once, so the user is never billed twice for the same step, even when the partner saw two calls.