System design interview question

Design Zapier

Q: Webhooks are better. Why poll at all, and how do you poll seven million Zaps without melting yourself or the apps?

For each trigger, prefer the integration’s subscribe/unsubscribe hooks: turning the Zap on registers a per-Zap URL, turning it off removes it. Otherwise, poll at the plan’s interval (1 to 15 minutes), scheduling each Zap at offset = hash(zap_id) mod interval so the fleet’s polls fill every second evenly. The scheduler is a set of time buckets (one per second) holding due Zap ids; pollers drain each bucket and reschedule. Polls that cannot get a rate-limit token are skipped rather than queued, since the next poll sees the same data. Zaps that error repeatedly back off and eventually pause with a notification instead of polling a dead endpoint forever.

Q: A poll returns the latest 100 rows. How do you know which of them you have not fired on yet?

When a polling Zap is turned on, run the trigger once and store the hashes of every returned id without firing: that is the baseline. On each later poll, compute hash(id) for each returned item, insert into the Zap’s set with an insert-if-absent operation, and enqueue a run only for ids that were absent. The set is bounded (last ~1 000 ids, or by age), sized well above the page size so an id cannot fall out while still being returned. Runs derive their id from (zap_id, item id), so even a race between two polls is caught at enqueue. Where an integration declares a reliable cursor, the trigger uses it instead and the set becomes a safety net. Zapier documents essentially this: triggers must return items newest first with a unique id, and new items are those whose id has not been seen.

Q: Thousands of third-party developers write integration code that runs on your servers with your users’ tokens. How do you contain it?

Run integration code in function sandboxes (Zapier has said it runs partner apps on AWS Lambda), one invocation per trigger poll, search or action, with a 30-second timeout and memory cap. The platform, not the integration, fetches the connection’s decrypted credentials and injects them into that invocation only; the integration never sees the vault. Egress goes through a proxy that logs and can block destinations, and outputs are size-limited. Keep warm pools for popular integrations to hide cold starts. Offer a declarative mode for simple REST APIs, where the platform builds the HTTP requests itself and there is no third-party code to run at all.

Q: Every user of the Slack integration shares Slack’s limits on your app. How do you stay under them, and what about token refreshes?

Before every outbound call, the sandbox takes a token from three buckets: the app’s global budget, the connection’s, and the endpoint’s where it has its own limit. Buckets are configured from each app’s documented limits and adjusted at runtime: a 429 with Retry-After drains the relevant bucket for that long, and sustained 429s lower its rate. Calls that cannot get a token are delayed (runs requeued with a delay, polls skipped). For OAuth, refresh is single-flight per connection: take a short lock, re-read the stored token, refresh only if it is still the expired one, and store the new pair with compare-and-swap on a token version, then release; waiters reuse the new token. Refresh proactively a few minutes before expiry to keep the race rare.

Q: One customer creates fifty thousand runs in two minutes. Why doesn’t everyone else’s Zap wait behind them?

Partition runs by account (logically, for example account-keyed sub-queues in a queue that supports many keys, or a sharded queue with an index of non-empty accounts). Runners pick the next account by weighted round robin over accounts with pending work, weights by plan, and take a small batch from that account. Runs blocked on rate limits are requeued with a delay instead of occupying a runner. Per-account concurrency caps stop one account holding every runner, and task quotas hold runs beyond the plan’s limit visibly. Measure fairness directly: queueing delay per account, at p50 and p99, should be flat across accounts that are not bursting.

Q: Step three of a five-step Zap fails. When you retry, how do you avoid redoing steps one and two?

The runner executes steps in order and writes each step’s status and output to history before starting the next. Transient errors (timeouts, 5xx, 429) requeue the run with exponential backoff, resuming at the failed step with its stored inputs. After the retry budget, the run is marked errored, and the user can replay it from that step once the cause is fixed, with optional automatic replay for transient failures. The step that was in flight during a crash has an unknown outcome and may repeat; integrations reduce that by passing a key derived from (run_id, step) where the app supports idempotency, or by upserting on a natural key. Say plainly that delivery is at-least-once per step, with completed steps never repeated.

Connect thousands of third-party apps: watch ten million triggers by polling or webhook, and run each new event through multi-step workflows on someone else’s APIs.

Last updated 2026-10-10. Difficulty: hard. Patterns: workflow-orchestration, polling, webhooks, multi-tenancy, rate-limiting. Reported at Zapier, Salesforce, HubSpot, Atlassian, Microsoft.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Skip to any stage or key moment, and ask the AI Mentor anything along the way.

Walk through it Take the mock interview

Or sit it in the style of Salesforce · HubSpot · Atlassian · Microsoft, run as that company’s round is reported to go.

The short answer

How to design Zapier in a system design interview, in five steps:

Scope it. Build a Zap, connect accounts, detect new events, run the steps, recover from failures, partners build integrations.
Set the targets. Scale: 10 M active Zaps · ~23 k polls/s; Each new item fires once: no missed and no duplicate triggers; Good citizen to third-party APIs: stay under every app’s rate limits; Tenant isolation: one account cannot delay another.
Do the numbers. Polls per second: ~23 k/s; Zap runs per second: ~1.2 k avg · ~3.5 k peak; Outbound API calls: ~27 k/s.
Draw the design. Zap API, Webhook ingress, Poll scheduler, Zap store, Run queues, Pollers, Seen-item store, Step runner, Task history.
Go deep where it is hard.
- Polling ten million triggers → Webhooks (subscribe on turn-on) wherever the app supports them; polling otherwise, spread into per-second buckets by a stable per-Zap offset.
- Working out what is new → Per-Zap set of seen item ids (hashed, last ~1 000), primed when the Zap is turned on.
- Running partners’ code → Each invocation in an isolated microVM or function sandbox, with only that connection’s credentials, a timeout and restricted egress.
- Sharing someone else’s rate limits → Token buckets per app, per connection and per endpoint, tuned from Retry-After; single-flight refresh per connection with a version check.
- Keeping one account from delaying everyone → Per-account partitions of the run queue, drained with weighted fair scheduling by plan.
- What “run once” means across five apps → Persist each step’s result before starting the next; retries and replays resume at the first incomplete step.

Functional requirements

Build a Zap. One trigger (“new row in this sheet”) and a sequence of actions (“post to Slack, then create a CRM contact”), with field mapping from earlier steps into later ones.
Connect accounts. OAuth or API keys for each app, stored once and reused across Zaps, refreshed automatically when they expire.
Detect new events. Instantly by webhook when the app supports it, otherwise by polling the app’s API every 1 to 15 minutes depending on plan. Each new item fires the Zap exactly once.
Run the steps. Each action calls the app’s API with mapped data; filters and paths decide which steps run. Each successful action counts as a billable task.
Recover from failures. Retry transient errors, pause on expired credentials and tell the user, and let failed runs be replayed from the step that failed.
Partners build integrations. Thousands of app integrations are written by the apps’ own developers on our platform. Their code runs on our infrastructure.
Out of scope. The visual editor’s UX, the template marketplace, billing itself, and AI features that write Zaps.

Non-functional requirements

Scale (10 M active Zaps · ~23 k polls/s). Most Zaps are idle most of the time, so the system spends far more work checking for events than running them.
Each new item fires once (no missed and no duplicate triggers). The requirement that drives the hardest trade-off: third-party APIs rarely offer reliable cursors, so “what is new” must be worked out by us, per Zap, from overlapping pages of results, without missing items or firing twice.
Good citizen to third-party APIs (stay under every app’s rate limits). We share each partner’s API limits across all our users on that app. Exceeding them gets our whole integration throttled or banned, which breaks every customer on it.
Tenant isolation (one account cannot delay another). A customer importing 50 k rows creates 50 k runs in minutes. Everyone else’s Zaps must keep running on time.
Latency (instant triggers start < 10 s). For webhooks, users notice delay. Polling latency is set by the plan’s interval, which is a product decision more than a technical one.
Security of credentials and code (partner code fully isolated). We hold OAuth tokens to millions of accounts and run code written by thousands of third parties. One integration must never read another’s data or tokens.

Back-of-envelope estimates

Polls per second: ~23 k/s. 10 M active Zaps, 70 % with polling triggers = 7 M. At an average interval of 5 minutes: 7 M ÷ 300 s ≈ 23 k polls/s, around the clock. About 95 % return nothing new.
Zap runs per second: ~1.2 k avg · ~3.5 k peak. 100 M runs a day ÷ 86 400 ≈ 1.2 k/s; business hours in the US and Europe push peaks to about 3×.
Outbound API calls: ~27 k/s. Polls (23 k/s) plus action steps: 100 M runs × ~3 actions = 300 M a day ≈ 3.5 k/s. Total ≈ 27 k/s calls to partner APIs, each through partner-written code in a sandbox. Polling, not running Zaps, dominates.
Webhook events per second: ~600/s. 3 M Zaps with instant triggers receiving 50 M events a day ≈ 580/s average, with bursts many times higher when a partner replays a backlog.
Seen-item store: ~56 GB. Per polling Zap, keep hashes of the last ~1 000 item ids at 8 B each = 8 KB; 7 M Zaps × 8 KB ≈ 56 GB. Small enough for a key-value store, read on every one of the 23 k polls a second.
Task history per day: ~1.5 TB. 300 M action steps × ~5 KB of input and output each = 1.5 TB/day, about 45 TB at 30 days’ retention. It is customer data from other apps, so retention is short and purge must be real.

Components

Zap editor: Builds Zaps, connects accounts through OAuth, tests steps with sample data, and shows run history. Turning a Zap on is the moment subscriptions are created or polling is scheduled.
Zap API: Creates and versions Zaps, starts OAuth flows, turns Zaps on and off, and serves history. Turning on a webhook trigger calls the integration’s subscribe code; turning on a polling trigger primes its seen-items set.
Webhook ingress (persist, then 200): Receives partner webhooks on per-Zap URLs, verifies signatures where offered, writes the event durably and answers 200 in milliseconds. Never runs a Zap inline, because senders time out and retry.
Poll scheduler (time buckets + jitter): Holds every polling Zap’s next due time, spread across the interval with a stable per-Zap offset, and releases each second’s bucket of due polls to the pollers. Keeps 23 k polls a second smooth instead of a spike at every whole minute.
Zap store (versioned definitions): Zap definitions (versioned, so a run uses the version it started with), connected account references, hook ids and trigger state. Read on every run and every poll, so heavily cached.
Run queues (per-account partitions): Runs waiting to execute, partitioned by account and drained with weighted fairness, so one account’s 50 k-run import waits behind its own share rather than in front of everyone else.
Pollers: For each due poll, invoke the trigger’s code in the sandbox, compare returned item ids with the Zap’s seen set, and enqueue a run for each genuinely new item.
Seen-item store (per-Zap id hashes): For each polling Zap, hashes of the last ~1 000 item ids it has fired on. The answer to “is this new?” when the app offers no reliable cursor. Primed when the Zap is turned on, so existing items never fire.
Step runner: Executes a run step by step: evaluates filters and paths, maps fields from earlier outputs, invokes each action in the sandbox, persists each step’s result before moving on, and schedules retries or halts.
Task history (30-day retention): Per run and per step: input, output, status and error. What the user reads to debug, what replay resumes from, and what billing counts. Deleted on schedule, because it is other companies’ customer data.
Credential vault (encrypted, per connection): OAuth access and refresh tokens and API keys, encrypted with per-tenant keys. Released to the sandbox only for the one invocation that needs them. Refreshes are single-flight per connection.
Integration sandbox (microVM per invocation): Runs partner-written integration code (trigger, search and action functions) in isolated, short-lived sandboxes with a timeout, no shared state and controlled egress. About 27 k invocations a second, so warm pools matter.
Rate limiter (per app and connection): Token buckets per app, per connected account and per endpoint, adjusted from Retry-After and 429 responses. Every outbound call takes a token first, so all users of an app share its limits fairly.
App APIs: Thousands of third-party APIs: inconsistent pagination, rate limits, error codes and auth. They send webhooks to us and receive our polls and actions. We control none of them.

User flows

A new spreadsheet row runs a Zap. The core path for most Zaps: a scheduled poll, partner code in a sandbox, working out what is new, and running the actions.
1. The poll scheduler releases this second’s bucket of due polling Zaps. Each Zap’s poll time is its interval plus a stable offset from hash(zap_id), so 7 M Zaps on 5-minute intervals land evenly at ~23 k a second instead of all at :00. The offset is stable so a Zap’s polls stay evenly spaced.
2. A poller invokes the trigger’s code in the sandbox with the Zap’s connection. The sandbox receives the access token for this one connection only, and takes a rate-limit token for the app before calling out. If the bucket is empty, the poll is skipped this round rather than queued, because the next poll will see the same items.
3. The integration calls the app’s API for its most recent items. Most triggers fetch the newest page (say 100 items, newest first). The page overlaps heavily with the last poll’s, which is why the next step exists.
4. The poller compares the returned ids with the Zap’s seen set and keeps only new ones. One new id out of a hundred returned is typical. The new id is added to the set atomically with the decision to enqueue, so two overlapping polls of the same Zap cannot both fire on it.
5. A run is enqueued in the account’s partition with the item as trigger output.
6. The step runner executes each action, persisting each step’s result before the next. Filters and paths are evaluated, fields are mapped from the trigger output into the action’s input, and the Slack post is sent. Each completed step is recorded (and billed as a task) before the runner moves on, so a crash resumes from the next step, not from the beginning.
An instant trigger by webhook. Seconds instead of minutes, and no polling cost, for apps that can push events.
1. The user turns on a Zap with an instant trigger; the platform subscribes to the app. The integration’s subscribe function registers a per-Zap URL with the app (the REST hooks pattern), and the returned subscription id is stored so it can be removed when the Zap is turned off.
2. A contact is created in the CRM, and the CRM posts the event to the hook URL.
3. The ingress resolves the hook to its Zap, writes the event durably and answers 200. Persist, then acknowledge, in milliseconds. Running the Zap inline would hold the sender’s connection for seconds; senders time out and retry, creating duplicates and making us look broken to the partner.
4. The run executes like any other, starting a few seconds after the event. If the sender retries the same event, the run id derived from the event’s id (where the app sends one) detects the duplicate. Apps that send no event id get a hash of the body within a short window instead.
One account imports fifty thousand rows. The scale case: a burst from one tenant, spread over shared partner limits, without delaying everyone else.
1. A customer pastes 50 k rows into a sheet; the next polls and a webhook backlog produce 50 k runs in minutes. Some triggers also cap items per poll, so a big import drains over several polls instead of one enormous batch.
2. Runs land in that account’s partition; the runners drain partitions with weighted fairness. Each account gets a share of runner capacity weighted by plan. The importer’s 50 k runs take their share and wait; other accounts’ single runs go straight through. A single global queue would put everyone behind the import.
3. Every action takes a token from the app’s and the connection’s rate-limit buckets. If Slack allows about one message a second per channel, 50 k messages take hours however much capacity we add. The bucket paces them; runs that cannot get a token are delayed rather than sent to fail.
4. Runs without a token are pushed back with a delay instead of hammering the API. A delayed retry costs a queue entry; a 429 costs the partner’s goodwill and, if repeated across thousands of users, our whole integration’s access.
5. The account reaches its plan’s task limit, and further runs are held, not dropped. Held runs are visible in history and can be released after an upgrade or at the next billing period. Silently dropping them would be data loss from the customer’s point of view.
A token expires and the app says 429. The failure path: refreshing credentials without racing, backing off politely, and resuming a run from the step that failed.
1. An action gets 401: the OAuth access token has expired. Hundreds of runs on the same connection may hit the same 401 within a second. If each refreshes independently, many providers invalidate the refresh token on first use, and all but one refresh fails.
2. One invocation refreshes the token under a per-connection lock; the others wait and reuse it. Single-flight: take a short lock on the connection, re-read the token (someone may already have refreshed it), refresh only if still stale, and write the new pair with a version check.
3. The retried call now gets 429 with Retry-After: 120. The rate limiter drains that connection’s bucket for 120 s, so other runs on the connection wait instead of making the 429 worse.
4. The runner records the step as waiting and requeues the run with a delay. Earlier steps’ results are already persisted. When the run resumes it starts at this step with the stored inputs, so the Slack message from step 2 is not sent again.
5. If it still fails after the retry budget, the run is marked errored for later replay. The user sees which step failed and why, and can replay it after fixing the cause; a 401 that refresh cannot fix (revoked access) pauses the Zap and emails the owner instead of failing every run.
A user disconnects an app. The cleanup and privacy path: stop watching, stop calling, and actually forget.
1. The user disconnects their CRM account; every Zap using it is turned off.
2. Webhook subscriptions are removed at the app, and polling stops. The unsubscribe function is called with the stored subscription id. Left behind, the app would keep posting events we must accept and discard, and some partners penalise endpoints that fail.
3. Tokens are revoked at the provider and deleted from the vault. Revoking at the provider, not just deleting our copy, means a leaked backup of our vault is useless for this account.
4. The Zaps’ seen-item sets are dropped and their history is purged on schedule. If the user reconnects later, the trigger re-primes its seen set, so old items do not fire. History follows the 30-day retention, or is deleted immediately on account deletion requests.

Deep dives

Polling ten million triggers

Webhooks are better. Why poll at all, and how do you poll seven million Zaps without melting yourself or the apps?

Many apps offer no webhooks, or webhooks for only some events. The only way to notice a new row, a new email or an updated deal in those apps is to ask. With 7 M polling Zaps at a 5-minute average interval, that is 23 k API calls a second, about 95 % of which return nothing new.

Polling cost is paid both by us (sandbox invocations) and by the partner (requests against their limits). The two levers are using push wherever it exists and spreading pulls so evenly that neither side sees spikes.

Webhooks (subscribe on turn-on) wherever the app supports them; polling otherwise, spread into per-second buckets by a stable per-Zap offset chosen
Cron-style: every Zap polled at the top of its interval rejected
One shared poll per account and app, fanned out to all Zaps that watch the same resource situational: many Zaps on the same connection and resource, for example a team’s shared inbox
Long-lived streaming connections to each app situational: the handful of apps with first-class streaming APIs

The answer: For each trigger, prefer the integration’s subscribe/unsubscribe hooks: turning the Zap on registers a per-Zap URL, turning it off removes it. Otherwise, poll at the plan’s interval (1 to 15 minutes), scheduling each Zap at offset = hash(zap_id) mod interval so the fleet’s polls fill every second evenly. The scheduler is a set of time buckets (one per second) holding due Zap ids; pollers drain each bucket and reschedule. Polls that cannot get a rate-limit token are skipped rather than queued, since the next poll sees the same data. Zaps that error repeatedly back off and eventually pause with a notification instead of polling a dead endpoint forever.

Could you poll less often when a Zap rarely sees new items?

Yes, an adaptive interval (back off when nothing changes, return to the plan interval after a hit) can cut polls substantially. The catch is latency after a quiet period, which some users notice. A safe version caps the back-off at a few times the plan interval and never exceeds what the plan promises without saying so.

A partner’s webhook delivery breaks silently for a day. How would you notice?

Track webhook volume per integration against its normal pattern and alert on a sudden drop to near zero. For critical apps, a low-frequency reconciliation poll can catch events missed while hooks were broken, deduplicated against what webhooks already delivered.

The poll scheduler loses a node. Are polls lost?

Due times are stored durably with each Zap; buckets are a cache of the next few minutes. A replacement rebuilds its buckets from the store and polls anything overdue. A poll that runs twice is harmless because the seen set decides what is new.

Why skip rather than queue a poll that hits the rate limit?

Because polls are idempotent snapshots: the next poll returns the same new items plus any newer ones. Queuing skipped polls just builds a backlog of redundant calls that will hit the same limit when they run.

Working out what is new

A poll returns the latest 100 rows. How do you know which of them you have not fired on yet?

Most APIs return “the latest N items”, not “items since my cursor”. Consecutive polls overlap almost entirely, items can be inserted out of order, and some APIs return items whose timestamps lie. Firing on every returned item would trigger each row hundreds of times; firing only on items newer than a timestamp would miss late arrivals.

Every item has an id that is stable, even when nothing else about the API is. If we remember which ids each Zap has fired on, “new” is simply “not seen before”, whatever order the items arrive in.

Per-Zap set of seen item ids (hashed, last ~1 000), primed when the Zap is turned on chosen
Cursor or “updated since” timestamp kept per Zap situational: APIs with a trustworthy, monotonic cursor such as a change feed or sequence number
Hash of the whole item, to fire on new and changed items situational: “updated item” triggers, hashing only the fields the user cares about

The answer: When a polling Zap is turned on, run the trigger once and store the hashes of every returned id without firing: that is the baseline. On each later poll, compute hash(id) for each returned item, insert into the Zap’s set with an insert-if-absent operation, and enqueue a run only for ids that were absent. The set is bounded (last ~1 000 ids, or by age), sized well above the page size so an id cannot fall out while still being returned. Runs derive their id from (zap_id, item id), so even a race between two polls is caught at enqueue. Where an integration declares a reliable cursor, the trigger uses it instead and the set becomes a safety net. Zapier documents essentially this: triggers must return items newest first with a unique id, and new items are those whose id has not been seen.

A user deletes 50 rows and pastes them back. What fires?

If the app assigns new ids, all 50 fire, which is usually right from the user’s point of view. If it reuses ids still in the seen set, none fire. The behaviour follows the app’s ids, which is why the integration, not the platform, decides which field is the id.

The newest 100 items all arrive in one second during an import, but the import added 500. What happens?

The poll sees only the newest 100; the other 400 are never returned and never fire, unless the trigger paginates. Integrations for bulk-prone apps should page backwards until they reach a seen id, with a cap per poll so one import cannot run one poll for minutes.

Why hash the ids instead of storing them?

Ids can be long strings (email message ids are often 100 bytes). An 8-byte hash keeps the set small and fixed-size; at 1 000 ids per Zap the chance of two ids colliding is negligible.

How do you test that a trigger’s dedupe works before thousands of users rely on it?

The developer platform’s validation runs the trigger twice and checks that every item has an id, that ids are stable across calls, and that items come newest first. Integrations that fail are not publishable.

Running partners’ code

Thousands of third-party developers write integration code that runs on your servers with your users’ tokens. How do you contain it?

Integrations are written by the apps’ own developers and by contractors, in JavaScript, against our platform SDK. Their code handles OAuth tokens for millions of accounts and runs about 27 k times a second. A bug can loop forever; a malicious or compromised integration could try to read other integrations’ tokens or other users’ data.

Each invocation is short, stateless and needs exactly one connection’s credentials. That makes per-invocation isolation affordable, which is the strongest kind.

Each invocation in an isolated microVM or function sandbox, with only that connection’s credentials, a timeout and restricted egress chosen
Integrations as plugins loaded into a shared runner process rejected
Declarative integrations only (HTTP request templates, no code) situational: simple REST APIs, which is a large share of integrations; offer it alongside code

The answer: Run integration code in function sandboxes (Zapier has said it runs partner apps on AWS Lambda), one invocation per trigger poll, search or action, with a 30-second timeout and memory cap. The platform, not the integration, fetches the connection’s decrypted credentials and injects them into that invocation only; the integration never sees the vault. Egress goes through a proxy that logs and can block destinations, and outputs are size-limited. Keep warm pools for popular integrations to hide cold starts. Offer a declarative mode for simple REST APIs, where the platform builds the HTTP requests itself and there is no third-party code to run at all.

An integration update starts timing out for every user. How do you limit the damage?

Version integrations and roll new versions out gradually, with automatic rollback on error-rate jumps. Timeouts cap the cost of each failure, and the error shows on that integration’s dashboard for its developer rather than as a platform outage.

Could an integration exfiltrate the token it is given?

It could send it to its own server, which is why egress is proxied and logged and why integrations are reviewed before publishing. Short-lived access tokens limit the window, and refresh tokens never leave the platform: the sandbox gets access tokens only.

How do you keep 27 k invocations a second affordable?

Warm pools sized per integration by traffic, batching where an integration supports bulk endpoints, and the declarative mode, which runs requests in a shared, trusted HTTP client with no sandbox. Polling is most of the volume, so cutting polls (webhooks, adaptive intervals) cuts sandbox cost most.

Where do integration logs go, and who can see them?

Per-invocation logs go to a log store scoped to the integration, with tokens and secrets scrubbed by the platform before writing. Developers see their own integration’s logs; users see their own runs’ inputs and outputs in task history.

Sharing someone else’s rate limits

Every user of the Slack integration shares Slack’s limits on your app. How do you stay under them, and what about token refreshes?

Partner APIs limit requests per app, per user token and per endpoint, and many punish the whole app (our integration) for repeated violations. Our users neither know nor care: one of them importing 50 k rows can exhaust limits that thousands of others depend on.

OAuth adds a race. Access tokens expire hourly, and many providers rotate the refresh token on use, invalidating the old one. If 200 runs on one connection all notice expiry at once and all refresh, one succeeds and 199 lose the connection.

Token buckets per app, per connection and per endpoint, tuned from Retry-After; single-flight refresh per connection with a version check chosen
Call freely and retry on 429 rejected
Each run refreshes its own token when it sees a 401 rejected

The answer: Before every outbound call, the sandbox takes a token from three buckets: the app’s global budget, the connection’s, and the endpoint’s where it has its own limit. Buckets are configured from each app’s documented limits and adjusted at runtime: a 429 with Retry-After drains the relevant bucket for that long, and sustained 429s lower its rate. Calls that cannot get a token are delayed (runs requeued with a delay, polls skipped). For OAuth, refresh is single-flight per connection: take a short lock, re-read the stored token, refresh only if it is still the expired one, and store the new pair with compare-and-swap on a token version, then release; waiters reuse the new token. Refresh proactively a few minutes before expiry to keep the race rare.

The limiter itself is down. Fail open or closed?

Fail open with a conservative local limit per runner host, because failing closed stops every Zap. Local limits are blunt, but a short window of slightly uneven pacing is better than a total outage; alert loudly so it is short.

How do you find out an app’s real limits when they are undocumented?

Learn them: start conservative, increase while calls succeed, and back off on 429 or 503, like TCP congestion control. Record the learned rate per app so restarts do not relearn from zero.

A refresh fails because the user revoked access in the app. What then?

The connection is marked broken, every Zap using it is paused, and the owner is emailed with a reconnect link. Retrying would only generate errors, and pausing makes the cause obvious.

Should a big customer get a bigger share of a partner’s app-wide limit?

Weighted by plan, yes, but with a floor for everyone, so small accounts are never starved. Per-connection limits come from the partner and apply regardless: no plan buys more than Slack allows on one token.

Keeping one account from delaying everyone

One customer creates fifty thousand runs in two minutes. Why doesn’t everyone else’s Zap wait behind them?

Runs arrive in bursts per account: imports, backfills, a webhook backlog replayed by a partner. With a single queue, a burst of 50 k runs at the head means every other account’s run waits until they are processed, and those runs may themselves be rate-limited to one per second.

Fairness has to be decided where work is dequeued, not where it is produced, because producers (pollers, webhooks) cannot know what else is waiting.

Per-account partitions of the run queue, drained with weighted fair scheduling by plan chosen
One global FIFO queue rejected
Separate runner clusters per plan tier situational: enterprise customers with contractual isolation, layered on top of per-account fairness

The answer: Partition runs by account (logically, for example account-keyed sub-queues in a queue that supports many keys, or a sharded queue with an index of non-empty accounts). Runners pick the next account by weighted round robin over accounts with pending work, weights by plan, and take a small batch from that account. Runs blocked on rate limits are requeued with a delay instead of occupying a runner. Per-account concurrency caps stop one account holding every runner, and task quotas hold runs beyond the plan’s limit visibly. Measure fairness directly: queueing delay per account, at p50 and p99, should be flat across accounts that are not bursting.

Should runs within one account stay in order?

Within one Zap, often yes: a “new row” Zap that creates invoices should process rows in order. Ordering per Zap (not per account) is cheap to keep by sequencing runs per Zap; ordering across all of an account’s Zaps is not worth the lost parallelism.

A huge account pays for priority. How do you give it more without starving others?

Higher weight in the round robin and a higher concurrency cap, but never strict priority. Strict priority lets one large backlog starve everyone below it indefinitely; weights guarantee every account progress.

How do you find the noisy accounts?

Queue depth and runs per minute per account, top-N, with alerts on sudden jumps. Often the cause is a Zap in a loop (it writes to the sheet it triggers on), which deserves automatic detection and a pause.

A Zap triggers itself in a loop. How do you stop it?

Detect it: runs of the same Zap whose action output becomes its own trigger input, or a run rate far above the Zap’s history. Pause the Zap and notify the owner. Without this, a loop burns the user’s task quota and the partner’s rate limits in minutes.

What “run once” means across five apps

Step three of a five-step Zap fails. When you retry, how do you avoid redoing steps one and two?

A Zap’s steps are calls to different APIs, most of which offer no idempotency keys. If a run is retried from the start, the Slack message from step 1 is posted twice and the CRM contact from step 2 is created twice. If it is not retried, a transient error loses the run.

We cannot make partner APIs transactional. We can make our own record of what has already happened precise enough that a retry only repeats the step whose outcome is unknown.

Persist each step’s result before starting the next; retries and replays resume at the first incomplete step chosen
Retry the whole run from the trigger rejected
Pass idempotency keys to every action situational: apps with idempotency keys (payments, some CRMs) or a field the integration can upsert on

The answer: The runner executes steps in order and writes each step’s status and output to history before starting the next. Transient errors (timeouts, 5xx, 429) requeue the run with exponential backoff, resuming at the failed step with its stored inputs. After the retry budget, the run is marked errored, and the user can replay it from that step once the cause is fixed, with optional automatic replay for transient failures. The step that was in flight during a crash has an unknown outcome and may repeat; integrations reduce that by passing a key derived from (run_id, step) where the app supports idempotency, or by upserting on a natural key. Say plainly that delivery is at-least-once per step, with completed steps never repeated.

The runner crashes right after the CRM call succeeded but before writing the result. What happens?

The run is picked up again at step 2, whose status is still “started”, and the CRM call is made again. If the integration upserts by email or passes an idempotency key, the second call finds the first contact; otherwise there are two contacts. This window is small, and integration guidelines push towards safe actions for exactly this reason.

A user edits the Zap while a run is waiting to retry. Which version does the retry use?

The version the run started with, recorded on the run. Changing field mappings under a half-finished run would mix two definitions and produce outputs that neither version would.

Should every errored run be replayed automatically?

Only transient failures, with limits. Replaying a run that failed on bad data or a missing field just fails again and burns tasks. Classify errors (retryable or not, from status codes and integration hints) and replay only the retryable ones.

How do you count billable tasks when steps can repeat?

Count a task when a step’s success is recorded in history, once per (run_id, step). A repeated attempt after a crash records success once, so the user is never billed twice for the same step, even when the partner saw two calls.

The theory behind it

Exponential backoff and jitter: an interactive playground. Send the same burst of clients at a busy server with five retry strategies and watch synchronized retry waves, retry storms and how jitter spreads them out.
Shuffle sharding: an interactive playground. Let one poison customer crash its workers on a shared fleet, in cells and with shuffle sharding, and see how many other customers go down, degrade or stay untouched.
Fair queuing: an interactive playground. Share eight workers between three tenants while one floods the queue, and compare a FIFO queue, per-tenant caps and fair round-robin queuing on waits and finish times.
Webhook delivery: an interactive playground. Send webhooks to an endpoint that goes down for 30 minutes, 12 hours or 4 days, and compare dropping, per-event backoff and a per-endpoint queue with a breaker on lost events, wasted calls and ordering.
Serverless cold starts: an interactive playground. Send quiet, steady and spiky traffic to a serverless function on demand, with a keep-warm ping or with provisioned concurrency, and compare cold starts, peak instances and monthly cost.
Designing webhooks: How to deliver webhooks reliably: event payloads, signing and verification, retries with backoff, ordering, idempotency on the receiver, per-customer isolation, and replay, as Stripe and GitHub do it.
Rate limiting algorithms: The algorithms behind rate limiters and how to run them across many servers: fixed window, sliding window log, sliding window counter, token bucket, leaky bucket and GCRA, with their accuracy, memory and burst behaviour, plus distributed counters, Lua atomicity and response headers.
Multi-tenancy: Designing SaaS systems that serve many customers from shared infrastructure: tenancy models from shared tables to dedicated stacks, tenant isolation for data and security, noisy neighbours, per-tenant limits, cells, enterprise requirements and tenant-aware operations.