Design AWS Step Functions

• Define a state machine: A JSON definition of states (Task, Choice, Wait, Parallel, Map, Pass, Succeed, Fail) and their transitions. Publishing creates an immutable version; an alias can split new starts between two versions, and a running execution keeps the version it started on.
• Start an execution, idempotently: StartExecution with a caller-chosen name, unique per state machine. The same name and input while that execution is running returns it; after it closes, or with different input, the call fails with ExecutionAlreadyExists. A name is reusable 90 days after close.
• Run task states: Call a function, any cloud API or an HTTPS endpoint directly; start a job and wait for it to finish; or offer work to the customer’s own activity workers, which long-poll for it.
• Retry, catch and time out: Declarative per-state retry rules (errors matched, interval, backoff rate, max attempts, jitter) and catch clauses, plus a timeout and an optional heartbeat on every task. Failure handling is configuration, not customer code.
• Wait: for a time, or for a callback: A Wait state that sleeps for seconds or months, and a task that pauses until someone calls back with its token: a human approval, a payment webhook, a job on another platform.
• Fan out: Parallel branches that join, and a Map over a list or over a million objects in a bucket, with a concurrency limit and a tolerated failure rate.
• Stop, inspect and redrive: Stop any execution. Read its full event history (every state, input, output, error and duration) for 90 days after it closes. Redrive a failed execution from the state that failed, within 14 days, without rerunning the steps that succeeded.
• Out of scope: The visual designer, IAM policy evaluation, billing, the JSONPath / JSONata data-transformation language, and the customers’ own function runtimes: we call them, we do not run them.

• Exactly-once state transitions (never forked, skipped or applied twice): The engine’s whole promise, and the requirement that drives the hardest trade-off: every transition must be durable before anything acts on it, which puts a replicated commit on every step and makes the steps of one execution strictly serial. Side effects inside customer code get a separate, weaker guarantee, and the design must say precisely which.
• Scale (2 B transitions/day · ~100 k/s peak): Across all customers in a region, about 200 M executions a day. Throughput must grow by adding hosts and partitions, never by making one execution or one partition faster.
• Long-lived executions (up to 1 year · ~50 M open): Most executions are asleep at any instant: waiting on a timer, an approval or a worker. They must live in storage and wake on an event; holding them in memory is not an option.
• Transition overhead (p99 < 100 ms per state): For short workflows the engine’s overhead per step is the customer’s latency. A ten-state checkout flow at 500 ms per transition would be unusable on a request path.
• Timer accuracy (fire within 1 s at p99): Waits, timeouts, retry backoffs and heartbeat deadlines must fire on time even when a million are due in the same second, and must never be lost: a lost timeout leaves an execution asleep forever.
• Availability and durability (99.99 % · nothing lost after a 200): Once StartExecution or SendTaskSuccess returns, the event is replicated across availability zones. Losing a host, or a whole zone, delays executions by seconds and loses nothing. Other services build on this one, so it must be more available than they are.
• Tenant isolation (per-account token buckets · 1 M open cap): Default quotas such as 300 starts/s (burst 1 300) and 5 000 transitions/s per account, enforced at the front door before work enters a shard, so one account’s runaway loop is throttled instead of slowing its neighbours.

• State transitions per second: ~23 k avg · ~100 k peak. 2 B transitions a day ÷ 86 400 s ≈ 23 k/s on average. Business hours, month-end batches and retry storms push peaks to about 4× ≈ 100 k/s. Design for the peak.
• Execution starts per second: ~2.3 k/s. An average execution makes ~10 transitions, so 2 B ÷ 10 = 200 M starts a day ≈ 2.3 k/s, around 10 k/s at peak.
• Executions open at once: ~50 M. Little’s law: 2.3 k starts/s × an average lifetime of ~6 hours (21 600 s) ≈ 50 M open. Most executions finish in seconds; a tail waiting days for approvals or timers dominates the average. Nearly all are idle at any instant, so this is a storage problem, not a compute problem.
• History written per day: ~2 TB · ~180 TB retained. A transition appends ~3 events (state entered, task scheduled, task succeeded, …) of ~300 B: about 1 KB. 2 B × 1 KB = 2 TB/day. At 90 days’ retention after close that is ~180 TB, ~540 TB with three replicas, which is why closed histories leave the hot store within days.
• Database row writes at peak: ~500 k rows/s. Each transition writes ~3 history events, the mutable-state row and ~1 task or timer row: about 5 rows, ~1.5 KB. 100 k transitions/s × 5 = 500 k row writes/s, ~150 MB/s before replication. Spread over thousands of partitions that is ordinary; what limits one execution is commit latency (~5 ms), not bandwidth.
• Shards needed at peak: ~2 000 busy, 16 k fixed. Commits within one shard partition are serialised; at ~5 ms each a shard tops out near 200 transitions/s, and we run it at a quarter of that: 100 k/s ÷ 50/s = 2 000 busy shards. A fixed 16 384 gives 8× headroom and small units to move between hosts.
• Timers: ~12 k created/s · ~50 M pending. Every task gets a timeout timer, nearly always cancelled within seconds when the task finishes: ~1 B task states a day ≈ 12 k timer creations/s, plus Wait states and retry backoffs. Pending timers track open executions, since nearly every sleeping execution is waiting on a deadline: ~50 M, about 3 000 per shard, of which only the next few minutes’ worth is in memory.
• Hosts to own the shards: ~160. 16 384 shards ÷ ~100 per host = ~164 engine hosts, each committing ~600 transitions/s at peak. Shards are the unit of failover: a dead host’s 100 small shards move to many different hosts and each recovers in seconds, rather than one giant partition failing over.

• Customer app: Starts and stops executions, sends task callbacks and reads history through the signed public API. It chooses the execution name, which makes starting idempotent, and it holds the task tokens it is handed until the person or system it waits on replies.
• Activity workers: The customer’s own processes for work the engine cannot call: on-premises jobs, GPU boxes, legacy systems. They long-poll for tasks by activity name, heartbeat while they work and report with the task token. They never know or care which shard a task came from.
• Frontend API (stateless · per-account quotas): Authenticates, enforces per-account token buckets and payload limits, then routes: execution calls to the shard owner (by hashing the execution, or by the shard inside a task token), activity polls to task matching. It holds no execution state, so any instance serves any call and the fleet scales by adding boxes.
• Definitions (versioned · immutable): State machine definitions with their published versions and aliases. A version is immutable and an execution records the version it started on, so editing a definition never changes the path of a running execution. Read on every transition and written rarely, so every engine host caches it.
• Shard membership (leases + range ids): Maps each of 16 384 shards to the engine host that owns it, through leases of about 10 s held in a small consensus-backed store. Taking a shard bumps its range id, which the history store checks on every write, so a former owner that missed its lease expiry is fenced out.
• Execution engine (owns shards): The heart of the system. For every shard it owns, it applies events to executions one at a time, evaluates the state machine and commits the result: new history events, the updated mutable state and any task and timer rows, made visible by one conditional write. Hot executions stay cached so a short workflow never rereads its state, and nothing acts on a transition until the write lands.
• History store (partitioned · conditional writes): Per execution, an append-only event history and a small mutable-state row: current state, definition version, pending tasks with their attempts, timers, next event id. Mutable state, outbox and timer rows share the shard’s partition, so the commit is one single-partition write conditional on the range id. History events are far too big for that partition, so they go to their own table just before the commit and only count once it lands.
• Task outbox (per shard · same commit): Work the engine has committed to: call this function, offer this activity, start this map run, publish this status event. Written in the same commit as the transition (a transactional outbox) and dispatched from memory straight after it; the rows exist so a new owner can replay them after a crash. Acknowledged in bulk by advancing an ack level rather than deleting row by row.
• Task matching (partitioned by activity): Activity task queues, partitioned by activity name, because a worker asks for “resize-image” work, not for a shard. A task that arrives while a worker is polling is handed straight to it without being stored (sync match); otherwise it waits in a durable backlog. Before handing a task out, matching asks the owning shard to record the attempt as started, which drops duplicates and tasks the execution no longer wants.
• Timer queue (per shard · by fire time): Durable timers (Wait states, task timeouts, retry backoffs, heartbeat deadlines) stored per shard, ordered by fire time and written in the transition’s commit. The owner keeps the next few minutes in an in-memory timing wheel and loads more as time advances, so fifty million sleeping executions cost disk, not memory. A fired timer is just another event, applied only if the execution still expects it.
• Service integrations: Makes the call for each started attempt: a function, any cloud API, an HTTPS endpoint, or a job it then watches to completion. Calls run with the execution’s role, never the engine’s, and carry the execution and state names that customers use as an idempotency key. Results, and errors classified as retryable or not, go back to the owner as events.
• Functions & services: What a workflow actually does: functions, queues, databases, containers, third-party APIs. A response can be lost after the work happened, so a retried attempt can repeat a side effect; customers make side effects idempotent with a key that does not change across retries.
• Map run coordinator: Runs a distributed Map: reads the item source in pages, batches items and starts child executions with deterministic names, at most MaxConcurrency (up to 10 000) at a time. It counts outcomes against the tolerated failure rate, checkpoints its progress and writes results to object storage, so the parent’s history records one map run, never a million iterations.
• Payload store (object storage): Item lists and results for distributed maps, payloads too large for the 256 KB limit, and the compacted histories of closed executions. Keeps the history store small and hot: it holds open executions and the last few days of closed ones, not 180 TB.
• Status events: Execution started, succeeded, failed, timed out or aborted, published for customers’ alarms, other workflows and the map run coordinator. Emitted from committed transitions through the outbox, so an event is never sent for a transition that did not happen and never missed for one that did.

1. Start an execution and run a task: The core path: a start that is idempotent, a transition that is durable before anything acts on it, an attempt handed out exactly once, and a result that advances the machine exactly once.
   1. The customer starts an execution with a name and an input.
   2. The frontend hashes the execution to a shard and finds the host that owns it.
   3. The owner resolves the definition version and commits the start.
   4. The owner records the attempt as started and hands it to the integration that calls the customer’s function.
   5. The result comes back as an event; the owner records it and decides the next state.
   6. At a terminal state the execution closes and a status event is published.
2. Pause for an approval, then a worker: A task that waits days for a callback: the execution sleeps in storage, wakes on a token or a deadline, rejects stale replies, then hands the next step to a polling worker without the worker knowing about shards.
   1. The workflow reaches an approval task, and the integration sends a message carrying a task token.
   2. The engine writes a timeout timer, and the execution leaves memory.
   3. Two days later the approver clicks, and the customer’s app sends the token back.
   4. The owner checks the token still matches the pending task and records the result.
   5. The timeout is cancelled, and the next state, an activity, is offered to workers.
   6. A worker’s long poll picks the task up, and the owner records the attempt as started.
   7. The worker heartbeats while it works, then reports success with the token.
3. A Map over a million objects: The scale case: one execution fans out to thousands of child executions without exploding its own history, flattening a downstream database or starving other customers.
   1. The workflow reaches a distributed Map state pointing at a bucket prefix.
   2. The coordinator reads the object list in pages and batches items.
   3. It starts child executions through the normal start path, up to the concurrency limit.
   4. Children run on their own shards, and their status events drive the coordinator’s counters.
   5. Results go to the payload store, and the coordinator completes the parent’s Map state like a callback.
4. An engine host dies mid-transition: The failure case: shards move in seconds, a paused owner is fenced, no transition is lost or applied twice, and a call whose outcome is unknown becomes an explicit, retryable error instead of a silent duplicate.
   1. An engine host stops renewing its shard leases.
   2. Other hosts acquire its 100 shards, each bumping the shard’s range id.
   3. The old host wakes up and tries to commit; the store rejects it.
   4. The new owner reloads outbox rows and due timers and dispatches what was never started.
   5. A result for a call the old host dispatched is delivered to the new owner.
   6. If the integration host itself died mid-call, the attempt’s deadline settles it.
5. A task fails, retries, is caught, is redriven: Retries are timers, not sleeping threads; the catch path is just another transition; and because the history keeps every result, a failed execution can resume from the failed step.
   1. The function returns a throttling error.
   2. The engine records the failure and writes a backoff timer instead of retrying at once.
   3. The timer fires and the task is dispatched again as the next attempt.
   4. Attempts run out, and the Catch clause routes the execution to a fallback state.
   5. If nothing catches the error, the execution fails and alarms fire.
   6. The customer raises the downstream limit and redrives the execution from the failed state.
   7. A few days after it finally closes, the history is compacted into the payload store.

Ask: Why keep a full event history? Why not just store each execution’s current state in a row?

Good answers name: Append-only history plus a mutable-state row, with one conditional write as the commit point, Current state only, overwritten in place, History only, rebuilding state by replaying it on every event (pure event sourcing), Replay customer workflow code over the history (Temporal / Cadence model).

Our pick: Each execution has an append-only history (ExecutionStarted, TaskScheduled, TaskStarted, TaskSucceeded, TimerFired, …) and a mutable-state row holding the current state, the definition version, pending tasks with their attempts, pending timers and the next event id. A transition first appends its new events to a history table partitioned by execution and keyed by (execution, event id), then makes them count with one conditional write in the shard’s partition: update the mutable state and insert any outbox and timer rows, if the range id still matches and the mutable state is still at the version it read. Events from an attempt whose commit failed sit past the recorded next event id; readers ignore them and the retry overwrites them. With a store that has multi-partition transactions, all of it can be one transaction; what matters is the single commit point. Because definitions are declarative, the engine never re-runs customer code to find out where it was: the mutable state says so. AWS has not published Step Functions’ internals; Cadence at Uber and its successor Temporal, whose designs descend from Amazon’s Simple Workflow Service, use exactly this split.

1. Your history is 180 TB. Is it really in the same partition as the mutable state?
   No, and it should not be: 180 TB over 16 384 shards is about 11 GB per partition, far past what one partition should hold. Only the small rows share the shard partition: mutable state, outbox, timers and the shard’s range id. History events go to their own table keyed by execution and event id, written just before the commit. If the commit fails, those events sit beyond the recorded next event id and are ignored, and the retry rewrites the same ids. The conditional write in the shard partition stays the single commit point.
2. Which database would you use, and why?
   Anything partitioned that offers conditional writes within a partition and range scans in key order: Cassandra or Scylla with lightweight transactions, DynamoDB with conditional writes, or sharded MySQL / Postgres. Conditional writes give fencing and optimistic concurrency inside the commit itself; ordered range scans give the outbox and the timer queue (rows keyed by fire time) for free. Cadence runs on Cassandra or MySQL, and Temporal adds Postgres. A single relational primary does not survive 500 k row writes a second, and a log like Kafka alone cannot cheaply answer “what state is execution X in?”.
3. A customer has a loop that runs 100 000 iterations in one execution. What happens?
   It hits the 25 000-event cap and fails, deliberately. A history that large makes its shard hot, makes every reload slow and is unreadable in the console. The answer is to restructure: a Map with child executions, or, as AWS documents, have the execution start a fresh copy of itself with its current state as input before it reaches the cap, then end.
4. Why is the definition version stored on the execution?
   So editing a state machine never changes the path of an execution already running. A year-long execution that started on version 3 keeps resolving states against version 3, even if version 9 renamed or removed them. New versions reach new starts through an alias, which can also split traffic (say 90 / 10) to canary a change and shift back if the new version’s failure rate rises.
5. How do you serve history, and does all 180 TB need to be hot?
   No. Open executions and those closed in the last few days are read often; the rest almost never. A few days after close, a job compacts an execution’s history into one compressed object in object storage and deletes the rows, leaving a pointer. The console pages history in event order from whichever tier holds it, and because events are immutable, pages are cacheable forever. The hot store then holds roughly the open executions plus a week of closes, a few tens of terabytes.

Ask: Two engine hosts receive results for the same execution at the same time. How do you stop them both advancing it?

Good answers name: Fixed shards (16 384), one owner per shard by lease, fencing by range id on every write, A distributed lock per execution for each transition, Optimistic concurrency only: compare-and-set on the mutable state version, any host may process.

Our pick: Hash every execution (account, state machine, name) to one of 16 384 shards, fixed for the life of the cluster. A membership service grants each shard to one engine host on a lease of about 10 s, renewed every 3 s; taking a shard increments its range id with a conditional write in the store, and every write for any execution in the shard is conditional on that range id. Within a shard, the owner processes each execution’s events one at a time from an in-memory queue, so concurrent results for the same execution are applied in order, and each commit also checks the mutable state’s version as a second line of defence. Frontends cache the shard map and follow “not my shard” redirects. A crashed host’s ~100 shards move to others within one lease period; a planned move (deploys, rebalancing) is a graceful handoff that costs milliseconds. Temporal works this way, and also fixes its shard count when a cluster is created.

1. Why not just rely on the lease? It expires before the new owner takes over.
   Because the old owner’s clock and scheduling cannot be trusted: a 30-second garbage-collection pause, a stalled VM or a network partition can leave it believing it holds a lease that expired long ago. Only a check at the storage layer, at commit time, is safe. The range id is that check; the lease only decides who should be trying.
2. How do you pick 16 384 shards rather than 1 000 or 1 million?
   Enough that peak load spreads with each shard far below its serial-commit ceiling (2 000 busy shards needed at 100 k/s), and enough that one host’s share is small to move. Few enough that per-shard overhead (leases, timer loops, in-memory queues) stays modest across ~160 hosts. Changing the count means re-sharding a live cluster, so pick with an order of magnitude of headroom and grow beyond it by adding clusters, not shards.
3. One account starts 5 M executions in a minute. Who suffers?
   Ideally only that account. Its start bucket (refill 300/s, burst 1 300 by default) rejects the excess with ThrottlingException before anything is written, and its transition bucket caps how fast its open executions can move. What does get through is hashed across all shards, so it is spread thin rather than piled on a few hosts. Inside each shard the owner serves ready executions from per-account fair queues, so one account’s burst waits behind itself. At the largest scale the region is split into cells, independent copies of the stack serving subsets of accounts, so even a pathological definition that crashes hosts reaches one cell.
4. One execution receives a burst of 10 000 events (a huge Parallel finishing). Is its shard in trouble?
   Its events are applied serially, so that execution slows down, but other executions on the shard keep moving because the owner interleaves work across executions. The engine can also apply several queued events for one execution in a single commit. The real protection is the per-execution event cap and steering customers to Map with child executions, which spreads the work over many shards.
5. How do you deploy new engine code to 160 hosts without customers noticing?
   Graceful shard handoff. Before restarting, a host stops taking new events for one shard at a time, finishes in-flight commits, persists its ack levels and releases the lease, and the new owner acquires it at once instead of waiting for expiry. Each shard pauses for tens of milliseconds, and the rollout moves a few hosts at a time, watching transition latency and error rates before continuing. Only a crash costs the full lease timeout, and fencing keeps even that safe.

Ask: How do you fire millions of timers on time when some are due in a second and others in eleven months?

Good answers name: Timer rows in each shard ordered by fire time; the owner loads the next window into memory, A separate global timer service, Message-queue delayed delivery, Hierarchical timing wheel in memory only.

Our pick: Each timer is a row in its shard’s partition keyed by (fire_time, timer_id), written or deleted in the same commit as the transition that creates or cancels it. The shard owner loads timers due in the next few minutes into an in-memory timing wheel, advances a read cursor as time passes, and on firing appends TimerFired or TaskTimedOut to the execution, but only after checking the mutable state still expects that timer, so a cancelled timer already in memory is harmless. Heartbeat deadlines are lazy: a heartbeat updates the last-heartbeat time in the mutable state, and the single timer row re-arms itself on firing if a heartbeat arrived since, instead of being rewritten on every heartbeat. Retry backoffs get jitter. Each shard reports timer lag, the age of its oldest due but unfired timer, and a growing lag moves the shard or adds hosts.

1. A million executions all have a Wait until midnight. What happens at 00:00:00?
   Those timers are spread across all 16 384 shards by execution hash, about 60 per shard, so each owner fires about 60 events in the first second, well within its capacity. The wake-ups then hit customers’ functions all at once, which is the customer’s burst to manage, through their own concurrency limits and our per-account transition bucket. Midnight bursts are a good reason to hash executions rather than group them by account.
2. The shard owner crashes with timers loaded in memory. Are any lost?
   No. Memory held copies; the rows are still in the store. The new owner rebuilds its window from rows due before now plus the next few minutes and fires anything overdue at once. A timer might fire twice across the handover, and the state check makes the second firing a no-op.
3. Why not compute timeouts lazily, checking only when an event arrives?
   Because a task that never responds generates no event: the timeout is the only thing that will ever wake the execution. Lazy checks work for limits you enforce only on activity, not for deadlines that must fire into silence.
4. Whose clock decides a timer is due?
   The shard owner’s, which is fine at a one-second target as long as hosts keep synchronised clocks, typically within milliseconds. A timer firing a few milliseconds early or late is harmless because firing only appends an event that the state check validates. Correctness never depends on clocks agreeing between hosts: fencing uses range ids, not time.
5. How accurate can timers be, and how do you measure it?
   Within about a second at p99 is realistic, because the window is loaded ahead of time and firing is in memory. Measure the gap between scheduled and actual fire time per shard as a histogram and alert on p99 lateness. Lateness concentrated in a few shards is a hot-shard problem; lateness everywhere is a capacity problem.

Ask: You promise “exactly once”. Can a customer’s function run twice?

Good answers name: Exactly-once transitions; each attempt started at most once; unknown outcomes become errors that Retry rules handle, Resend from the outbox until acknowledged (at-least-once per attempt), Mark the task complete before calling, with no deadline, Express: run the whole execution in memory, no per-step durability.

Our pick: Standard executions commit each transition, including the outbox row for the next task, in one commit. Before an attempt leaves the engine, the owner records it as started; a replayed outbox row or a duplicate matching task finds it already started and is dropped, so each attempt is handed out at most once. Every attempt has a deadline: the state’s timeout or heartbeat, capped by the integration’s own maximum (15 minutes for a function, 60 s for an HTTP task). If the outcome is lost, the deadline records States.Timeout and the state’s Retry rules decide whether the next attempt runs. Results are accepted only for the exact attempt still pending, so late and duplicate results are no-ops. Customers who retry calls with side effects pass a key that does not change across attempts, Execution.Id plus State.Name from the context object, as the idempotency key to their database or payment provider. Express trades all of this for throughput, and the product makes the choice at creation time: a state machine’s type cannot change later.

1. The function charged a card, then its response was lost. What does the customer see?
   The attempt’s deadline fires and records States.Timeout. If the state retries on timeouts, attempt 2 calls the function with the same Execution.Id and State.Name; if the function passed those to the payment provider as its idempotency key, the provider returns the original charge and the execution moves on with one charge. If it built the key from the retry count, or used none, the card is charged twice. With no Retry rule the error goes to Catch, where a person or a reconciliation step decides, which is the right default for money.
2. Why record “started” instead of just resending until acknowledged?
   Because resending creates duplicates the customer did not configure and cannot see: the history would show one attempt while the target saw two. Recording the start costs one commit per task, about 5 ms, and in exchange every repeat is an explicit attempt with its own history event. Temporal does the same: an activity attempt is recorded as started when a worker picks it up, and duplicate copies of the task are dropped there.
3. Why not a two-phase commit with the target?
   Functions, HTTP APIs and most cloud services cannot take part in one, and those that could would hold locks across a call that may last fifteen minutes, blocking everything else touching that data. Idempotency keys give the useful part, no duplicate effect, with no coordination at all.
4. How is Express cheaper? Same engine, fewer writes?
   It skips the durable history entirely: the execution runs in memory on one host, history goes asynchronously to logs, and a host failure reruns the whole execution (asynchronous Express, at-least-once) or fails the call (synchronous Express, at-most-once). That removes the commits, which are most of the cost, so Express is priced per request and duration instead of per transition and has no transition rate limit. The price is no callbacks, no activities, no redrive and a five-minute ceiling.
5. Could the engine guarantee exactly-once effects for its own integrations, like writing to a queue?
   Only where the target offers idempotent writes the engine can key, such as a de-duplication id on a FIFO queue or a conditional put on a database. Where it can, it derives the key from the execution and state. Where the target cannot de-duplicate, no engine can promise more than at-least-once once retries are on.

Ask: An external system replies days later with “done”, and customers run their own workers. How does work reach them and results get back, safely?

Good answers name: Signed, self-describing task tokens, plus activity queues partitioned by name with a start check at the owner, Random token with a token → execution index, Workers poll the shards directly, Signal the execution by id instead of answering a task.

Our pick: When a callback task starts, the engine mints a token holding the shard, execution id, scheduled event id, attempt and a key id, signed with a service key, and hands it to the integration (an email, a queue message, an HTTP call). The execution writes a timeout timer and, if configured, a heartbeat deadline, then sleeps in storage. SendTaskSuccess, SendTaskFailure or SendTaskHeartbeat at any frontend verifies the signature and the caller’s account, routes on the embedded shard, and the owner applies the result only if that event id and attempt are still pending; a second click or a late reply gets a clear error and changes nothing. Activity tasks go from the outbox to task matching, partitioned by activity name. A task that arrives while a worker’s long poll is waiting is handed over in memory (sync match); otherwise it is stored in a backlog. Before returning any task, matching asks the owning shard to record the attempt as started, which drops duplicates and tasks for executions that were stopped. Temporal’s matching service works this way, and Step Functions’ activity API (60-second long polls, heartbeats, tokens) has the same shape.

1. The approver never clicks. How long can the execution wait, and what wakes it?
   Until the task’s timeout, bounded by the one-year execution limit. The timeout timer is the only thing that will ever wake it, so it must exist from the moment the task starts; a callback task with no timeout set is a classic way to strand executions. On firing it records States.Timeout, which Retry or Catch rules handle like any other error.
2. What are heartbeats for, if there is already a timeout?
   A timeout bounds the whole task; a heartbeat bounds silence. A worker processing a six-hour job sends a heartbeat every few minutes; if it dies an hour in, the heartbeat deadline fires after a few minutes of silence instead of waiting five more hours for the timeout. And a heartbeat that comes back TaskTimedOut tells a slow worker to stop, because a retry may already be running.
3. A token leaks into a log. What is the risk?
   Anyone with the token and credentials allowed to call the API in that account could complete that one task, once, while it is pending. The credential check limits the blast radius to the account’s own principals, and the token is useless the moment the task completes or times out. If that is not enough, the integration can wrap the token in the customer’s own signed link and keep the raw token server-side.
4. How do you rotate the signing key without breaking tokens issued a month ago?
   Carry a key id in the token and accept a set of recent keys, retiring a key only after the longest possible task lifetime, a year, has passed since it last signed. New tokens always use the newest key, and a compromised key can be revoked early at the cost of failing the tasks it signed.
5. One activity name has 10 000 workers polling. Is its matching partition a hotspot?
   It can be, so a busy activity queue is split into several partitions, say 4 to 16, each on a different matching host, and polls and tasks are spread across them at random. A poll that finds its partition empty can forward to a sibling before parking, so a task does not wait in one partition while a worker idles on another. Most queues stay at one partition; only hot ones are split. Temporal does the same with task queue partitions.

Ask: A customer wants to process a million objects in one workflow. Why not run a loop inside the execution?

Good answers name: Distributed Map: a coordinator starts batched child executions with a concurrency cap and a failure threshold, Inline Map inside the parent execution, The customer starts a million executions themselves.

Our pick: A distributed Map state hands the item source (a bucket prefix, a manifest, a CSV) to a map run coordinator, and the parent waits on it like any callback task. The coordinator reads the source in pages, groups items into batches, and starts child executions named deterministically from the map run and batch index, so a restarted coordinator cannot double-start a batch. It keeps at most MaxConcurrency children running (up to 10 000), counts outcomes from child status events, stops early if failures pass the tolerated percentage, and writes results and a manifest to object storage. Children are ordinary executions on their own shards under the account’s quotas, so other customers are unaffected, and short idempotent items can run as Express children, which are far cheaper per item. A failed map run can be redriven, rerunning only the failed and unstarted children.

1. The coordinator restarts halfway through a million items. How does it resume?
   Its progress (listing position, batches started, counts) is checkpointed in its own durable record. On restart it reloads the checkpoint and re-issues starts for batches that may not have begun; deterministic child names make those starts idempotent, returning the existing child if it already ran.
2. Each child calls the same downstream database. How do you stop the map from flattening it?
   MaxConcurrency is the throttle: 1 000 concurrent children means at most 1 000 concurrent callers, and the bounded start rate ramps up gradually. Retry with backoff and jitter on throttling errors handles the rest. Without a concurrency cap, a distributed map is an excellent load-testing tool aimed at your own database.
3. Why batch items into children instead of one child per item?
   Per-execution overhead (a start, several transitions, a close) is fixed, so for small items it dominates the work. Batching 100 items per child cuts the engine overhead per item a hundredfold and turns a million executions into ten thousand.
4. Standard or Express children: which would you choose?
   Express for short, idempotent per-item work, such as transforming a file and writing the result under a deterministic key: there are no per-transition commits, so a million items cost a fraction as much, and a rare rerun of a batch is harmless. Standard when an item calls something non-idempotent, needs a callback or a human, or can run past five minutes. The coordinator does not care: both are executions it starts and counts.
5. How does the customer see progress on a run that takes six hours?
   The map run has its own record with counters (pending, running, succeeded, failed), updated from child status events and surfaced in the console and as metrics. They can stop the run, which stops new children and lets running ones finish or cancels them, and later redrive only the failures.