How do you find, every second, which of ten million jobs should run now?

A next_run_at column indexed with the partition key, scanned by the owning dispatcher once a second for a 60-second look-ahead with a bounded batch size, feeding a hierarchical timer wheel that fires with millisecond accuracy. The claim and the advance of next_run_at happen in one transaction, so a crash loses an enqueue rather than duplicating an occurrence. Creation-time jitter spreads jobs within their minute where the expression allows.

Is exactly-once achievable, and if not, what do you promise instead?

At-least-once delivery with an execution id of job_id:scheduled_time:attempt, passed to the target as an idempotency key and stable across reaps and requeues. Leases prevent concurrent duplicates in the common case; the key handles the uncommon one. Documentation says at-least-once plainly and tells tenants how to deduplicate, and the duplicate-delivery rate is monitored so a regression in the lease path is visible.

How does the system know a worker is still alive and still owns a job?

Per-execution leases with a 60-second TTL renewed every 20 seconds while the job runs, held in a small fast store. A reaper requeues executions whose lease expired without a recorded outcome, keeping the execution id so the delivery is recognisably a repeat. Leases are advisory rather than authoritative — the target's idempotency is what makes a stale-lease duplicate harmless — and the number of reaps per execution is capped so a poison job cannot cycle forever.

How do you keep a customer with a million midnight jobs from delaying everyone else?

Queue partitions per tenant, drained by workers with weighted fair queueing and a per-tenant cap on concurrent executions. Dispatcher partitions are also assigned by a hash of tenant and job id so one tenant cannot dominate a single dispatcher's scan. Quotas on total jobs and creation rate limit the blast radius at the source, and very large tenants can be given dedicated worker pools as a paid tier.

A job scheduled every five minutes takes seven. What should happen?

A concurrency policy per job, defaulting to forbid: a due occurrence whose previous run is still active is skipped and recorded as skipped rather than queued. Allow permits overlap for jobs that declare themselves safe, and replace cancels the running attempt and starts the new one for jobs where only the latest matters. Enforcement uses a per-job lock in the lease store, and a run that exceeds its maximum duration is terminated so a hang does not silently stop the job forever.

System design interview question

Design a Distributed Job Scheduler

Run millions of cron jobs and one-off tasks on time across a fleet, exactly once each, and keep going when a worker dies halfway through one.

Last updated 2026-09-22. Difficulty: medium. Patterns: scheduling, leases, idempotency, queues, fault-tolerance. Reported at Amazon, Google, Stripe, Uber, Datadog, Atlassian.

Sit this as an AI interview and be asked it one question at a time; Study shows every answer, and Practice hides them until you have produced your own.

Functional requirements

Schedule recurring jobs. Cron expressions with a time zone, because "09:00 every weekday" means different instants in March depending on where the customer is.
Schedule one-off jobs at a future time. Send this reminder in 40 days, expire this hold in 10 minutes. The same machinery, with a schedule of one.
Run each due execution once. The hard requirement. Charging a card twice because a worker retried is the failure this system exists to prevent.
Retry failures with backoff. A bounded number of attempts with exponential backoff and jitter, then a dead-letter queue and an alert rather than an infinite loop.
Respect concurrency and ordering per job. A job that overruns its interval must not start a second copy unless it says it may, and some jobs must not run concurrently with each other.
Observability and control. History per job, why the last run failed, and the ability to pause, resume, trigger now and backfill a missed window.
Out of scope. The execution environment itself (containers, capacity), workflow orchestration with dependencies between jobs, and data pipeline lineage.

Non-functional requirements

Scheduling accuracy (within 1 s of due time at p99). Late is usually fine, early never is. "Run at 09:00" that fires at 08:59 breaks business logic that assumes the day has started.
Scale (10 M scheduled jobs, 100 k executions/s peak). The peak is at the top of the minute and especially the top of the hour — a self-inflicted thundering herd.
Delivery (at least once, with idempotent execution). Exactly-once execution is not achievable across a network. Say so, then make duplicates harmless.
Durability (no scheduled job lost). A job accepted and then forgotten is invisible until someone notices a report never arrived. Silent loss is the worst failure mode here.
Availability (99.99 %). A scheduler outage is not a crash, it is a delay — but a delay long enough becomes a missed window, which for some jobs is a lost day.
Isolation (no tenant starves another). One customer scheduling a million jobs at midnight must not delay everyone else's.
Long jobs (up to 6 hours). The execution model must tolerate a task that outlives any single lease renewal interval.

Back-of-envelope estimates

Scheduled jobs: ~10 M. 100 k tenants × ~100 jobs each = 10 M definitions. At ~1 KB each that is 10 GB of definitions: small. The volume is in executions, not definitions.
Executions per day: ~1 B. 10 M jobs averaging ~100 runs a day (a mix of per-minute, hourly and daily schedules) = ~1 B executions/day ≈ 12 k/s average.
Peak burst: ~100 k/s. Everything cron lands on the minute, and a large share on the hour. If 30 % of daily executions are hourly jobs clustered in the first second of the hour, the instantaneous rate is ~100 k/s — roughly eight times the average.
Due-index scan size: ~100 k rows/s. A scan every second for jobs due in the next window returns the burst-sized set: ~100 k rows at the top of the hour, a few hundred otherwise. The index must make that a range scan, never a table scan.
Execution record storage: ~300 GB/day. 1 B executions × ~300 B (job id, attempt, start, end, status, error) = ~300 GB/day, retained 30 days hot then rolled up. Execution history is far larger than the jobs themselves.
Workers needed: ~2 000. Average execution 200 ms of work; 12 k/s × 0.2 s = 2 400 concurrent executions. At ~20 concurrent per worker that is ~120 workers for the average and ~1 000 for the peak — so the queue absorbs the burst rather than the fleet.
Lease renewal traffic: ~2 k/s. ~2 400 concurrent executions renewing a lease every 30 s = ~80/s normally, rising with long jobs. Trivial, which is what makes leases affordable as the ownership mechanism.

Components

Tenant service: The application that registers jobs and receives their executions, usually as an HTTP callback or a message. It owns the idempotency of its own handler, which is the other half of the correctness story.
Schedule API: Create, update, pause, delete, trigger now and backfill. Validates the cron expression and time zone, computes the first due time, and writes the definition.
Job store (sharded SQL, index on next_run_at): One row per job: schedule, payload, target, retry policy, concurrency policy, and next_run_at. The index on (shard, next_run_at) is the whole design — every dispatcher query is a range scan on it.
Dispatcher (one owner per partition): Wakes each second, claims jobs due in the next window from its partitions, enqueues an execution for each, and advances next_run_at. Partition ownership is leased, so exactly one dispatcher claims any given job.
Near-term timer wheel (in-memory, next 60 s): Jobs due within the next minute held in a hierarchical timer wheel so firing is O(1) per tick rather than a database query per second. The store remains the authority; this is a latency optimisation that can be rebuilt at any time.
Execution queue (partitioned by tenant): Absorbs the top-of-the-hour burst and decouples deciding from doing. Partitioned per tenant with weighted draining so one tenant's midnight flood cannot starve everyone else.
Worker pool: Takes an execution, acquires a lease, invokes the target, renews the lease while it runs, and reports the outcome. Stateless and autoscaled on queue depth.
Lease store (Redis / etcd with TTL): execution_id → worker, with a short TTL renewed by a heartbeat. A worker that dies stops renewing and its lease expires, which is how work is recovered without anyone having to detect the death directly.
Job target: The thing actually run: an HTTP endpoint, a queue message, a container. Given an idempotency key derived from the execution so a duplicate delivery is recognisable as one.
Execution history (time-partitioned): One row per attempt with status, timing and error. The answer to "did it run and what happened", and the source of the per-job dashboards.
Retry scheduler: Turns a failure into a future execution with exponential backoff and jitter, up to the policy limit, then dead-letters. Reuses the same due-time mechanism rather than inventing a second one.
Reaper: Finds executions whose lease expired without an outcome and requeues them. The only thing standing between a worker dying and a job silently never running.
Monitoring & alerts: Lateness distribution, failure rate per tenant, dead-letter depth, and missed-window alerts. A scheduler that is quietly an hour late looks healthy from the inside unless lateness is measured explicitly.

User flows

A daily job fires and runs. The happy path, and the shape of the whole system: decide, enqueue, lease, invoke, record, reschedule.
1. A tenant registers "every weekday at 09:00 Europe/London".
2. The dispatcher that owns this partition scans for jobs due in the next window.
3. Jobs due within the minute are loaded into the in-memory timer wheel.
4. At 08:00:00 UTC the timer fires and an execution is enqueued.
5. next_run_at is advanced to the next occurrence in the same transaction as the claim.
6. A worker takes it, acquires a lease, and invokes the target.
7. The outcome is recorded and the lease released.
Midnight: everything is due at once. The scale-breaking case, and it is self-inflicted: cron expressions cluster on round numbers.
1. At 00:00:00 several hundred thousand jobs become due in the same second.
2. Dispatchers claim in bounded batches rather than in one enormous scan.
3. Executions are enqueued as fast as they are claimed; the queue absorbs the rest.
4. Workers drain at their sustainable rate; jobs run late, not lost.
5. Per-tenant queue partitions stop one tenant's flood from starving the rest.
6. New schedules are jittered at creation to flatten future peaks.
A worker dies halfway through a job. The failure path. Nobody detects the death directly; the lease simply stops being renewed.
1. A worker acquires a 60-second lease and starts a job that takes minutes.
2. The worker is terminated mid-execution.
3. The lease expires because nothing renews it.
4. The reaper finds an expired lease with no recorded outcome and requeues the execution.
5. A new worker picks it up and invokes the target with the same key.
6. Repeated reaping is capped and the execution is dead-lettered with an alert.
A job fails and is retried. Retries reuse the scheduling machinery rather than inventing a parallel one.
1. The target returns a 500 and the worker records the failure.
2. The retry scheduler computes the next attempt time with exponential backoff and jitter.
3. The attempt is written as a future execution in the same due-time index.
4. The attempt runs with the same execution id and an incremented attempt number.
5. After the policy limit the execution is dead-lettered and the tenant is alerted.
6. A tenant-wide failure rate above a threshold pauses that tenant's retries.
The clocks go back. The correctness case people forget. "09:00 every day" is ambiguous twice a year and the answer must be deliberate.
1. A job is defined as 01:30 daily in a zone that moves its clocks.
2. The next occurrence is computed in local time, then converted to UTC.
3. A skipped local time is resolved by a documented rule.
4. A repeated local time fires once, not twice.
5. A time-zone database update shifts future occurrences and they are recomputed.

Deep dives

Finding the jobs that are due. How do you find, every second, which of ten million jobs should run now? A next_run_at column indexed with the partition key, scanned by the owning dispatcher once a second for a 60-second look-ahead with a bounded batch size, feeding a hierarchical timer wheel that fires with millisecond accuracy. The claim and the advance of next_run_at happen in one transaction, so a crash loses an enqueue rather than duplicating an occurrence. Creation-time jitter spreads jobs within their minute where the expression allows.
Running a job exactly once. Is exactly-once achievable, and if not, what do you promise instead? At-least-once delivery with an execution id of job_id:scheduled_time:attempt, passed to the target as an idempotency key and stable across reaps and requeues. Leases prevent concurrent duplicates in the common case; the key handles the uncommon one. Documentation says at-least-once plainly and tells tenants how to deduplicate, and the duplicate-delivery rate is monitored so a regression in the lease path is visible.
Ownership without a failure detector. How does the system know a worker is still alive and still owns a job? Per-execution leases with a 60-second TTL renewed every 20 seconds while the job runs, held in a small fast store. A reaper requeues executions whose lease expired without a recorded outcome, keeping the execution id so the delivery is recognisably a repeat. Leases are advisory rather than authoritative — the target's idempotency is what makes a stale-lease duplicate harmless — and the number of reaps per execution is capped so a poison job cannot cycle forever.
Stopping one tenant starving the rest. How do you keep a customer with a million midnight jobs from delaying everyone else? Queue partitions per tenant, drained by workers with weighted fair queueing and a per-tenant cap on concurrent executions. Dispatcher partitions are also assigned by a hash of tenant and job id so one tenant cannot dominate a single dispatcher's scan. Quotas on total jobs and creation rate limit the blast radius at the source, and very large tenants can be given dedicated worker pools as a paid tier.
When a run outlives its interval. A job scheduled every five minutes takes seven. What should happen? A concurrency policy per job, defaulting to forbid: a due occurrence whose previous run is still active is skipped and recorded as skipped rather than queued. Allow permits overlap for jobs that declare themselves safe, and replace cancels the running attempt and starts the new one for jobs where only the latest matters. Enforcement uses a per-job lock in the lease store, and a run that exceeds its maximum duration is terminated so a hang does not silently stop the job forever.