Fair queuing for multi-tenant job queues: an interactive simulator

When many tenants share one pool of workers, a single first-in, first-out queue lets whoever submits the most work take everything: one customer’s bulk import delays every other customer’s jobs for minutes. Fair queuing keeps a queue per tenant and serves them in turn, so each tenant with work waiting gets an equal share of the workers, and a tenant alone gets them all.

The playground shares 8 workers between three tenants. Pick a scenario, compare the schedulers in the table, and click one to see who held the workers and how long jobs waited.

FIFO, caps and fair queuing

When tenant A drops 600 jobs at once, a FIFO queue runs them before anything B and C submit afterwards, so their one-second jobs wait up to 74 seconds. Fair queuing serves the three in turn: B and C are served almost at once, and A still gets every worker they do not use, finishing at 160 seconds instead of 85. The total work done is the same; only who waits changes.

A per-tenant concurrency cap looks similar and is simpler, but it has two weaknesses. It is not work-conserving: when A is alone, the cap still holds it to 4 workers while 4 sit idle, doubling its finish time. And it does not protect anyone when several tenants are at their cap at once: with A and B both flooding, their caps add up to all 8 workers and C waits up to 74 seconds anyway.

• Weights. Deficit or weighted round robin gives paying tiers a bigger share without starving anyone.
• Job sizes. Fairness by job count favours tenants with big jobs; schedulers such as DRR charge each tenant for the time or bytes it used instead.
• Many tenants. A queue per tenant is cheap in memory, but a scheduler scanning thousands of queues needs an active list of tenants that have work, as in the code below.
• Where it applies. Job runners, API gateways, database connection pools, network links: Linux’s fq_codel and Kafka’s quotas both rest on the same idea.

A fair queue in code

from collections import deque

class FairQueue:
    """One queue per tenant, served round robin; deficit round robin adds weights."""
    def __init__(self, weights=None):
        self.queues, self.order = {}, deque()
        self.weights, self.credit = weights or {}, {}

    def put(self, tenant, job):
        if tenant not in self.queues:
            self.queues[tenant] = deque()
            self.order.append(tenant)
            self.credit[tenant] = 0
        self.queues[tenant].append(job)

    def get(self):
        while self.order:
            tenant = self.order[0]
            if not self.queues[tenant]:                 # nothing waiting: drop out of the rotation
                self.order.popleft(); del self.queues[tenant]; continue
            if self.credit[tenant] <= 0:                # turn used up: top up, go to the back
                self.credit[tenant] += self.weights.get(tenant, 1)
                self.order.rotate(-1); continue
            self.credit[tenant] -= 1
            return self.queues[tenant].popleft()
        return None

In a system design interview

Mention it whenever a shared queue serves many customers: “Jobs go into a queue per tenant, and workers pull from tenants round robin, weighted by plan, so one tenant’s backfill of a million jobs cannot delay anyone else’s. A tenant alone still gets the whole fleet. Per-tenant rate limits sit at the API, and a tenant’s backlog size is a metric we alert on.” Expect the follow-ups: priorities within a tenant, fairness across a distributed set of workers, and what happens when one tenant’s jobs are much longer than others’.

Read the guide to multi-tenancy or background jobs, see shuffle sharding and bulkheads for isolation, or watch a strong candidate design a job scheduler.

Questions people ask