Design a Project-to-Contractor Matching System

Match posted projects with qualified, available contractors across a 500 k-person talent pool, and learn from who actually gets hired.

Last updated 2026-09-26. Difficulty: hard. Patterns: recommendations, ranking, marketplace, exploration, search. Reported at Mercor, Upwork, LinkedIn, Indeed, Thumbtack.

Functional requirements

• Project owners post openings with requirements. Skills, seniority, start date, hours per week, duration, timezone window, budget ceiling, and legal eligibility (work authorization, country, background check). Each requirement is marked must-have or nice-to-have.
• Contractors maintain a profile, availability and preferences. Verified skills, rate, weekly hours free, preferred project types, and the maximum number of concurrent engagements they will take.
• Owners get a ranked shortlist per opening. Top 20 candidates, each with reason codes ("4 of 5 required skills verified, 6 h timezone overlap"). This is a recommendation list, not automatic assignment: a human picks.
• Contractors get a feed of recommended projects. The same match, seen from the other side. A match only happens if both sides say yes, so the contractor feed is half of the product.
• Invite, accept and decline, with reasons. Owner invites, contractor accepts or declines within 48 h. Decline reasons ("rate too low", "not interested in the domain") are both a label and a preference update.
• Respect contractor capacity. A contractor at their concurrent-engagement limit must stop receiving invites, and one with several pending invites must not be shown to everyone.
• Learn from outcomes. Shown, invited, accepted, hired, and project success weeks later. The ranker improves from these without only reinforcing whoever it already showed.
• Out of scope. Payments and invoicing, the vetting interview itself, contracts, messaging, and fraud detection. Vetting results arrive as features.

Non-functional requirements

• Scale (2 M contractors · 500 k active · 15 k openings/day). Request rate is small. The difficulty is in what to show, not how fast.
• Shortlist latency (p99 < 800 ms). An owner waits on a page load. Retrieval, scoring 1 k candidates and re-ranking must fit together.
• Eligibility correctness (0 ineligible candidates shown). Showing someone without work authorization for the country, or already at capacity, is a real harm and a legal problem. This is the requirement that drives the hardest tradeoff: hard rules must never be traded off against a learned score.
• Freshness (availability change visible < 1 min). A contractor who just accepted their second project and hit their limit must drop out of shortlists quickly, or owners invite someone who cannot say yes.
• Match quality (invite → accept > 40 % · hire → success > 80 %). Measured per cohort. Accept rate is the two-sided signal; success is the one that matters but arrives weeks late.
• Fair exposure (new contractors get ≥ 5 % of impressions). A marketplace that only shows proven people starves its supply. Concentration is monitored like latency.
• Explainability and audit (every impression reproducible). Both sides need a reason for a recommendation, and an audit must be able to replay why a person was or was not shown. Log model version, features and score with each impression.

Back-of-envelope estimates

• Ranking requests: ~14/s avg · ~55/s peak. 500 k active contractors × 2 feed loads a day = 1 M, plus 200 k owner shortlist loads = 1.2 M/day ÷ 86 400 ≈ 14/s, ×4 at peak. Throughput is not the hard part.
• Candidates after hard filters: ~2 k per opening. 500 k active × ~5 % with the required skills ≈ 25 k; availability, timezone, rate and eligibility keep ~1 in 12 ≈ 2 k. Retrieval hands the top 1 k to the ranker.
• Scoring work: ~55 k scores/s peak. 55 requests/s × 1 k candidates = 55 k model evaluations/s. A gradient-boosted model scores ~100 k rows/s per core, so a few cores. Feature fetches (1 k keys per request) dominate the latency, not the model.
• Embedding index: ~2 GB. 2 M contractors × 256-dim float32 vector = 2 M × 1 KB = 2 GB. One in-memory ANN index per replica, rebuilt nightly and patched on profile change.
• Interaction events: ~25 M/day · ~5 TB/yr. 1.2 M ranking requests × 20 shown = 24 M impressions/day, plus clicks, invites and outcomes ≈ 25 M events × ~500 B (with score, position, propensity, model version) ≈ 12 GB/day ≈ ~5 TB/yr.
• Concurrent engagements: ~450 k. 15 k openings/day × ~30-day average engagement ≈ 450 k live engagements. With a typical limit of 2 each, that needs at least 225 k distinct contractors, 45 % of the active pool. Capacity binds: the same top 5 % cannot fill the market.
• Labels per month: ~450 k hires · success at +30 days. 15 k hires/day × 30 = 450 k hire labels a month against ~720 M impressions: a positive rate of ~0.06 %. Success labels arrive a month later. So the ranker also trains on nearer signals (invite, accept) and weights them.

Components

• Owner app: Where a project owner posts openings, reads the ranked shortlist with reason codes, relaxes constraints when the list is thin, and sends invites.
• Contractor app: Profile, availability and preferences, a feed of recommended projects, and the inbox where invites are accepted or declined with a reason. The contractor side is half the match: a list the contractor would decline is a bad list.
• Email / push (SES · APNs · FCM): Delivers invites and reminders. Invites expire after 48 h, so delivery latency and the reminder at 24 h directly move the accept rate.
• Profile service (skill taxonomy): Owns contractor profiles and project specs. Normalises free-text skills onto a taxonomy ("ReactJS", "React.js" → react) and records which skills are verified by assessment. Separates must-have from nice-to-have on every project requirement at write time, so the matcher never has to guess.
• API gateway (auth · rate limits): Authenticates owners and contractors, routes shortlist and feed requests to the match service and invite actions to the allocation service. Stamps a request id that follows the impression into the logs.
• Core DB (Postgres): Source of truth for contractors, projects, openings, invites, engagements and outcomes. Change data capture streams every row change onto the event log, which feeds both the search index and the warehouse.
• Match service (stateless orchestrator): Runs one request end to end: compile the opening into hard filters, retrieve, score, re-rank for capacity and exposure, attach reason codes, and log the impression with its propensity. Hard constraints are applied here as filters and are never inputs the model can outweigh.
• Exposure & capacity (re-ranker · invites): Takes the ranked list and makes it safe for the marketplace: removes anyone at capacity, demotes contractors with many pending invites, caps daily exposure, and reserves exploration slots. Also owns the invite lifecycle and the capacity holds that go with it.
• Event log (Kafka): Impressions (with position, score, propensity and model version), clicks, invites, responses and CDC changes from the core DB. Keyed by contractor id so a contractor's availability changes apply in order.
• Candidate index (OpenSearch · filters + kNN): One document per contractor: taxonomy skills, availability, timezone, rate, eligibility flags, open capacity and a profile embedding. Serves filtered retrieval: hard constraints as filter clauses, then lexical and vector similarity inside the survivors. Updated from CDC within seconds.
• Ranker (GBDT, multi-task): Scores ~1 k candidates with separate heads for P(owner invites), P(contractor accepts) and P(success | hired), combined into one expected-value score. Returns the per-feature contributions that become reason codes.
• Capacity ledger (Redis · slots + holds): Per contractor: max concurrent engagements, active count, pending invite holds with TTL, and impressions today. Every accept is a compare-and-set here so two projects cannot both take the last slot.
• Warehouse (S3 + Iceberg): Every impression and outcome, joined by impression id, with delayed labels filled in as they arrive (accept in days, success in a month). The training set and the audit trail are the same tables.
• Feature store (online Redis · offline Iceberg): Contractor features (verified skills, response rate, past success, rate percentile), project features and pair features (skill overlap, timezone overlap). Online and offline values come from the same definitions, so training and serving see the same numbers.
• Training & eval (daily · counterfactual): Retrains the ranker from logged impressions weighted by inverse propensity, evaluates new models counterfactually on exploration traffic before any A/B test, and runs the concentration and cohort-exposure monitors.

User flows

1. Owner opens a ranked shortlist. Filter, retrieve, score, re-rank, explain, log. The order of those steps is the design: eligibility comes first and is never part of the learned score.
   1. Owner opens the shortlist for an opening. The request carries the opening id only. Everything the match needs is read server-side, so a client cannot widen its own filters or see contractors it should not.
   2. Match service loads the opening and compiles its must-haves into filters. Must-haves become boolean filter clauses: verified skills ⊇ required, available hours ≥ requested, timezone overlap ≥ 4 h, rate ≤ budget ceiling, eligibility flags for the project's country, open capacity > 0. Nice-to-haves become ranking features instead. The split is decided by the owner at posting time, not inferred by the model.
   3. Candidate index returns the top 1 k eligible contractors by relevance. One query: filters first, which cut 500 k active to ~2 k, then hybrid relevance inside the survivors (BM25 on skills and titles plus kNN on the profile embedding). Filtering before the vector search matters: a vector search that filters afterwards can return 1 k near neighbours of which 5 are eligible.
   4. Ranker scores the 1 k with features and combines its heads into one expected value. Score = P(invite) × P(accept | invite) × E[success | hired]. The accept head is what makes this two-sided: a candidate the owner loves who will decline because the rate is 30 % below their floor is a wasted slot. One batched feature read of 1 k contractor keys plus the project key, ~15 ms.
   5. Exposure & capacity re-ranks the list for the marketplace. Drops anyone whose slots filled since indexing, demotes contractors whose pending invites already exceed their open slots, applies a daily exposure budget, and fills 2 of 20 positions with exploration candidates sampled from lower-ranked eligible contractors. Each position records the probability it was shown there: that propensity is what makes the logs learnable.
   6. Match service attaches reason codes and returns the top 20. Reasons come from the must-have filter (all satisfied, stated plainly) and the top positive feature contributions ("shipped 3 RLHF projects", "8 h overlap with your team"). Never a reason derived from a protected attribute, and never "because others clicked".
   7. The impression is logged with positions, scores, propensities and model version. Also the ids that passed the filters but were not shown, sampled. Without the not-shown set and the propensities, the training data can only say who was chosen from among the people the old model liked.
2. Invite, accept, and the capacity race. A match needs both sides to agree, and a contractor has a finite number of slots. Holds and a compare-and-set keep the capacity promise honest.
   1. Owner invites three contractors from the shortlist. Each invite places a soft hold on the contractor with a 48 h TTL. Holds do not block acceptance elsewhere, but they count toward "pending pressure", which the re-ranker uses to stop showing a contractor who already has five invites and one open slot.
   2. The invite is stored and the contractor is notified. The invite row carries the impression id and the reasons shown to the owner, so the contractor sees why they were picked. A reminder fires at 24 h; unanswered invites expire at 48 h and count as a soft decline.
   3. Contractor accepts.
   4. Allocation commits the engagement only if a slot is still free. Compare-and-set on the ledger: active < max, then increment and release this invite's hold, in one script. Then the engagement row is written. If two projects race for the last slot, exactly one wins and the other gets a clear "no longer available" instead of a silent overbooking.
   5. The capacity change reaches the index, so the contractor stops appearing where they cannot say yes. CDC turns the engagement insert into an index update in seconds. The ledger check at re-rank time covers the gap, so the index being a few seconds stale shows up as a slightly shorter list, never as an ineligible candidate.
   6. Another contractor declines with a reason; the reason is logged as a label and a preference. "Rate too low" updates the contractor's effective rate floor; "not my domain" lowers similar projects in their feed; "no time" should have been caught by availability, so it prompts them to update it. Declines are the cheapest honest signal the system gets.
   7. A contractor who declined for lack of time updates their availability. The decline screen links straight to the availability form, prefilled. Accurate availability is worth more than any model feature: it removes a whole class of invites that could never be accepted. The change reaches the index through the same CDC path as an engagement.
3. A star contractor floods every shortlist. The best-scored profiles win every ranking, collect dozens of invites they cannot accept, and starve everyone else. The fix lives in the re-ranker, not the model.
   1. Monitoring shows the top 1 % of contractors receiving 35 % of impressions and invites. Concentration is tracked daily: impression share of the top 1 %, a Gini coefficient over impressions, and median invites per contractor per open slot. The symptom that owners feel is a falling accept rate, because the people they invite are drowning in invites.
   2. The re-ranker compares each contractor's pending invites with their open slots. Pressure = pending holds ÷ open slots. Above 3, the contractor's score is multiplied down smoothly (for example by 1 / (1 + pressure - 3)). They are not hidden, just no longer shown first to everyone.
   3. A daily exposure budget spreads impressions in proportion to capacity. Budget ≈ k × open slots per day. Past it, the contractor can still appear when they are far ahead of the next candidate, but ties go to someone with room. This turns a winner-take-all ranking into one that respects the fact that one person can only say yes twice.
   4. When the star's slots fill, the capacity change removes them from retrieval entirely. open_slots = 0 is a hard filter, so a fully booked contractor disappears from all shortlists within seconds. Their feed still works; they can see projects starting after their current engagement ends if they set a future availability date.
   5. The quality cost of spreading is measured, not assumed. An A/B test at the opening level compares accept rate, time to hire and 30-day success between the capped and uncapped re-ranker. Typical outcome on two-sided marketplaces: accept rate rises because invites go to people who can say yes, and success barely moves because the second-best candidate is usually nearly as good.
4. No one meets every hard constraint. The honest answer is an empty list with a map of which constraint to relax, not a list that quietly ignores a must-have. Plus the ranker being down.
   1. Retrieval with every must-have returns 3 candidates. Too few to be useful. The match service never silently drops a must-have to fill the list: the owner said it was non-negotiable, and a candidate without US work authorization is not a near miss.
   2. Match service runs count queries with each negotiable must-have relaxed in turn. Only the constraints the owner may relax: budget, hours, timezone, a secondary skill. Legal eligibility and certification requirements are never candidates for relaxation. Counts are cheap (no scoring) and run in parallel, ~6 queries.
   3. Owner sees the 3 matches plus the tradeoffs that would widen the list. "3 match everything. Raising the budget to $150/h adds 48; accepting 2 h of timezone overlap adds 21." The owner decides. This converts a dead end into a product conversation and gives the marketplace a demand signal about what supply is missing.
   4. The thin result is logged as unmet demand for supply planning. Openings with fewer than 10 eligible candidates, grouped by skill and region, tell the supply team whom to recruit. A matching system that only ranks never learns what it cannot match.
   5. Separately, the ranker times out on a request. Budget 250 ms. On timeout the match service falls back to retrieval order with a simple rule score (verified skill overlap, then response rate), still after the hard filters and the capacity re-rank. Eligibility never depends on the model being up.
   6. The fallback impression is logged with its own policy name. model: "fallback-rules" with its own propensities. Training either uses these impressions with the right weights or excludes them; mixing them in as if the main model had chosen them corrupts the counterfactual estimates.
5. Learning from outcomes without exposure bias. Only shown contractors can be hired, so naive training teaches the model to prefer whoever it already showed. Propensities, exploration and delayed labels fix that.
   1. Impressions, responses and engagement outcomes land in the warehouse. Impressions from the match service, invites and responses and engagement completions from CDC. Everything joins on impression_id and contractor_id, so every label points back to the exact list it came from.
   2. Delayed labels fill in as they arrive. Invite within the session, accept within 48 h, hire within ~a week, success at 30 days (completed, owner rating ≥ 4, or extended). Training on success alone would wait a month and see 450 k positives; so each head trains on its own label, and recent impressions are held back until their label window closes rather than counted as negatives.
   3. Training weights each shown example by the inverse of its propensity. An exploration candidate shown with probability 0.02 who gets hired counts 50 times as much as a top-ranked one shown with probability 1. Weights are clipped (e.g. at 20) to control variance. This corrects for "the model only learned about people it chose to show".
   4. A candidate model is evaluated offline on the exploration traffic. Self-normalised inverse propensity scoring estimates how many accepts and hires the new ranking would have produced. Offline NDCG on the logged lists is reported too but not trusted alone, because it rewards reproducing the old model's choices.
   5. Cohort and concentration checks gate the release. Exposure share for new contractors, by region and by rate band, and top-1 % concentration, compared with the current model. A model that wins on accept rate by showing only veterans fails the gate.
   6. The model ships to the ranker in shadow, then to an opening-level A/B. Shadow mode scores live traffic without serving, checking latency and score drift. The A/B randomises by opening, not by user, because two owners competing for the same contractor interfere: a test split by user leaks through shared supply. Primary metric: hires per opening at 14 days; guardrails: accept rate, exposure share, success at 30 days.

Deep dives

Related

• Design a News Feed
• Design Uber
• Design Search Autocomplete
• Run this question for a friend: the interviewer kit