Graceful shutdown and connection draining: an interactive rolling deploy

Servers are stopped all the time: every deploy, every scale-in, every node that is drained for maintenance. Each stop can fail requests, by cutting off the ones in progress or by receiving new ones after the server has stopped listening. Graceful shutdown avoids both: take the server out of the load balancer, wait until that has reached every router, stop accepting, finish what is in flight, then exit.

The playground runs a rolling deploy of four servers three ways and marks every failed request. Add long uploads and change how long a server may take to drain.

Two ways a stop fails requests

• Cutting off work in progress. Killing the process ends every request it is handling. In the playground that is 69 requests across the deploy, plus 800 more sent to dead servers before the load balancer’s health checks noticed.
• The deregistration race. Even a process that handles SIGTERM correctly, stopping its listener and finishing its requests, fails requests: the load balancer learns that the server is leaving a few seconds later and keeps sending traffic meanwhile. In Kubernetes the pod receives SIGTERM at the same time as it is removed from the service’s endpoints, and proxies and ingress controllers update after a delay. That costs 303 refused requests here.

The recipe

• Leave the load balancer first. Mark the server not ready, or deregister it, before it stops accepting.
• Wait for that to propagate, a few seconds, with a preStop hook that sleeps, while the server keeps serving whatever still arrives.
• Then stop accepting and drain: close the listener, tell keep-alive clients to reconnect elsewhere, finish in-flight requests, flush logs and metrics, and exit.
• Bound the drain. Kubernetes kills the pod after terminationGracePeriodSeconds, 30 by default, so it must cover the wait plus the longest request you want to protect.

In the playground this fails nothing for quick API requests, at the price of a slower deploy, 60 seconds instead of 40. Long requests stretch it further: with uploads of up to 40 seconds and a 30-second drain, 3 uploads are still cut off, and raising the limit to 60 seconds saves them while the deploy takes over three minutes. For really long-lived work, such as websockets or large uploads, make clients reconnect or resume rather than waiting for them.

Graceful shutdown in Kubernetes and Go

# Kubernetes: leave the load balancer, wait, then let the app drain
spec:
  terminationGracePeriodSeconds: 45        # preStop plus the longest drain, with margin
  containers:
  - name: api
    lifecycle:
      preStop:
        exec: { command: ["sleep", "10"] }    # let endpoint removal reach every proxy
    readinessProbe:
      httpGet: { path: /ready, port: 8080 }

// Go: on SIGTERM, stop accepting, finish in-flight requests, then exit
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()
go srv.ListenAndServe()
<-ctx.Done()
drainCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
srv.Shutdown(drainCtx)   // closes listeners and idle connections, waits for active requests

In a system design interview

When you say “zero-downtime deploys”, say how: “Pods roll one at a time. On termination a pod fails its readiness check and its preStop hook sleeps 10 seconds so every proxy stops routing to it; then the server stops accepting, finishes in-flight requests within 30 seconds, and exits. Websocket clients get a close frame and reconnect to another pod, and long uploads are resumable.” Expect the follow-ups: background jobs that are mid-task at shutdown, and connection pools to the database during the deploy.

Read the guide to deployment strategies or Kubernetes basics, see canary deployments, or watch a strong candidate design a payment system.

Questions people ask