Graceful Shutdown in Containers: SIGTERM, Draining and Timeouts

The short answer

Every deploy, scale-down or node drain stops containers. If the application dies instantly, in-flight requests fail with errors and background jobs are cut in half. A graceful shutdown means:

1. the signal actually reaches your process;
2. the process stops accepting new connections and jobs;
3. it finishes what is in flight, closes database connections and flushes buffers;
4. it exits before the platform loses patience and sends SIGKILL.

How Docker and Kubernetes stop containers

Docker: docker stop sends SIGTERM (or the signal set by STOPSIGNAL) to the container's PID 1, waits for a timeout (10 seconds by default, configurable with --stop-timeout or -t) and then sends SIGKILL.

Kubernetes does more, and partly in parallel:

• the pod is marked Terminating and is removed from Service endpoints;
• the preStop hook runs, if defined;
• then the container receives SIGTERM;
• if it is still alive when terminationGracePeriodSeconds (30 by default) runs out, it gets SIGKILL. The preStop hook counts towards this period.

The important detail: endpoint removal propagates to kube-proxy, ingress controllers and load balancers asynchronously. For a few moments after SIGTERM, traffic can still arrive. An app that closes its listener immediately will reject those requests.

PID 1 and signal forwarding

The process with PID 1 in a container behaves differently: the kernel does not apply default signal actions to it. If PID 1 has no SIGTERM handler, the signal is simply ignored and the container waits for SIGKILL.

Typical traps:

• Shell form CMD npm start runs /bin/sh -c, which becomes PID 1 and does not forward signals to your app.
• npm, yarn and similar wrappers may not pass signals reliably. Run the runtime directly.
• Entrypoint scripts must end with exec "$@" so the app replaces the shell.

# exec form: node itself is PID 1
CMD ["node", "server.js"]

If the process cannot handle signals or spawns children, use a tiny init: docker run --init or tini as the entrypoint. It forwards signals and reaps zombie processes.

Handling SIGTERM in code

Node.js: stop the HTTP server, wait for open requests, then close resources. A hard timeout protects against hanging keep-alive connections.

process.on("SIGTERM", () => {
  server.close(async () => {
    await db.end();
    process.exit(0);
  });
  setTimeout(() => process.exit(1), 25000).unref();
});

Python: servers like Gunicorn and Uvicorn already handle SIGTERM gracefully; make sure they are PID 1. For your own worker loop, set a flag and finish the current job:

import signal

stopping = False

def on_sigterm(signum, frame):
    global stopping
    stopping = True

signal.signal(signal.SIGTERM, on_sigterm)

while not stopping:
    process_next_job()
close_connections()

Go: http.Server.Shutdown stops listeners and waits for active requests until the context expires.

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()

go srv.ListenAndServe()
<-ctx.Done()

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
srv.Shutdown(shutdownCtx)

preStop and timeouts in Kubernetes

To cover the endpoint propagation race, add a short delay before SIGTERM:

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "5"]

exec needs a sleep binary in the image; newer Kubernetes versions also offer a built-in sleep action for preStop. Build a time budget: preStop delay + longest normal request + cleanup must fit inside terminationGracePeriodSeconds, and your in-app hard timeout should fire before Kubernetes sends SIGKILL.

Checklist

• Exec form in CMD/ENTRYPOINT, exec in shell scripts, or --init.
• SIGTERM handler that stops intake, drains and exits with code 0.
• Readiness probe starts failing once shutdown begins.
• preStop delay for endpoint propagation.
• Grace period longer than the drain time.
• Queue consumers acknowledge messages only after processing, so an interrupted job is redelivered.
• Long-lived connections (WebSocket, streaming) get a close message so clients reconnect elsewhere.