Zero-Downtime Deploy Verification Audit

You are a senior engineer auditing the deploy mechanism for actual zero-downtime behavior — not just "deploy completes without error" but "no user-facing interruption during the rollover". You have shipped rolling deploys on a self-hosted container platform where the new container started, passed healthcheck, traffic shifted, the old container drained in-flight requests for 30 seconds, then exited cleanly — producing zero 502s; you have caught deploys where the healthcheck returned 200 immediately on container start (before the app was actually ready) so traffic shifted to a not-yet-ready container and users got 502s; you have rebuilt deploys where the old container was killed before draining, dropping in-flight requests including a 30-second LLM call. Your goal is to verify the deploy mechanism, healthcheck quality, drain handling, and the absence of user-visible interruption — and prescribe specific changes.

Methodology: Trigger a deploy under traffic (load generator running, or schedule when real traffic is low). Observe: HTTP error rate during deploy, p99 latency during deploy, in-flight request handling. Audit healthcheck: does it actually verify the app is ready, or just that the process started? Audit drain: does the old container drain in-flight requests before exiting? Audit traffic shift: is it gradual (rolling) or all-at-once (recreate)? Audit Dockerfile HEALTHCHECK and container orchestration (Kubernetes, ECS, Nomad, or a self-hosted PaaS such as Coolify, Dokku or Kamal) deployment strategy.

Scope: Whether a deploy interrupts users: readiness versus liveness checks, the image's health check, platform settings, graceful shutdown, drain time, long requests, migrations during rollover, and rollback. Any deploy platform.

Mode: Default is observe a deploy under light synthetic traffic on staging, then report. Production deploys are observed, not triggered, in this pass.

Adapt to this codebase first: The commands, paths and library names in this prompt are written against one common stack as a worked example, not as a requirement. Before running anything, identify what this repository actually uses (languages, frameworks, data layer, package manager, hosting, directory layout) from its manifests, lockfiles and config, then translate every file glob, path and API name to the equivalent here. A command that returns nothing because it was written for another stack is not a clean result: rewrite it for this stack and run it again. If the technology this prompt is named for is not used here, say that first and apply only the checks that carry over to what is used instead. Where a check has no equivalent, say so in the report instead of skipping it silently.

Run these first:

# 1. Health and readiness endpoints
grep -rnEi 'healthz?|readyz?|liveness|readiness|HEALTHCHECK' $(git ls-files | grep -vE 'lock$|\.min\.|\.md$|node_modules/|vendor/') 2>/dev/null | head -15

# 2. Signal handling and drain
grep -rnEi 'SIGTERM|SIGINT|graceful|shutdown|server\.close|drain|stop_grace_period|terminationGracePeriod' $(git ls-files | grep -vE 'lock$|\.min\.|\.md$|node_modules/|vendor/') 2>/dev/null | head -15

# 3. Errors during a deploy window (replace the placeholder, run while deploying)
for i in $(seq 1 120); do curl -s -o /dev/null -w '%{http_code} %{time_total}\n' "https://<staging-host>/api/health"; sleep 1; done | sort | uniq -c | sort -rn | head

What good looks like: Deploys produce zero 5xx errors visible to users (verified via load test or production monitoring). Healthchecks verify actual readiness: app started, dependencies (DB, cache) reachable, optional warmup complete. Old container drains in-flight requests (30-60 second grace period); SIGTERM handler stops accepting new connections, waits for active to complete, then exits. Traffic shift is rolling (new container ready → some traffic shifts → old container drains → traffic fully on new); not all-at-once (recreate). For long-running requests (multi-minute LLM calls), the drain period is sized for the longest-running request to complete (or the request migrates to the new container). Dockerfile HEALTHCHECK uses node -e "http.get('http://127.0.0.1:3000/...')", NOT wget/curl with localhost (Alpine resolves localhost to ::1 while Node binds IPv4). Exactly one layer owns the health check: a platform-level check stacked on a Dockerfile HEALTHCHECK (Coolify's health_check_enabled is one example) can disagree with it and cause rolling-update rollbacks.

Healthcheck Readiness Checklist

The healthcheck endpoint verifies actual readiness:
- HTTP server is listening
- Database is reachable (a simple query)
- Critical dependencies (cache, message queue) are reachable
- Any required warmup is complete (Prisma client connected, JIT compilation done)
Returns 200 only when ready; 503 when not
Returns quickly (< 500ms) so probes don't time out

Healthcheck-Vs-Liveness Distinction Checklist

Liveness check: is the process running? Restart if not.
Readiness check: is the process ready to serve traffic? Don't route to it if not.
For Docker: HEALTHCHECK serves both roles
For Kubernetes: separate livenessProbe and readinessProbe
A failing readiness probe should not restart the container; a failing liveness probe should

Dockerfile HEALTHCHECK Configuration Checklist

Use node -e "http.get('http://127.0.0.1:3000/...')" not wget/curl with localhost (Alpine resolves localhost to ::1, but Next.js binds to 0.0.0.0)
Or use wget --spider http://127.0.0.1:3000/api/health with explicit IP
HEALTHCHECK interval: typically 30s; start period (grace before failures count): 60-120s
HEALTHCHECK timeout: less than interval (e.g., 5s); retries before marking unhealthy: 3

Platform-Specific Configuration Checklist

List every layer that runs a health check (Dockerfile HEALTHCHECK, platform setting, proxy, load balancer) and confirm they agree; a platform-level check stacked on a Dockerfile HEALTHCHECK (on Coolify, health_check_enabled: true) can conflict with it and cause rolling-update rollbacks
Identify the proxy that shifts traffic (Traefik, Caddy, nginx, a cloud load balancer) and verify its own health check and retry behavior during a rollover
After a deploy, poll a known-changed endpoint to confirm new code is live (retry every ~15s with a bounded loop)

Graceful Shutdown Checklist

SIGTERM handler in the app:
- Stop accepting new connections (close server listener)
- Allow in-flight requests to complete (await each)
- Close database connections (Prisma $disconnect())
- Exit with success code
Default Node behavior on SIGTERM: exit immediately (drops in-flight requests)
For Express: server.close() waits for in-flight to complete; combine with grace period
For Next.js standalone: SIGTERM handling is built-in but may need explicit prisma disconnect

Drain Period Sizing Checklist

The drain period is the time the old container has to finish in-flight requests
For typical requests (sub-second), 10-30 seconds is enough
For long-running requests (LLM calls, exports), drain is harder — either size for the longest, or migrate the request
Self-hosted PaaS (Coolify, Dokku, CapRover): often not configurable directly; the platform sends SIGTERM, then SIGKILL after a fixed grace period, so confirm what that period is
Docker default grace: 10 seconds; configure with docker stop --time=N

Long-Running Request Handling Checklist

For LLM streaming (multi-minute), draining for the full request length is impractical
Options: (a) accept some failures during deploy and retry on the client; (b) move long requests to a queue and worker pattern; (c) sticky session to old container until done (rare in stateless apps)
For the chat use case specifically, the connection close → frontend retry pattern is most common

Rolling vs Recreate Strategy Checklist

Rolling: new containers start, healthcheck passes, traffic shifts, old containers drain (zero downtime if done correctly)
Recreate: old container stops, new starts (causes downtime — visible interruption)
Self-hosted PaaS: usually rolling by default (verify in the app's settings rather than assuming)
For stateful services (single-instance DB, in-memory cache), recreate may be required; accept the downtime window
For stateless apps, always rolling

Pre-Deploy Verification Checklist

New container passes healthcheck before traffic routes to it
For long startup (large bundle, JIT warmup), start_period in HEALTHCHECK gives grace
For apps that need data to warm caches, the readiness check should require the warmup complete
Without pre-routing healthcheck, traffic hits an unready container and users see errors

In-Flight Request Tracking Checklist

Backend tracks in-flight request count
During shutdown, log the in-flight count; verify it drops to zero before exit
For metrics, expose in-flight as a gauge

Database Migration During Deploy Checklist

If Prisma migrate deploy runs at container start, the healthcheck start period must cover migration time
For long migrations (large table changes), the deploy waits; use a multi-deploy migration choreography
Failed migration halts the deploy: new container exits non-zero, old container continues serving — this is the safety net
Verify the migration handles the multi-container case: only one runs migrations (most platforms start the new container, run its entrypoint, then shift traffic; confirm the order on yours); multiple containers running migrations simultaneously is a race condition

Stateful Connection Handling Checklist

WebSocket / SSE connections: existing connections to old container can stay open until they close naturally; new connections route to new container
For long-lived connections, the deploy doesn't drop them mid-stream
For connection pools (DB), each container has its own; new container creates new pool; old container's pool drains as requests complete

Production Deploy Observation Checklist

During each deploy, monitor: 5xx error rate, p99 latency, in-flight count
Spike during deploy = the deploy is interrupting users
Smooth across deploy = working correctly
Track deploy frequency vs incidents; correlation is a smell

Deploy Replay & Rollback Checklist

For broken deploys, rollback should restore the previous version quickly
Self-hosted PaaS: redeploy from the previous commit or the previous image tag
For DB schema changes that aren't backward compatible, rollback is harder (the new code already migrated; old code can't read new schema) — a multi-deploy migration choreography prevents this

Canary & Progressive Rollout Checklist

For high-risk deploys, canary: send a small % of traffic to new version, monitor, ramp up
Single-instance platforms (most self-hosted PaaS setups) don't natively support canary; for that, you'd run two app instances and split traffic at the load balancer
For most apps, all-or-nothing rolling is fine; canary is for very high-stakes deploys

Pre-Deploy Migration Verification Checklist

For big schema changes, the migration runs against a copy of production data first to estimate runtime
For multi-step choreography, the steps are sequenced across multiple deploys

Post-Deploy Smoke Test Checklist

After deploy, run a smoke test: hit critical endpoints, verify expected responses
Poll a known-changed endpoint until it returns the new response (bounded retries, e.g. every 15s for up to 6 minutes)
For automated CI/CD: smoke tests block the pipeline until pass

Evidence rules: Interruption is Confirmed only from requests recorded during an observed deploy, with counts by status. Configuration findings cite file and line.

Calibration

Don't over-engineer for zero-downtime if the app accepts brief interruptions. The audit's value is matching deploy quality to user expectations. Don't recommend canary infrastructure for an app deploying twice a week with stable releases. Calibrate to actual incident rate during deploys: if every deploy causes 502 spikes, the audit matters; if deploys are silent, less.

Severity:
- Critical — Every deploy causes user-visible 502 errors; old container killed before drain; healthcheck returns 200 before app is ready
- High — two conflicting health check layers causing rolling rollbacks; SIGTERM not handled (in-flight requests dropped); HEALTHCHECK uses unreachable host (Alpine wget vs localhost)
- Medium — Drain period too short for long requests; pre-deploy migration not verified; smoke test missing
- Low — Cosmetic healthcheck improvements; missing in-flight counter
- Inverse (Over-Engineered) — Canary infrastructure for low-risk deploys; complex traffic-splitting for a single-instance app; multi-stage rollout for a dev-only feature
Confidence ratings: Confirmed (deploy run under traffic, error rate measured at zero, in-flight drain observed), Likely (configuration obviously incomplete), Speculative (general best practice).
Anti-hallucination guard: Don't claim healthcheck works without testing. Verify Dockerfile HEALTHCHECK with the actual command in the actual image. Read the platform's own documentation for its health check, drain and rollover settings before recommending a change to them.

Output Format

Start with a 3–5 line executive summary: deploy interruption signal (5xx during deploy?), drain handling, healthcheck quality, the highest-leverage fix.

Healthcheck Findings — Readiness verification, response time, return value semantics
Liveness vs Readiness Findings — Distinct probes, restart vs route policy
Dockerfile HEALTHCHECK Findings — Command, IP vs localhost, interval/timeout/retries
Platform Configuration Findings — which layer owns the health check, deploy strategy, post-deploy poll
Graceful Shutdown Findings — SIGTERM handler, in-flight wait, prisma disconnect
Drain Period Findings — Sizing per request profile, long-running request handling
Long-Running Request Findings — Streaming/queue alternative, frontend retry on close
Rolling vs Recreate Findings — Per service: appropriate strategy
Pre-Deploy Verification Findings — New container ready before traffic, warmup complete
In-Flight Tracking Findings — Count exposure, log on shutdown
Database Migration Findings — migrate deploy discipline, multi-container coordination
Stateful Connection Findings — WebSocket / SSE handling across deploy
Production Observation Findings — 5xx rate during deploy, latency spike
Rollback Findings — Quick rollback path, schema compatibility
Canary Findings — Where applicable, infrastructure
Pre-Deploy Migration Findings — Production-data rehearsal
Smoke Test Findings — Post-deploy verification, automated gating
Over-Engineered Findings — Excessive infrastructure for stable apps
Positive Findings — Deploys with zero observed interruption

For each finding: configuration location, severity, confidence, the specific change, and the impact (deploy reliability, user-visible interruption rate).

Calibration

Output Format

Want this applied to a live stack?