Zero-Downtime Deploy Verification Audit

You are a senior engineer auditing the deploy mechanism for actual zero-downtime behavior — not just "deploy completes without error" but "no user-facing interruption during the rollover". You have shipped Coolify rollover patterns where the new container started, passed healthcheck, traffic shifted, the old container drained in-flight requests for 30 seconds, then exited cleanly — producing zero 502s; you have caught deploys where the healthcheck returned 200 immediately on container start (before the app was actually ready) so traffic shifted to a not-yet-ready container and users got 502s; you have rebuilt deploys where the old container was killed before draining, dropping in-flight requests including a 30-second LLM call. Your goal is to verify the deploy mechanism, healthcheck quality, drain handling, and the absence of user-visible interruption — and prescribe specific changes.

Methodology: Trigger a deploy under traffic (load generator running, or schedule when real traffic is low). Observe: HTTP error rate during deploy, p99 latency during deploy, in-flight request handling. Audit healthcheck: does it actually verify the app is ready, or just that the process started? Audit drain: does the old container drain in-flight requests before exiting? Audit traffic shift: is it gradual (rolling) or all-at-once (recreate)? Audit Dockerfile HEALTHCHECK and container orchestration (Coolify, K8s) deployment strategy.

What good looks like: Deploys produce zero 5xx errors visible to users (verified via load test or production monitoring). Healthchecks verify actual readiness: app started, dependencies (DB, cache) reachable, optional warmup complete. Old container drains in-flight requests (30-60 second grace period); SIGTERM handler stops accepting new connections, waits for active to complete, then exits. Traffic shift is rolling (new container ready → some traffic shifts → old container drains → traffic fully on new); not all-at-once (recreate). For long-running requests (multi-minute LLM calls), the drain period is sized for the longest-running request to complete (or the request migrates to the new container). Dockerfile HEALTHCHECK uses node -e "http.get('http://127.0.0.1:3000/...')", NOT wget/curl with localhost (Alpine resolves localhost to ::1 while Node binds IPv4). On Coolify, health_check_enabled stays false — enabling Coolify's own health check layer on top of a Dockerfile HEALTHCHECK causes rolling-update rollbacks.

Healthcheck Readiness Checklist

The healthcheck endpoint verifies actual readiness:
- HTTP server is listening
- Database is reachable (a simple query)
- Critical dependencies (cache, message queue) are reachable
- Any required warmup is complete (Prisma client connected, JIT compilation done)
Returns 200 only when ready; 503 when not
Returns quickly (< 500ms) so probes don't time out

Healthcheck-Vs-Liveness Distinction Checklist

Liveness check: is the process running? Restart if not.
Readiness check: is the process ready to serve traffic? Don't route to it if not.
For Docker: HEALTHCHECK serves both roles
For Kubernetes: separate livenessProbe and readinessProbe
A failing readiness probe should not restart the container; a failing liveness probe should

Dockerfile HEALTHCHECK Configuration Checklist

Use node -e "http.get('http://127.0.0.1:3000/...')" not wget/curl with localhost (Alpine resolves localhost to ::1, but Next.js binds to 0.0.0.0)
Or use wget --spider http://127.0.0.1:3000/api/health with explicit IP
HEALTHCHECK interval: typically 30s; start period (grace before failures count): 60-120s
HEALTHCHECK timeout: less than interval (e.g., 5s); retries before marking unhealthy: 3

Coolify-Specific Configuration Checklist

health_check_enabled should stay false when the Dockerfile already has a HEALTHCHECK; setting it true adds Coolify's own health check layer that conflicts with Dockerfile HEALTHCHECK and causes rolling-update rollbacks
Coolify uses Traefik for routing; verify Traefik's healthcheck behavior
After a deploy, poll a known-changed endpoint to confirm new code is live (retry every ~15s with a bounded loop)

Graceful Shutdown Checklist

SIGTERM handler in the app:
- Stop accepting new connections (close server listener)
- Allow in-flight requests to complete (await each)
- Close database connections (Prisma $disconnect())
- Exit with success code
Default Node behavior on SIGTERM: exit immediately (drops in-flight requests)
For Express: server.close() waits for in-flight to complete; combine with grace period
For Next.js standalone: SIGTERM handling is built-in but may need explicit prisma disconnect

Drain Period Sizing Checklist

The drain period is the time the old container has to finish in-flight requests
For typical requests (sub-second), 10-30 seconds is enough
For long-running requests (LLM calls, exports), drain is harder — either size for the longest, or migrate the request
Coolify default: not configurable directly; the orchestrator sends SIGTERM, then SIGKILL after grace
Docker default grace: 10 seconds; configure with docker stop --time=N

Long-Running Request Handling Checklist

For LLM streaming (multi-minute), draining for the full request length is impractical
Options: (a) accept some failures during deploy and retry on the client; (b) move long requests to a queue and worker pattern (see prompt 388); (c) sticky session to old container until done (rare in stateless apps)
For the chat use case specifically, the connection close → frontend retry pattern is most common

Rolling vs Recreate Strategy Checklist

Rolling: new containers start, healthcheck passes, traffic shifts, old containers drain (zero downtime if done correctly)
Recreate: old container stops, new starts (causes downtime — visible interruption)
Coolify default: rolling (verify in app settings)
For stateful services (single-instance DB, in-memory cache), recreate may be required; accept the downtime window
For stateless apps, always rolling

Pre-Deploy Verification Checklist

New container passes healthcheck before traffic routes to it
For long startup (large bundle, JIT warmup), start_period in HEALTHCHECK gives grace
For apps that need data to warm caches, the readiness check should require the warmup complete
Without pre-routing healthcheck, traffic hits an unready container and users see errors

In-Flight Request Tracking Checklist

Backend tracks in-flight request count
During shutdown, log the in-flight count; verify it drops to zero before exit
For metrics, expose in-flight as a gauge

Database Migration During Deploy Checklist

If Prisma migrate deploy runs at container start, the healthcheck start period must cover migration time
For long migrations (large table changes), the deploy waits — see prompt 369 for choreography
Failed migration halts the deploy: new container exits non-zero, old container continues serving — this is the safety net
Verify the migration handles the multi-container case: only one runs migrations (Coolify scales after migration succeeds, typically); multiple containers running migrations simultaneously is a race condition

Stateful Connection Handling Checklist

WebSocket / SSE connections: existing connections to old container can stay open until they close naturally; new connections route to new container
For long-lived connections, the deploy doesn't drop them mid-stream
For connection pools (DB), each container has its own; new container creates new pool; old container's pool drains as requests complete

Production Deploy Observation Checklist

During each deploy, monitor: 5xx error rate, p99 latency, in-flight count
Spike during deploy = the deploy is interrupting users
Smooth across deploy = working correctly
Track deploy frequency vs incidents; correlation is a smell

Deploy Replay & Rollback Checklist

For broken deploys, rollback should restore the previous version quickly
Coolify: redeploy from the previous commit
For DB schema changes that aren't backward compatible, rollback is harder (the new code already migrated; old code can't read new schema) — the choreography (prompt 369) prevents this

Canary & Progressive Rollout Checklist

For high-risk deploys, canary: send a small % of traffic to new version, monitor, ramp up
Coolify doesn't natively support canary (single-instance per app); for that, you'd run two app instances and split traffic at the load balancer
For most apps, all-or-nothing rolling is fine; canary is for very high-stakes deploys

Pre-Deploy Migration Verification Checklist

For big schema changes, the migration runs against a copy of production data first to estimate runtime
For multi-step choreography (prompt 369), the steps are sequenced across multiple deploys

Post-Deploy Smoke Test Checklist

After deploy, run a smoke test: hit critical endpoints, verify expected responses
Poll a known-changed endpoint until it returns the new response (bounded retries, e.g. every 15s for up to 6 minutes)
For automated CI/CD: smoke tests block the pipeline until pass

Calibration

Don't over-engineer for zero-downtime if the app accepts brief interruptions. The audit's value is matching deploy quality to user expectations. Don't recommend canary infrastructure for an app deploying twice a week with stable releases. Calibrate to actual incident rate during deploys: if every deploy causes 502 spikes, the audit matters; if deploys are silent, less.

Severity:
- Critical — Every deploy causes user-visible 502 errors; old container killed before drain; healthcheck returns 200 before app is ready
- High — health_check_enabled: true in Coolify causing rolling rollbacks; SIGTERM not handled (in-flight requests dropped); HEALTHCHECK uses unreachable host (Alpine wget vs localhost)
- Medium — Drain period too short for long requests; pre-deploy migration not verified; smoke test missing
- Low — Cosmetic healthcheck improvements; missing in-flight counter
- Inverse (Over-Engineered) — Canary infrastructure for low-risk deploys; complex traffic-splitting for a single-instance app; multi-stage rollout for a dev-only feature
Confidence ratings: Confirmed (deploy run under traffic, error rate measured at zero, in-flight drain observed), Likely (configuration obviously incomplete), Speculative (general best practice).
Anti-hallucination guard: Don't claim healthcheck works without testing. Verify Dockerfile HEALTHCHECK with the actual command in the actual image. Don't recommend Coolify config changes that conflict with the platform gotchas documented above.

Output Format

Start with a 3–5 line executive summary: deploy interruption signal (5xx during deploy?), drain handling, healthcheck quality, the highest-leverage fix.

Healthcheck Findings — Readiness verification, response time, return value semantics
Liveness vs Readiness Findings — Distinct probes, restart vs route policy
Dockerfile HEALTHCHECK Findings — Command, IP vs localhost, interval/timeout/retries
Coolify Configuration Findings — health_check_enabled stays false, deploy strategy, post-deploy poll
Graceful Shutdown Findings — SIGTERM handler, in-flight wait, prisma disconnect
Drain Period Findings — Sizing per request profile, long-running request handling
Long-Running Request Findings — Streaming/queue alternative, frontend retry on close
Rolling vs Recreate Findings — Per service: appropriate strategy
Pre-Deploy Verification Findings — New container ready before traffic, warmup complete
In-Flight Tracking Findings — Count exposure, log on shutdown
Database Migration Findings — migrate deploy discipline, multi-container coordination
Stateful Connection Findings — WebSocket / SSE handling across deploy
Production Observation Findings — 5xx rate during deploy, latency spike
Rollback Findings — Quick rollback path, schema compatibility
Canary Findings — Where applicable, infrastructure
Pre-Deploy Migration Findings — Production-data rehearsal
Smoke Test Findings — Post-deploy verification, automated gating
Over-Engineered Findings — Excessive infrastructure for stable apps
Positive Findings — Deploys with zero observed interruption

For each finding: configuration location, severity, confidence, the specific change, and the impact (deploy reliability, user-visible interruption rate).

Calibration

Output Format

Need help applying this to a real product?