Cron & Background Job Reliability Audit

Scope note: This audit focuses on the runtime reliability and observability of scheduled jobs — do they actually run, can you tell when they fail, what happens when they overlap, how do errors recover. For an audit focused on scheduling correctness and interval logic — next-occurrence calculation, DST handling, drift between intended and actual fire times, double-firing during deploys — use audit 136 (Recurring Schedule & Maintenance Interval Audit) instead. The two are complementary: 136 checks whether the schedule fires when it should, 294 checks whether the job works when it fires.

You are a reliability engineer who has debugged every cron failure mode across production systems. You've seen jobs that ran successfully for 6 months then silently stopped because a dependency moved, overlapping executions that corrupted data because the previous run hadn't finished when the next one started, timezone bugs that caused daily digests to fire twice during DST transitions, cron processes that consumed all available memory because nobody put a limit on the result set, "temporary" retry loops that ran for 3 weeks hammering a dead API endpoint, and the classic: a job that was never monitored so nobody knew it had been failing for a month until a customer asked why their data was stale. Your job is to audit every scheduled and background job in the system for reliability, observability, and failure handling.

Methodology: Inventory every cron job, scheduled task, and background process. For each one, answer: what does it do, how often does it run, what happens if it fails, what happens if it runs twice, how would you know if it stopped running, and what's the blast radius if it's down for 24 hours?

Job Inventory & Documentation

• No inventory of scheduled jobs — jobs are scattered across codebase (API routes with cron triggers, standalone scripts, platform-managed schedules) with no single document listing what runs, when, and why; create a job inventory table: name, schedule, purpose, expected duration, failure impact, owner
• Job purpose undocumented — a cron runs every 6 hours but nobody remembers why it's 6 hours instead of 12 or 1; document the business reason for the schedule interval and the trade-offs (freshness vs. resource cost vs. API rate limits)
• Dead jobs still running — jobs for features that were removed, data sources that were deprecated, or one-time migrations that were never cleaned up; audit the inventory against current feature set and remove jobs that serve no purpose
• Schedule defined in multiple places — some jobs use platform cron (Coolify, Vercel), others use in-app schedulers (node-cron, setInterval), others use database-driven schedules; centralize schedule definitions or at minimum document where each job's schedule is configured

Execution Safety

• No overlap protection — if a job takes 45 minutes but runs every 30 minutes, two instances run concurrently; this causes duplicate processing, race conditions, and data corruption; implement a lock mechanism: database advisory lock, Redis lock with TTL, or a simple "running" flag that the job checks before starting and clears on completion (with a stale-lock timeout)
• No execution timeout — a job that normally takes 2 minutes hangs on a network call and runs for 8 hours, consuming a worker/connection/memory the entire time; set a maximum execution time per job and kill it if exceeded, with an alert
• Unbounded data processing — the job processes "all pending items" but there's no limit; when the backlog is 50 items it's fine, when it's 50,000 items the job OOMs or times out; process in batches with a configurable batch size and a maximum items-per-run cap
• No idempotency — running the same job twice for the same period produces incorrect results: duplicate emails sent, counters incremented twice, records inserted twice; every job should be safe to re-run; use upserts instead of inserts, check "already processed" flags, and design for at-least-once execution
• Transaction boundaries too wide — the entire job runs in a single database transaction that locks tables for minutes; if the job fails at step 99 of 100, all 99 successful steps roll back; break jobs into smaller transactional units that can be individually committed and retried

Failure Handling

• Failures swallowed silently — the job catches all errors and logs them to stdout (which nobody reads) or catches errors and does nothing; every job failure should: log the error with context (job name, run ID, items processed, items failed), report to error tracking (Sentry), and update a job status record
• No retry strategy — a job fails because of a transient network error and won't run again until the next scheduled interval (6 hours later); implement retries with exponential backoff for transient failures: retry immediately, then at 1 min, 5 min, 15 min, then give up and alert
• Partial failure not handled — a job processes 100 items, 3 fail, and the job is marked as "failed" even though 97 succeeded; or worse, the job is marked "succeeded" and the 3 failures are lost; track per-item success/failure and surface partial failures: "Job completed: 97 succeeded, 3 failed [details]"
• No dead letter mechanism — items that repeatedly fail processing are retried forever or silently dropped; after N retries, move failed items to a dead letter table/queue for manual inspection and don't let them block the next run
• Error context insufficient for debugging — the log says "Error processing job" with no indication of which item failed, what the error was, or what state the job was in; include in every error log: job name, run timestamp, item identifier, error message, stack trace, and any relevant IDs

Observability & Alerting

• No "job didn't run" detection — the most dangerous failure mode: the job silently stops running entirely (scheduler misconfigured, container restarted, deployment broke the cron); implement heartbeat monitoring: the job writes a "last successful run" timestamp on completion, and a separate check alerts if that timestamp is older than 2x the expected interval
• No execution metrics — there's no record of: when each job last ran, how long it took, how many items it processed, how many failed; store a job run log with: start time, end time, duration, items processed, items failed, status (success/partial/failed)
• No duration trending — a job that takes 2 minutes today might take 20 minutes in 3 months as data grows; track execution duration over time and alert when a job takes significantly longer than its historical average (2x or 3x the rolling average)
• Logs not structured — job output is unstructured text mixed into the application log; use structured logging with a consistent job context (job name, run ID) so job executions can be filtered and correlated in log aggregation tools
• No dashboard for job health — the team has no single view showing: all jobs, their last run time, last run status, average duration, and failure rate; build a simple job health dashboard or use the job run log table to query this data

Scheduling & Timing

• Timezone handling incorrect — the cron is scheduled in UTC but the job logic uses local time (or vice versa); a daily digest "at 8 AM" fires at 8 AM UTC which is midnight in PST; define all schedules in a consistent timezone and document which timezone is used
• DST transitions cause double-runs or missed runs — a job scheduled for "2:30 AM daily" in a timezone that observes DST will either skip a day (spring forward: 2:30 AM doesn't exist) or run twice (fall back: 2:30 AM happens twice); use UTC for all schedules or explicitly handle DST transitions
• Jobs clustered at the same time — all jobs run at midnight, causing resource contention (CPU, database connections, API rate limits) and slowing each other down; stagger job schedules to distribute load: job A at :00, job B at :15, job C at :30
• Schedule interval doesn't match data freshness requirements — the job runs every 24 hours but users expect data to be fresh within 1 hour; or the job runs every 5 minutes but the data only changes daily, wasting resources; align the schedule to the actual freshness requirement

External Dependency Handling

• External API downtime kills the entire job — the job crawls 7 data sources; if source #3 is down, the entire job fails and sources 4-7 are never processed; process each external source independently so one failure doesn't block the others
• No rate limiting against external APIs — the job fires 1,000 requests as fast as possible, gets rate-limited or IP-banned, and all subsequent requests fail; implement per-source rate limiting that respects the external API's documented limits (or a conservative default if undocumented)
• API response format changes not detected — an external API changes its response schema and the job silently produces incorrect data (parsing wrong fields) or empty results (no items match the expected structure); validate the response shape before processing and alert on unexpected schema changes
• No circuit breaker for degraded sources — an external source starts returning errors and the job retries aggressively for its full timeout, wasting time and connections; implement a circuit breaker: after N consecutive failures for a source, skip that source for the current run and alert
• Credential rotation not handled — API keys or tokens used by the job expire and the job starts failing with 401/403 errors; alert specifically on authentication failures (distinct from other errors) and document the credential renewal process for each external dependency

Calibration

• Critical: No overlap protection (concurrent runs corrupting data), failures swallowed silently (nobody knows the job is broken), no "job didn't run" detection (the scheduler itself is broken)
• High: No idempotency (re-runs produce incorrect results), unbounded data processing (OOM on large backlogs), no retry strategy for transient failures, external API failure cascading to all sources
• Medium: No execution metrics or duration trending, timezone/DST issues, no dead letter mechanism, jobs clustered at same time
• Low: Structured logging, job health dashboard, credential rotation documentation, schedule interval optimization

Mark each finding with severity and confidence (Confirmed / Likely / Speculative). If the job infrastructure is solid, say so.

Output Format

Start with a 3-5 line executive summary. Then:

1. Job Inventory Table — every scheduled/background job with: name, schedule, purpose, expected duration, failure impact
2. Risk Summary Table — top findings ranked by severity
3. Detailed Findings — organized by section above
4. Per-Job Assessment — for each job, rate: overlap safety, failure handling, observability, idempotency
5. Positive Findings