How to monitor background jobs (queues, workers, and cron)

Why background jobs fail silently

Whether it is a cron entry, a queue worker, or an in-process scheduler, background work shares one trait: nothing is watching it in real time. There is no user waiting on the response, so a failure produces no error page and no angry request — just an absence. Absences are hard to notice.

• The process that runs the schedule crashed or was scaled to zero.
• A deploy renamed, moved, or disabled the job.
• The job threw and the error went to a log file nobody reads.
• A dependency (Redis, the database, an API) was briefly unavailable.

The pattern that works everywhere: heartbeats

Instead of trying to reach into every runtime, have each job report out when it succeeds. Give the job a unique URL and ping it at the end of a successful run. A monitor that knows the expected interval alerts you when a ping is late or missing — which is the only reliable way to detect a run that never happened.

async function processJob(data) {
  await doWork(data);
}

// after a successful cycle, check in
await processJob(payload);
await fetch('https://cronmint.com/ping/YOUR-TOKEN').catch(() => {});

Per-job pings vs a liveness ping

For scheduled jobs, ping once per successful run and set the monitor interval to the schedule. For always-on workers with no fixed cadence, add a low-frequency liveness ping (say once a minute) so a dead worker is detected even between jobs. Many real systems use both.

Add retries and flap control

Not every hiccup deserves a 3am alert. Give jobs a small, bounded retry, and only skip the heartbeat when a run genuinely fails. Let your monitor absorb a single missed ping before alerting so one transient blip does not page you, while a job that is actually stuck still does.

Make the alert reach a human

Monitoring that logs to a dashboard nobody opens is not monitoring. Route alerts to email and Slack, include the job name and the last known status, and make sure the person who can fix it is the one who gets pinged.

Frequently asked questions