Background jobs — the 2026 basics done right

As of: 2026-01

The invariants

1. Idempotency — handlers assume retries: dedupe on a natural key (task id + attempt), make writes conditional/upsertable. At-least-once delivery is the contract everywhere.
2. Backoff + jitter — exponential with random jitter; synchronized retries without jitter are a self-DDoS.
3. Dead-letter queues — after N attempts, park it. DLQs get: a dashboard, replay tooling, and an alert on growth rate. A DLQ nobody watches is where data quietly dies.
4. Visibility — queue depth, oldest-job age, per-queue error rate. Alert on age (consumer down) and depth growth (producer burst), not just job failures.

Patterns by scale

• Small: Postgres-backed jobs (SKIP LOCKED) — zero new infra, transactional with your data.
• Mid: Redis-backed (BullMQ/Sidekiq/Celery) — fast, ubiquitous.
• Complex workflows (multi-step, human-in-loop, long-running): Temporal-class durable execution — state machine as code, checkpoints every step; the 2026 answer to "workflow engine" builds.