Queues and background jobs
What moves to the background
Sending email and notifications, generating reports, processing media, syncing with external systems, and any call to a service whose availability you don't control.
The result: a fast answer to the user, and resilience — when the external service goes down, the jobs wait instead of failing.
Three rules per job
Repeatable. A double run won't create a duplicate. Achieve this with a unique key per action.
Small. A job that runs two hours is one you can't retry. Split it.
Reported. Start, finish, failure, and retry count.
What's forgotten
A dead-letter queue for jobs that failed repeatedly, with an alert — otherwise they vanish silently. Retry with growing delay, so as not to bring down a service already struggling. And order: if order matters, set a partition key; otherwise don't assume it.
Going deeper
Monitor three metrics: queue length, age of the oldest job, and failure rate. A queue length climbing slowly over days is the early symptom of most load incidents, and it shows long before a user feels anything.