Designing Reliable Asynchronous Workflows with RabbitMQ, Redis, Callbacks, and Distributed Locking

On an architecture diagram, a distributed workflow usually looks simple:

1. Receive a request.
2. Publish a message.
3. Process it asynchronously.
4. Notify another system when the work is done.

Then it goes to production.

Messages get delivered more than once. A consumer crashes after committing its data but before acknowledging the message. The system you're supposed to call back is down. Two workers try to update the same resource. A lock expires while its owner is still running. And sometimes the network fails at exactly the wrong moment, so one service has no idea whether another one finished its work.

I've spent a good part of my career building asynchronous workflows in enterprise and regulated environments, and the main thing I've learned is that no single technology makes them reliable. RabbitMQ, Redis, and database transactions each solve a different piece of the problem. The reliability comes from how you put those pieces together.

This article walks through an architecture I've found practical for workflows that need to stay predictable when things go wrong, which they will.

The example workflow

Imagine an API that kicks off a long-running operation:

The API should respond quickly while the work continues in the background. When the operation reaches a final state, the system that started it gets a callback.

The happy path is easy. A production design also has to answer the harder questions:

• What if the message can't be published?
• What if it's published twice?
• What if a worker crashes halfway through?
• What if several workers pick up operations that compete for the same resource?
• What if the callback endpoint is down?
• How does an operator find out where an operation is stuck?
• Can the workflow recover without someone editing the database by hand?

These questions should shape the design from day one, not get bolted on after the first incident.

Start with an explicit state machine

Every asynchronous operation needs a durable identity and a clear lifecycle. For example:

Depending on the domain, you might add states like RETRYING, CANCELLED, or CALLBACK_PENDING. What matters is that every transition is deliberate and validated.

A simplified operation record could look like this:

operation_id
status
version
request_reference
result_reference
created_at
updated_at
completed_at

The API generates an immutable operation_id when it accepts the request, and that ID travels with the workflow everywhere: messages, logs, metrics, database records, and callbacks.

Without a stable identifier, debugging a distributed failure is a lot harder than it needs to be.

Transitions also need guardrails. A completed operation shouldn't quietly slip back to PROCESSING because a late message showed up. You can enforce this with conditional updates, optimistic locking, or transition rules in the domain layer.

Publishing reliably with the transactional outbox

One of the first consistency problems shows up when the API has to update its database and publish a RabbitMQ message.

Take this sequence:

1. Insert the operation into PostgreSQL.
2. Commit the transaction.
3. Publish a message.

If the application crashes between steps two and three, you end up with an operation in the database that no worker will ever pick up.

Swapping the order doesn't help. If the service publishes first and the transaction then fails, a worker can receive a message for an operation that doesn't exist.

The transactional outbox pattern solves this. The API writes the business record and an outbox event in the same transaction:

BEGIN;

INSERT INTO operations (
    operation_id,
    status,
    version
) VALUES (
    :operation_id,
    'RECEIVED',
    0
);

INSERT INTO outbox_events (
    event_id,
    operation_id,
    event_type,
    payload,
    status
) VALUES (
    :event_id,
    :operation_id,
    'operation.requested',
    :payload,
    'PENDING'
);

COMMIT;

A separate publisher reads pending outbox events and sends them to RabbitMQ. Once RabbitMQ confirms it has taken responsibility for a message, the publisher marks the event as published. If you run more than one publisher, have each one claim rows with SELECT ... FOR UPDATE SKIP LOCKED (or a leased status column) so two publishers don't grab the same event at once.

Publisher confirms matter here. Writing bytes to a socket doesn't prove the broker actually accepted them, and RabbitMQ recommends confirms for any publisher that cares about reliable delivery. When a confirmation is uncertain, the producer may have to send the message again, so duplicates are still possible. RabbitMQ reliability guide

For example, the publisher might crash after RabbitMQ accepts the event but before it marks the outbox row as published. When it comes back up, it publishes the event again.

That's fine, as long as the consumers are idempotent.

Accept at-least-once delivery and design for duplicates

It's tempting to describe a workflow as "exactly once." In practice, end-to-end exactly-once processing is rarely a useful assumption to build on.

Reliable delivery usually gives you at-least-once behavior: the system works hard not to lose messages, but any given message might be processed more than once.

RabbitMQ automatically requeues unacknowledged deliveries when a consumer's connection or channel closes, which is why its documentation tells you to design consumers with idempotence in mind. RabbitMQ acknowledgements and confirms

So every message needs a unique event ID:

{
  "eventId": "01J...XYZ",
  "eventType": "operation.requested",
  "occurredAt": "2026-09-10T10:30:00Z",
  "correlationId": "01J...ABC",
  "causationId": "01J...ABD",
  "operationId": "01J...DEF",
  "schemaVersion": 1,
  "payload": {
    "resourceId": "..."
  }
}

Before applying an event, the consumer checks whether it has already processed that eventId. A common approach is to record processed IDs in the same transaction as the business change:

BEGIN;

INSERT INTO processed_events (event_id, processed_at)
VALUES (:event_id, CURRENT_TIMESTAMP)
ON CONFLICT DO NOTHING;

If no row was inserted, an earlier attempt already handled the event. The consumer rolls back and can safely acknowledge the message.

If the event is new, the consumer applies the domain update and commits both changes together:

UPDATE operations
SET status = 'COMPLETED',
    result_reference = :result,
    version = version + 1,
    completed_at = CURRENT_TIMESTAMP
WHERE operation_id = :operation_id
  AND status = 'PROCESSING';

COMMIT;

The consumer acknowledges the RabbitMQ delivery only after that commit succeeds.

There's still a gap: the commit can succeed and the consumer can crash before it sends the ack. RabbitMQ will deliver the message again, but the idempotency record means the second attempt has no duplicate business effect.

It helps to keep the roles straight:

• RabbitMQ acknowledgements protect message delivery.
• Database transactions protect local state.
• Idempotency ties the two together.

Classify failures before retrying

Not every failure deserves a retry.

A network timeout, a database that's briefly unavailable, or a 503 from a downstream service might well succeed on the next attempt. An invalid payload or an unsupported operation won't magically become valid on the twentieth try.

I usually sort errors into three buckets.

Transient failures

Timeouts, dropped connections, rate limits, and dependencies that are temporarily down. Retry these with exponential backoff and jitter.

Permanent failures

Failed validation, an unsupported schema version, a missing required field. Retrying without changing the input just burns resources and hides the real problem.

Unknown failures

Unexpected exceptions call for caution. You can retry them a limited number of times, but eventually they have to land in front of an operator.

Retries should always have a limit. A poison message that gets requeued immediately can spin in a loop forever, eating resources the whole time.

To limit retries, you have to count them. RabbitMQ updates the x-death header every time a message is dead-lettered, so a message cycling through a retry queue carries its own attempt count. Quorum queues also expose x-delivery-count. The consumer, or whatever republishes retries, reads that count and sends the message to the dead-letter exchange once it goes over the limit.

A practical topology might look like this:

Once it runs out of attempts, the message lands in a dead-letter queue along with enough context to diagnose it and replay it safely.

However you enforce the limit, configure dead-lettering on purpose. It shouldn't become a dumping ground you only look at during an emergency. RabbitMQ quorum queues

Use Redis locks only for real coordination problems

Idempotency stops the same event from having the same effect twice. It doesn't necessarily stop different events from modifying the same resource at the same time.

Say two workers try to update the same account, document, or operation. You have several options:

• A database uniqueness constraint
• An atomic conditional update
• Optimistic locking with a version column
• Pessimistic database locking
• Partitioning messages by resource ID
• A distributed lock

When the invariant you're protecting lives in the database, I reach for constraints or optimistic concurrency first. They're easier to reason about because the coordination and the data share the same transaction boundary.

A Redis lock earns its place when coordination spans several processes, or operations that can't fit into a single database transaction.

A correct basic Redis lock needs three properties:

1. Acquiring it is atomic.
2. It expires.
3. Only the owner can release it.

Acquisition looks roughly like this:

SET lock:resource:{id} <unique-owner-token> NX PX <lease-duration>

Redis documents this pattern using a unique value for each lock request. To release the lock, you compare the stored value with your token before deleting it, and that compare-and-delete has to be atomic too: either a Lua script or, on Redis 8.4 and later, DELEX ... IFEQ. A plain DEL could remove a lock someone else acquired after yours expired. Redis distributed-lock documentation

The expiry stops an abandoned lock from living forever, but it introduces a new risk.

Say Worker A takes a 30-second lock and then stalls for 40 seconds because of a long garbage-collection pause or a network delay. The lock expires and Worker B acquires it. Worker A wakes up and carries on as if nothing happened, even though it no longer holds the lock.

If that would break correctness, the lock alone isn't enough. The protected write needs its own safeguard, such as:

• A version check
• A fencing token
• A state-transition condition
• A database uniqueness constraint

My rule is simple: a distributed lock reduces contention, but wherever possible the durable data store should still be the one defending the critical invariants.

Treat callbacks as their own asynchronous workflow

A callback isn't just the last HTTP request in the chain. It's a delivery process with its own needs for persistence, retries, idempotency, security, and observability.

When processing finishes, the worker shouldn't fire the callback and then update the database. That's the same dual-write problem we hit when publishing the original message.

Instead, the worker records a pending callback in the same transaction as the final operation state:

BEGIN;

UPDATE operations
SET status = 'COMPLETED',
    result_reference = :result,
    version = version + 1,
    completed_at = CURRENT_TIMESTAMP
WHERE operation_id = :operation_id
  AND status = 'PROCESSING';

INSERT INTO callback_deliveries (
    callback_id,
    operation_id,
    destination,
    payload,
    status,
    attempt_count,
    next_attempt_at
) VALUES (
    :callback_id,
    :operation_id,
    :destination,
    :payload,
    'PENDING',
    0,
    CURRENT_TIMESTAMP
);

COMMIT;

A callback dispatcher then takes care of delivery. It picks up rows whose next_attempt_at has passed, claims them the same way the outbox publisher does, and on failure reschedules them with exponential backoff until delivery succeeds or the attempts run out.

A production callback contract should spell out:

• Which HTTP status codes count as success
• Which failures are retryable
• The maximum number of attempts and how long retries continue
• Connection and response timeouts
• Authentication or request-signing requirements
• Payload versioning
• An idempotency identifier
• Whether callbacks can arrive out of order
• How the client can fetch the current operation status

Every delivery should carry a stable callback ID. If the dispatcher sends a callback successfully but never gets the response, it will retry, and the recipient needs to be able to recognize the duplicate.

For security, a callback can include a timestamp and an HMAC signature computed over a canonical form of the request. The receiver checks the signature and rejects anything outside an acceptable time window. Build secret rotation into the protocol from the start; it's much harder to add later.

The client should also be able to ask the API for the authoritative status. Callbacks are notifications, not the only way to find out how things turned out.

Observability is part of correctness

You can't reliably run a distributed workflow on application logs alone.

Every component should pass along:

• operationId
• eventId
• correlationId, shared by every event in one workflow
• causationId, the event that directly triggered this one
• Message type and schema version
• Processing attempt
• Callback attempt

With these fields you can reconstruct a workflow across the API, the outbox publisher, the broker, the consumer, the database, and the callback dispatcher.

Metrics worth tracking:

• Pending outbox events
• Age of the oldest unpublished event
• Queue depth
• Age of the oldest queued message
• Consumer throughput
• Processing latency
• Retry count by reason
• Dead-letter count
• Lock-acquisition failures
• Pending callbacks
• Callback success rate and latency
• Operations stuck in non-terminal states

Queue depth on its own can mislead you. Ten messages that just arrived are probably fine; one message that's been waiting for hours probably means something is broken. Tracking both volume and age gives you a much clearer picture.

Alerts should map to user impact or to something someone needs to do. A rising retry rate might be an early warning. An old unpublished outbox event might mean a request you already accepted has stopped moving.

Recovery should be designed, not improvised

Sooner or later, an operator will need to replay a dead-lettered message, retry a callback, or unstick an operation.

Those actions deserve proper tools and procedures.

A safe replay process should:

1. Keep the original event ID wherever idempotency depends on it.
2. Record who triggered the replay and why.
3. Check that the operation is still in a compatible state.
4. Go through the normal business rules, not around them.
5. Emit the same logs and metrics as regular processing.

Editing the database by hand should be the last resort. It's hard to audit, and it can create states the application was never built to handle.

The resulting reliability model

Put these patterns together and you get a workflow whose guarantees you can actually explain:

• The transactional outbox ensures an accepted operation never silently loses its initial event.
• Publisher confirms tell you whether RabbitMQ took responsibility for a message.
• Durable queues and manual acknowledgements make message recovery possible.
• Idempotent consumers make redelivery safe.
• Database constraints and version checks protect business invariants.
• Redis locks coordinate the cross-process operations that need it.
• Persistent callback delivery makes notifications retryable.
• Dead-letter queues isolate failures that need investigating.
• Correlation IDs, metrics, and audit records make the workflow observable and recoverable.

No single component gives you end-to-end reliability. The guarantees come from how the boundaries between them are handled.

Final thoughts

Reliable asynchronous systems aren't systems that never fail. They're systems that expect failure, limit its blast radius, and recover predictably.

The questions that show whether a design gets there are rarely about the happy path:

• What happens if this line runs twice?
• What happens if the process dies right after this transaction?
• Which component is responsible for the work right now?
• How would we know that progress has stopped?
• Can an operator safely retry this?
• If the lock expires, what still protects the data?

Answer those first. The technology should serve the reliability model, not dictate it.