Idempotency and retries

In the previous deep dive, a user uploaded a document. We asked whether making it searchable had to finish inside the upload request, and derived a queue-backed path from that requirement. Now the document is ready. The user opens the same product, types “What are the key decisions in this document?”, and presses Send.

From their point of view, there is one question waiting for an answer. Behind the screen, a worker might crash, a connection might time out, or a message might be processed again. The new question is: if answering that question takes more than one attempt, what stops those attempts from creating extra replies?

We will use a background reply task so that accepted work can outlive the browser connection. That is a choice for this example, not a claim that every chat request needs a queue. The same failure reasoning also applies when the request is handled directly.

The chat API saves the user’s message and creates a reply task: produce one answer to this message. It also records a recoverable handoff to the queue, so a crash after accepting the request cannot quietly lose the work. A dispatcher may publish that handoff more than once; each message still points to the same task.

A worker receives the task and calls the LLM provider. The returned text is a candidate answer. It becomes the saved response only when our application accepts it. In this design, saving that response and marking the task complete happen together in one database operation.

Only after that durable save does the worker report success to the messaging system. The browser can fetch the saved response, reconnect to it later, or receive a notification that it is ready. Queue progress does not wait for the user’s screen to display it.

These names matter because one reply task can have several model attempts. We are not treating several provider calls as one physical call. We are keeping them attached to one piece of work and deciding which result, if any, becomes the reply.

Suppose the provider produces an answer, but the worker crashes before saving it. Another worker later encounters the unfinished task and tries again. The provider may now have done the expensive work twice. This can happen with SQS or Kafka, but their recovery mechanisms are different.

SQS Standard: a message is hidden, not removed

Receiving an SQS message starts its visibility timeout. For that period, the message is normally hidden from other workers. If the worker finishes, it deletes the message. If it does not delete it before the timeout expires, the message becomes available again. A long-running worker can extend the timeout. Thirty seconds is an example window, not a recommended duration for every model call. SQS visibility timeout documentation.

That helps avoid overlapping work, but it is not a correctness lock: SQS Standard can still deliver duplicates, including during the visibility window. The application must tolerate repeated processing. SQS at-least-once delivery documentation.

Kafka: the record stays; the group saves its place

In a traditional Kafka consumer group, each partition is assigned to one consumer in that group at a time. Reading a record does not remove it. A committed offset records where the group should resume. For this example, automatic commits are disabled and we commit only after durable processing.

Imagine the saved restart position is 41. The worker reads record 41, advancing its local read position to 42, then crashes before saving the reply or committing progress. After recovery or reassignment, processing resumes from 41. After successfully handling that record, with earlier records also handled, the consumer commits 42. The commit does not delete record 41.

That is rereading from saved progress, not a per-message visibility timeout. With concurrent processing, do not commit past unfinished records in the same partition. Kafka transactions also do not automatically include an external LLM call. Kafka consumer positions and manual offset control.

Start with the simplest protection. When a candidate answer arrives, the database saves it only if this task has no saved response yet. The check and the save must be atomic: the database makes them one indivisible decision. Two workers must not both read “empty” and then independently append their answers.

The first successful save fills the task’s response slot and completes the task. A later attempt finds that the reply already exists and leaves it unchanged. The chat displays that saved response, rather than appending every candidate answer it receives.

This is the idempotent behavior we need at this boundary. It does not mean the model returns identical text, the provider is called exactly once, or an answer is guaranteed to exist. All attempts could fail. The safety promise is at most one saved response per reply task; retry and recovery policies improve the chance of completing it.

The browser creates a request ID for this press of Send and reuses it if submission must be retried. The API uses a unique key scoped to the authenticated caller or workspace and operation. Repeating the submission returns the existing reply task and its status, rather than creating another message and task.

Bind that key to the original inputs, including the relevant conversation and document version. Reject reuse with different inputs instead of silently answering a different question. Keep the deduplication record for the supported retry and replay window; expiring it too early makes old work look new. These are application choices based on an explicit idempotent API contract.

But if the user presses Regenerate, they are asking for another answer. That creates a new reply task, even if the prompt is identical. Hashing the prompt alone would confuse intentional regeneration with a network retry. An attempt ID may help trace each execution, but it must not replace the stable task identity.

So, do we avoid enqueuing the same request twice? We avoid creating new logical work for a repeated submission. We still design for repeated queue messages and worker attempts for the existing task. Request deduplication and worker idempotency solve different parts of the problem.

Before calling the model, a worker checks the shared task record. If the reply is already complete, it returns without another model call. If the task is available, it asks the database to assign it the work. Checking availability and recording the assignment happen together. Only the worker whose claim succeeds proceeds.

A permanent “running” flag is not enough. If the worker crashes, nobody could take over. Instead, the assignment has a renewable time limit, often called a lease. The worker extends it while making progress. After it expires, recovery can assign the task to another worker, subject to the retry policy.

This database assignment is separate from SQS visibility. One coordinates our right to work on a task; the other controls delivery of a queue message. Neither timer can cancel a provider call that is already running. Do not hold a database transaction open while waiting for the LLM; keep claims, renewals, and completion writes short.

The worker sends the request. The connection then fails. Maybe the provider never received it. Maybe it is still generating. Maybe it finished, charged for the work, and the answer was lost on the return trip. Locally, all three can look like “no answer received.”

First ask whether the provider offers a documented way to recover the existing operation: a stable idempotency key with defined behavior, or durable lookup by an operation ID we already saved. Verify the relevant endpoint, retention window, payload rules, and concurrent-request behavior. A request ID that only helps support find logs is not automatically either guarantee.

If recovery is unavailable, there is a real product decision. Retry and accept the possibility of duplicate model work, or stop automatic recovery and risk leaving the user without an answer. In this chat example, we choose a bounded retry because completing the reply is worth some duplicate computation. That is a policy, not a property of idempotency.

Another product may make the opposite choice. Even then, “we will not automatically resend an ambiguous request” is narrower than promising exactly one physical provider execution. Client libraries and downstream services may have their own retry behavior.

A worker can lose contact without dying. While it is delayed, its assignment expires and another worker takes over. The old worker later returns with an answer. An empty response slot alone would let that old worker win, even though our recovery process has already chosen a replacement.

Give each new assignment an increasing version number. The worker remembers the version it received. Before accepting its answer, the database checks that this version is still current, the assignment is valid, and there is no saved response. This is often called fencing: an old assignment cannot authorize a new write.

BEGIN;

SELECT task_id
FROM reply_tasks
WHERE task_id = :reply_task_id
FOR UPDATE;

UPDATE reply_tasks
SET saved_response = :candidate_answer,
    status = 'completed'
WHERE task_id = :reply_task_id
  AND assignment_version = :my_assignment_version
  AND status = 'running'
  AND lease_until > clock_timestamp()
  AND saved_response IS NULL
RETURNING task_id;

COMMIT;

Acquire the task-row lock first, then check the lease against clock_timestamp() in the guarded update on the same connection and transaction. This authorizes completion while holding the lock; a takeover must wait. The transaction may commit after that check. CURRENT_TIMESTAMP instead reports transaction-start time, so it cannot establish a fresh expiry check after waiting for a lock. Keep this transaction short and use equivalent locking and time semantics for claims and renewals. PostgreSQL time functions, row locks.

One returned row means the answer was accepted. Zero rows means it was not: read the task’s current state to distinguish a saved reply, an expired assignment, or a takeover. In PostgreSQL’s Read Committed mode, a competing update waits and rechecks its condition against the updated row. This simple single-row example relies on that behavior; use equivalent guarantees in your chosen database. PostgreSQL concurrent update semantics.

The assignment version must increase atomically on every claim or takeover, and renewals must check the same version. If responses live in another table, use a unique task key and put the accepted response and completion update in one transaction. A separate unconditional write would reopen the race.

What if this save succeeds, but its acknowledgement is lost? Read the durable task before deciding to generate again. What if the worker crashes after saving but before reporting queue progress? The next worker finds the completed task and skips the model call.

Rejecting old assignments is the design chosen here. A different product could accept the first valid answer from any attempt for immutable inputs. That can reduce wasted results, but it is a different acceptance rule. Do not accidentally mix that policy with “only the current worker may complete.”

“Retry with backoff” is not enough by itself. Start with what we know about the failure, then choose an action. These are alternative paths, not four retries every task should perform.

For retries we choose to allow, backoff means increasing the delay between attempts. Jitter means varying it so workers do not all retry together. Add a maximum attempt count or elapsed-time budget, and respect cancellation and deadlines. Do not assume every server error means nothing happened remotely.

Account for retries across layers: browser, API, queue worker, and provider SDK. Letting every layer independently retry several times can multiply calls. The task record should make the overall budget and terminal result visible, rather than letting redelivery reset the allowance.

For this example, stable task identity, a short worker claim, and a guarded completion write work together. They reduce avoidable calls and protect the saved reply. But system design also means knowing the other options and what each gives up.

For example, if both a local database change and the consumer’s restart position are stored in the same database, one transaction can cover them. That does not bring the remote model execution into the transaction. Always say which effect is protected before saying “exactly once.”

And yes, there are times when we accept something that is not idempotent. An LLM may produce different candidate text on each attempt, and we may pay for repeated computation. We tolerate that here while protecting the reply the product saves. A separate external action, such as sending a notification or charging a customer, needs its own effect-specific contract; this response guard does not cover it.

Return to document ingestion. The logical work might be “index this version of this document,” with the processing stage included in its identity. Retrying that task should not add duplicate index entries. A new document version is new work, and a late worker must not make an older version current over a newer one.

The data model and storage checks will differ from chat. The reasoning stays the same: what counts as the same work, which repeated effect would hurt, where can we enforce a decision atomically, what can the downstream system recover, and which is worse when we are uncertain: another effect or no result?

Test the failure boundaries

These are assertions to test against your implementation. The diagrams illustrate the expected outcomes; they are not evidence that a real queue-worker service has passed these tests.

Technical references

• Amazon SQS: at-least-once delivery
• Amazon SQS: visibility timeout
• Apache Kafka: consumer positions, manual commits, and storing offsets externally
• Amazon Builders’ Library: making retries safe with idempotent APIs
• PostgreSQL: transaction isolation and concurrent updates