Begin with the product requirement
Adding a queue is easy. Explaining why it belongs in the design is the part that matters.
Candidates sometimes add Kafka, SQS, or another messaging system as soon as they notice a slow operation. But “this operation is slow” is not enough to justify a queue. A stronger explanation connects the queue to a requirement, identifies the coupling in the current design, and states both what the queue improves and what it cannot solve.
Running example
Make an uploaded PDF useful inside the product
A user uploads a PDF. The goal is to make the document searchable so chat, search, or another product surface can reference it. For now, ignore the internal processing steps and focus on one question: must all of that work remain inside the upload request?
The product needs fast acknowledgement of a durable upload. Making the PDF searchable can finish afterward.
If document processing does not need to finish before we respond to the user, when and why should we move that work behind a queue?
In the diagram, “fast acknowledgement” means the system has taken responsibility for the document and its ingestion job. “Searchable document” is a later state in which the required processing is complete and the document is available to retrieval. That requirement creates an asynchronous boundary. The queue is a possible mechanism for implementing it, not the starting assumption.
Start without a queue
The simplest version performs the entire ingestion pipeline before returning success.
The response returns only after OCR, compliance, embeddings, and search indexing have finished.
This can be reasonable at low volume. “Synchronous” does not mean the service processes only one request at a time. The service may handle many requests concurrently. The problem is that every open request remains coupled to the slowest and most variable stage.
Concurrent clients keep requests open while slow OCR and throttled embeddings constrain the processing path.
As the system reaches one of those limits, new requests wait behind the constrained stage. They may slow down, time out, or be rejected. Client retries can then add work to a system already under pressure.
Say this out loud
Request-handling capacity is coupled to document-processing capacity, even though the user does not need processing to finish inside the request.
Separate acceptance from processing
Move only the work that can safely finish after the response.
Validation and the ingestion record belong before the response. OCR, compliance, embeddings, and indexing can finish later.
Durable-acceptance invariant
Do not return success while a process failure could leave an accepted job with no recoverable path to the queue. One implementation stores the document first, then atomically creates the ingestion job and a pending handoff record. A dispatcher publishes that pending work to the queue. The mechanism can vary; the invariant cannot.
The API can return 202 Accepted with an ingestion ID. In HTTP, that means the request was accepted for processing but processing has not completed. The status code is intentionally noncommittal and does not guarantee eventual success by itself.
This API defines a stronger application contract: after it returns 202 Accepted, the document and a recoverable record of the ingestion work are durable, and the system tracks the job to a terminal status.
{
"ingestion_id": "ing-123",
"status": "ACCEPTED",
"status_url": "/ingestions/ing-123"
}The status can move from ACCEPTED to QUEUED, PROCESSING, and finally READY or FAILED.
Improve the design with a queue-backed path
Introduce the queue after the accepted upload so pending work can wait durably until processing capacity becomes available.
The API returns 202 Accepted after durable acceptance, then the queue holds small pending jobs until workers can run OCR, checks, embeddings, and indexing.
The queue does not process the PDF. Workers process the PDF.
The queue holds a message that tells a worker which job is ready.
{
"ingestion_id": "ing-123",
"document_id": "doc-456",
"tenant_id": "tenant-7",
"object_key": "uploads/doc-456.pdf"
}The message contains identifiers and a storage reference instead of PDF bytes. Object storage owns the document. The queue carries enough information for a worker to find and process it.
A worker fetches the PDF, marks the job as processing, performs the pipeline, marks it ready, and acknowledges the queue only after success. At-least-once delivery makes idempotent processing necessary.
| Term | Meaning in this design |
|---|---|
| Request | The client's HTTP interaction with the ingestion service |
| Job | The durable unit of document-ingestion work |
| Message | The queue entry announcing that a job is ready |
| Dispatcher | The process that publishes recoverable pending jobs to the queue |
| Queue | The component that stores and distributes pending messages |
| Worker | The process that executes or coordinates the ingestion job |
A queue changes where the work waits
It can make waiting durable and controllable. It cannot make the bottleneck disappear.
Shorter request path
The client waits for durable acceptance, not OCR, embeddings, and indexing.
Explicit waiting
Pending work becomes jobs and messages instead of open requests or fragile in-process tasks.
Controlled concurrency
Workers can pull at a rate that protects a bounded downstream dependency.
Independent scaling
The request path and worker pool can scale according to different pressures.
A queue does not create processing capacity.
We changed where the document waits and how that waiting is managed. We did not make one document finish faster.
Temporary burst: the backlog drains
Arrival falls below processingUploads briefly arrive faster than workers can process them. Once the spike ends, the workers keep consuming jobs until the queue is empty.
Temporary burst
Sustained overload: the backlog keeps growing
Arrival stays above processingWorkers remain busy, but new jobs arrive faster than they finish. The queue eventually reaches a limit, so the system must scale, throttle, or reject work.
Sustained overload
Adding workers helps only until they encounter the same shared OCR, embedding, database, or indexing constraint. A queue is effective when the processing system can catch up after a finite burst. It is not a capacity plan when the arrival rate remains above the processing rate.
Choose from the required behavior
Only after deriving the need for a queue should you name a product.
The requirements we have today
- Each document creates an independent job
- One worker from the pool should process it
- Global ordering is not required
- Historical upload events do not need replay
- Several consumer groups do not need the same event
Independent jobs with no replay or global-ordering requirement point toward SQS Standard. Retained history, replay, partition ordering, or multiple consumer groups make Kafka more compelling.
Choose for this design
Amazon SQS Standard
A sensible AWS starting point for independent pending jobs claimed by one logical worker pool. Standard queues use at-least-once delivery and best-effort ordering, so processing must tolerate duplicate delivery.
Reconsider when requirements change
Apache Kafka
More compelling when retained history, replay, partition-level ordering, or several independent consumer groups become central to the design.
| Requirement | Starting abstraction |
|---|---|
| Independent background jobs claimed by a worker pool | Work queue such as SQS |
| Retained event history and replay | Event log such as Kafka |
| Several independent consumer groups need the same event | Event log such as Kafka |
| Ordering within a defined stream is central | A system with explicit ordering semantics |
The comparison is not “SQS is good and Kafka is bad.” For these requirements, start with SQS. If replay, retained history, multiple independent consumers, or partition ordering becomes central, reconsider Kafka. Naming the requirement that would reverse your decision is strong interview signal.
Know when a queue belongs in the design
Look for evidence in the workload and product contract before reaching for a queue or naming a technology.
Queue selection framework
The pillars for deciding whether to introduce a queue
Start with the lifecycle of the request, then test whether the work needs a durable lifetime, buffering, or controlled consumption. These signals tell you when a queue solves a real problem and when it would only add complexity.
Request lifecycle
Does the work need to finish before the user receives a response?
Queue signal: The product can acknowledge a durable upload now and let expensive processing finish later.
Why it matters: This is the first reason to create an asynchronous boundary. If the result is required immediately, a queue may not help the request contract.
Work duration and variability
Is the processing slow, unpredictable, or likely to exceed request timeouts?
Queue signal: OCR, compliance checks, embeddings, or indexing vary enough that holding the request open creates fragile coupling.
Why it matters: Queues are useful when the lifetime of the work should not be constrained by the lifetime of one HTTP request.
Durability and recovery
Must accepted work survive a process crash, deployment, or temporary dependency outage?
Queue signal: The system needs a recoverable handoff and a durable place for pending jobs to wait.
Why it matters: A queue becomes valuable when work must outlive the service instance that accepted it.
Rate mismatch and burstiness
Can arrivals temporarily exceed the rate at which workers or dependencies can process them?
Queue signal: A finite spike creates a backlog that the processing system can drain after arrivals fall.
Why it matters: This is the buffering case a queue solves well. If arrivals remain above throughput, capacity or admission control is still required.
Downstream protection
Does a bounded dependency need protection from uncontrolled concurrency?
Queue signal: Workers should pull jobs at a controlled rate that OCR, the embedding service, or the search index can sustain.
Why it matters: A queue can turn a flood of requests into managed concurrency and make backpressure explicit.
Delivery and consumption model
Should one worker claim each job, and what delivery, ordering, replay, or fan-out behavior is required?
Queue signal: Independent jobs fit a work queue; retained history or several independent consumers may point toward an event log.
Why it matters: Recognizing that a queue belongs in the design is separate from choosing the correct queue technology and processing semantics.
Documents arrive faster than the workers can process them. Does adding a queue make the backlog drain? What needs to change?
Compare your reasoning
The queue holds pending work, but it does not add processing capacity. A temporary burst drains only when effective processing throughput exceeds new arrivals. Sustained overload requires more effective capacity, less work per document, or admission control, while respecting downstream limits.
Scope boundary
This resource stops at introducing and defending the queue-backed boundary. Retry policies, visibility timeouts, dead-letter queues, worker autoscaling, multi-stage workflow orchestration, and a complete document-ingestion platform are separate deep dives.