Backpressure and capacity

Imagine users uploading documents to an AI assistant. The API accepts each document, puts a processing job in a queue, and lets workers make the document searchable. The product target is to finish each accepted document within three minutes.

Now uploads arrive faster than the workers can finish them. The API can still respond quickly, but documents spend longer waiting. If that continues, users miss the promised completion time.

This deep dive is about capacity—how much work the system can finish—and backpressure—feedback that asks the sender to reduce the work it sends when the receiving system cannot keep up.

Our goal is to keep serving demand and finish documents on time. We will first use the queue to absorb a spike, then increase worker capacity when it is needed. Backpressure protects the system when enough useful capacity is not available, including while new workers are starting. Slowing uploads is not the outcome we are trying to optimize for.

If you want to understand why we added the queue, start with when to add a queue. Here, the queue already exists. We need to decide how the system should respond when demand changes.

Normally, four documents arrive per second. Suppose we have tested this workload and found that our current workers can finish about five documents per second when enough work is available. These are illustrative numbers, not measurements of InterviewLayer.

That five-per-second figure is our current capacity. With only four arriving and no backlog, we will actually finish about four per second. That actual completion rate is throughput.

For the examples below, documents require similar amounts of work, waiting documents start in arrival order, and completions are spread roughly evenly over time. We count waiting documents separately from documents already being processed. These assumptions let us make useful estimates; they do not promise an exact deadline for every document.

There are two demand patterns worth separating: a spike that ends and higher demand that continues.

Uploads jump from four to fifteen documents per second for 30 seconds. We accept them all, but the workers still finish only five per second. Once they are fully occupied, waiting work grows by about:

15 accepted − 5 completed = 10 more waiting documents/second

After 30 seconds, approximately 300 documents have accumulated. Then uploads return to four per second.

The existing workers can now catch up. They finish five per second while four new documents arrive, so the backlog shrinks by one per second:

300 waiting ÷ (5 completed − 4 accepted) = 300 seconds to clear

The queue has done something useful: it held the temporary excess until workers could catch up. But should we accept five minutes of recovery, or add workers to recover faster?

That depends on how long individual documents will wait, not just how long the entire queue takes to empty. We will work out that waiting budget after seeing our second example. A small spike that stays within the budget may not need any scaling.

Start again with an almost empty queue. This time, fifteen documents arrive per second and keep arriving. Our capacity remains five per second.

The backlog grows by ten per second: roughly 300 waiting documents after 30 seconds and 900 after 90 seconds. At the 90-second checkpoint, fifteen documents are still arriving every second. There is no return to the normal rate and no recovery period.

A larger queue gives us more storage, not more completions. Waiting longer does not fix this mismatch. We need to increase useful processing capacity, reduce the rate of accepted work, or combine both. For this system, our preferred response is to serve the higher demand by adding workers where doing so will help.

A worker is a running copy of the processing program. Several worker instances can take different documents from the same queue and process them concurrently. Depending on the deployment, those instances might run in processes, containers, or on separate machines. Adding a worker does not necessarily mean adding one physical computer.

For this first scaling example, assume workers are the bottleneck and the services they call have enough spare capacity. More ready workers can then increase the rate at which documents become searchable.

Use a new, explicit snapshot: when the additional workers are ready, 300 documents are waiting and fifteen still arrive per second. Compare two amounts of capacity:

At twenty completions per second, those 300 waiting documents take approximately:

300 ÷ (20 − 15) = 60 seconds to clear

The 60 seconds starts from the moment that capacity is ready, not from the moment we request more workers. If startup adds more waiting work, we must use the backlog that actually exists then. The key distinction is enough capacity to keep up versus enough capacity to catch up.

The same response can help with the temporary spike. We can temporarily add workers to shorten recovery, then reduce the worker count when demand and backlog allow it. Neither pattern means “always scale immediately”; we still need a policy tied to the completion target.

The queue provides metrics; the scaling controller makes the scaling decision. It reads queue and worker metrics, compares them with a configured target, and asks the infrastructure to adjust the worker count. It repeats this check as demand changes, so we do not need someone watching the queue and manually starting every worker.

For example, a policy can track backlog per ready worker: waiting work divided by the number of workers currently able to process it. The same queue size means something different with two workers than with twenty. Processing time and acceptable queue delay help determine a useful target. AWS documents this approach for SQS and EC2 Auto Scaling using backlog per in-service instance. AWS backlog-based scaling guidance.

The animation uses an illustrative target of fifteen waiting documents per ready worker, with five to twenty workers allowed. With 300 waiting and five ready, the measured backlog is sixty per worker—above the target. Dividing 300 by the target of fifteen gives a recommendation of twenty workers in total. The infrastructure starts the extra fifteen; the controller tracks that pending capacity rather than repeatedly requesting the same increase. Real policies also account for measurement windows, startup, and scale-in stability.

We then check the result. Did more documents actually become searchable per second? Did waiting time improve? Adding instances is an action; increasing useful capacity is the outcome we need.

Startup time matters too. A requested worker may still be loading code, opening connections, or initializing before it can process useful work. Scaling decisions must allow for that delay and avoid repeatedly reacting to transient startup signals. AWS's warmup controls are one concrete implementation of this concern. AWS instance warmup guidance.

Set a maximum worker count consistent with cost and downstream limits. When demand drops and the backlog recovers, scale down carefully: preserve enough capacity for incoming work, and let in-progress work finish or be safely recovered if a worker stops. AWS's queue-worker example also covers scale-in protection for work in progress. Worker scale-in protection.

We now have a reason to calculate waiting time: it tells us whether buffering and scaling are meeting the product target, and how much time remains to react.

Return specifically to the temporary-spike recovery snapshot: 300 documents waiting, capacity of five per second, and four new arrivals per second. Do not carry over the twenty-per-second capacity from the scaling example.

A new document joins behind those 300. Its estimated wait before processing starts is:

300 waiting ÷ 5 completions/second = 60 seconds

That differs from the five minutes needed to clear the whole backlog. Documents arriving later join behind this document, so they do not extend its wait in our arrival-order example. They do keep adding to the total work that must be cleared.

The three-minute product target includes waiting and processing. Suppose we reserve 60 seconds for processing, moving between services, and variation. That leaves:

180 seconds total − 60 seconds reserved = 120 seconds for waiting

What would use up those two minutes? Imagine a new document joining behind 600 waiting documents, instead of 300. At five documents per second, getting through the work ahead takes about 120 seconds:

600 waiting ahead ÷ 5 documents/second ≈ 120 seconds before processing starts

The new document would spend its entire waiting allowance just getting to the front. It still needs time for its own processing afterward. This is where the rough 600-document limit comes from: five documents per second multiplied by the 120 seconds we allowed for waiting.

Does that mean we wait until 600 documents are queued before adding workers? No. When the backlog is growing, more documents can arrive while new workers are starting. We need the extra capacity ready before waiting gets too long. Some documents also take longer than expected, and workers do not always start in the same amount of time. Act earlier and leave room for those delays. Measure the real workload to choose that margin; 600 is an estimate for this example, not a guarantee that every document meets its deadline. If enough capacity cannot arrive in time, temporarily limit new work.

Finally, count documents waiting outside the main queue too. In a separate counting example, suppose 400 documents are in the queue and workers have fetched another 200 into local buffers but have not started them. That is 600 waiting documents, not 400. Moving a document out of the queue does not mean its processing has started. Documents already being processed are a separate group, and moving accepted work between buffers or retry queues does not reset its deadline.

Adding workers helped in our first scaling example because workers were the bottleneck. Now change that assumption: the worker fleet could handle twenty documents per second, but every document needs an embedding service that can handle only five of these documents per second.

End-to-end capacity is still at most five per second. More worker instances cannot get around that shared limit. We need more capacity at the dependency, less work per document while preserving the promised result, or a different processing path. Until then, the system still needs protection from excess work.

There is a separate limit that more workers cannot fix: the time one document must spend on a required step. Suppose, in a different hypothetical, every document requires a dependency call that takes three minutes. Even with an empty queue, that document cannot finish in under three minutes. Other necessary work leaves no room for a reliable three-minute end-to-end promise either.

More concurrency might let us process more documents at once; it does not shorten that required call. Backpressure cannot fix an impossible processing deadline. We must change the critical path or the promise.

Our document-count examples also assume similar work. A two-page PDF and a 500-page PDF may cost very different amounts to process. For a mixed workload, track a tested estimate of remaining work and resource demand rather than treating every queued item as interchangeable. Google's overload guidance explains why request counts alone can misrepresent workload. Google SRE overload guidance.

Autoscaling and backpressure are complementary. New workers may still be starting, a maximum may have been reached, or a dependency may be limiting completions. We should not accept unlimited work while hoping capacity will eventually arrive. The goal is to make protection unnecessary when the system can safely serve demand—not to claim that it is always rare.

Start a separate protection example with 300 documents waiting, capacity of five per second, and callers trying to send fifteen per second. Enough additional capacity is not ready.

If a batch uploader responds to feedback by reducing its rate to four documents per second, we can accept all four. Five finish each second, so the backlog shrinks by one per second. Work not yet sent remains with the uploader; our service has not accepted it.

That is backpressure: the receiving side communicates its limits upstream, and the sender adjusts. It need not mean pausing completely. Reactive Streams gives a concrete demand-signalling model.

But not every caller follows that feedback. If callers continue sending fifteen per second while our safe acceptance rate is four, the boundary must enforce it:

15 sent = 4 accepted + 11 refused before acceptance

Deciding what to accept is admission control. Refusing excess work before accepting it is load shedding. A feedback signal without an enforced boundary does not protect us from a sender that ignores it. Accepting only four is an illustrative recovery choice, not a tuned production setting; accepting five would hold the backlog steady rather than shrink it.

This snapshot still has room in its waiting budget. If a new document would already wait too long, accepting fewer documents does not instantly remove the work ahead of it. Pause or refuse new work under the usual promise until its estimated wait fits again with a safety margin, while continuing to handle work already accepted.

For temporary overload, the API can return 503 Service Unavailable before placing the rejected document's job in our processing queue. An optional Retry-After header suggests when to try again. HTTP status definition, Retry-After definition.

HTTP/1.1 503 Service Unavailable
Retry-After: 30
Content-Type: application/json

{
  "message": "We cannot accept more documents right now. Please try again later."
}

The 30 seconds is an illustrative retry delay, not a guaranteed available place. For an enforced request-rate limit, such as a caller's quota, 429 Too Many Requests is the related response. 429 definition.

Once we have said “accepted,” the work remains our responsibility. Save it reliably, keep processing and tracking it, and report delays or a clear failure outcome. We cannot quietly discard accepted documents or put refused work into another hidden queue. A slower option for new uploads needs an explicit, different promise.

Retries belong in the same capacity budget as first attempts. If a dependency handles five attempts per second and a completed document requires two attempts on average, useful capacity is only about 2.5 documents per second. Bound retry attempts and use backoff with jitter so retries do not synchronize into another spike. Idempotency can prevent duplicate committed effects within the boundary it protects; it does not make retry work free. Idempotency and retries deep dive, Google SRE retry guidance.

Fifteen documents arrive every second and continue arriving. The system can currently finish five per second, and 300 documents are waiting. Workers are the bottleneck, and dependencies have spare capacity. You can scale to twenty completions per second, but new workers take time to become ready. The three-minute target and illustrative 120-second waiting allowance still apply.

What would you do now? What would you watch while workers start? How would your answer change if the dependency could finish only five documents per second?

In a system design interview, the decision can be stated simply:

“The queue absorbs temporary excess work. I'll use workload and waiting-time metrics to scale workers, and verify that completions actually increase. Sustained demand needs enough capacity both to keep up and to clear the backlog. If capacity cannot grow quickly enough, or a dependency is the bottleneck, I'll use backpressure and enforce what we can safely accept.”

The goal is not a permanently empty queue, nor routinely blocking users. It is to serve demand and finish accepted work on time: buffer what we can safely absorb, scale useful capacity, and protect the system when capacity falls short.