The sale ended forty-three minutes ago

The sale ended at noon. At 12:43, a phone lit up with “Sale ends at noon. Last chance to shop.” The notification service had handed the message to a push provider only moments earlier and recorded a successful handoff. Helpful, in its own way.

Imagine a service that sends campaigns for many merchants. One merchant's sale has been running all morning. At 11:45, it submits a final reminder that the offer ends at noon. The campaign API accepts it promptly, while another merchant's much larger promotion is still being sent. The acceptance response says nothing about where those reminders will be when the sale ends.

The API records a campaign before fanout identifies recipients or sender workers hand messages to the push provider. Accepting the request is quick. Fulfilling it can involve hundreds of thousands of separate deliveries, each spending some of its useful life in a queue.

The queue keeps the difference

A queue gives workers time to recover from a brief surge. If arrivals keep exceeding completions, it stores the growing difference as waiting work.

At 11:30, a larger campaign begins expanding faster than the workers can send. Suppose 120,000 messages enter the queue each minute while 80,000 leave it. Fifteen minutes at that pace adds 600,000 messages. A new one would face roughly seven and a half minutes of work ahead of it at the same handoff rate, before any delay at the provider or device. These hypothetical numbers assume messages of similar cost and do not represent a benchmark.

Queue depth shows how much work has accumulated. The age of its oldest message shows how much of the delivery window has already been spent. A backlog of shipping updates may remain useful after the burst, even as an expiring campaign loses its purpose in the same queue. Worker utilization alone will not expose that loss, particularly when workers are waiting on a slower provider. The growing delay between fanout and handoff is the more useful sign of saturation, consistent with the queue behavior described in Google SRE's guidance.

Decide what can enter

At 11:45, the admission decision depends on the seven-and-a-half-minute backlog, the campaign's audience size, and handoff capacity available before noon. If those estimates show that much of the audience cannot be handed off in time, an early refusal gives the merchant a chance to narrow the campaign or change its plan.

The service can estimate handoff to the provider and should base acceptance on that boundary. Provider and device behavior determine when a phone displays the notification.

The service needs an estimate of the delivery work this campaign will create, together with the capacity already committed before its deadline and the merchant's share of what remains. It cannot guarantee a handoff time, yet it gives admission more to work with than a count of API calls. A million-recipient campaign counts as one request only in a particularly forgiving accounting system.

A queue buffers work. In this design, a bound on the delivery queue gives fanout a point at which it must pause. Part of an accepted campaign then waits upstream of the queue. If the API holds new requests open until space appears, waiting moves into client connections, where timeouts invite retries. I would shed new campaigns at admission while that decision is still cheap and the merchant can act on it.

Refusing a campaign is an admission-control decision and, when preserving capacity for accepted work, a form of load shedding. The response becomes a backpressure signal when it explains the constraint and causes the merchant's scheduler to change what it sends. The scheduler might submit less, defer a campaign, or abandon work whose moment has passed. An immediate resubmission leaves the API spending resources on the same refusal again.

A merchant can be within its usual rate limit and still face a temporary capacity refusal. The response needs an unambiguous status and, when a later attempt is useful, guidance for when to try again. Calling every such response “throttling” would leave the merchant guessing whether the campaign entered the system at all.

Messages have an expiration date

The campaign's deadline should travel with its deliveries. As the remaining window shrinks, fanout can stop creating work it no longer expects to hand off in time. A sender worker should also check the deadline before each handoff, so expired messages do not occupy capacity needed by useful ones. Cancellation should stop future fanout and queued sends. Work already handed to a provider may be beyond the service's reach.

Messages may wait again after handoff. Firebase Cloud Messaging distinguishes acceptance for delivery from delivery to a device and lets senders set a message lifespan. The service can pass along the campaign's remaining lifetime so the provider does not retain a stale reminder. That setting cannot recover time spent in the service's own queue.

Some shipping updates can tolerate a delay that makes a last-chance reminder ridiculous. Expiration should follow what the message means to its recipient, rather than a universal timeout chosen for the queue.

One merchant should not own the queue

The large promotion belongs to another merchant. In a single first-in, first-out queue, its earlier batch can occupy the senders long enough to delay a smaller tenant's work. The two merchants each submitted one campaign, yet claimed very different amounts of shared capacity.

I would give each merchant an admission budget based on estimated delivery work, then schedule across merchants so one campaign cannot occupy every sender until it finishes. A genuine deadline may deserve priority within that policy. It should not become a way for a large merchant to claim the entire service.

Fanout reveals the actual audience and send costs, so admission estimates need revision as work proceeds. I would also watch queue age for each merchant because an aggregate handoff rate can hide how long one tenant's messages have waited behind another's batch.

More senders, same doorway

The service adds sender workers as the queue grows, and handoffs initially rise. Once the push provider slows or rejects requests, those workers spend more time waiting or retrying against the same downstream allowance. Adding another row of workers will not widen that doorway.

The sender stage needs a limit on calls in flight to protect its own resources and a way to pace new sends when the provider pushes back. A provider such as Firebase can return throttling responses and retry hints during traffic spikes. Sender concurrency should respond to that allowance.

The slower handoff rate makes the delivery queue fill, which pauses fanout. That local limit protects the senders. Work already accepted now waits earlier in the pipeline, so admission must use the lower effective send rate when it considers the next campaign. If the signal stops at the queue, the API keeps accepting commitments the service can no longer meet. The signal completes its trip when the admission response changes what the merchant's scheduler submits.

An error is another arrival

Transient provider rejections can trigger more send attempts when downstream capacity is already scarce. An API submission timeout creates a similar ambiguity upstream: the merchant's client may resend before it knows whether fanout began. Without controls, the provider receives repeated deliveries and the service may expand the same audience twice.

For a transient failure, a worker may try again within a retry budget, using backoff and jitter so the workers do not return together and deepen the provider's overload. A provider's retry hint should guide that timing when one is supplied. Invalid or expired deliveries should leave the retry path entirely.

The API also needs a stable campaign identity. It can then recognize the merchant's second submission as another attempt at the first and avoid starting a second fanout. If the client invents a new identity after every capacity refusal, the service continues spending resources rejecting what is logically the same work.

The merchant's scheduler completes the control loop by honoring the admission response. It should cap further attempts and abandon the campaign when a later send would have no value. Otherwise, a refusal at the API becomes another arrival as soon as the connection clears.

Measure what the service can promise

An operator needs arrivals and handoffs on the same view, with queue age by merchant and campaign beside the deadlines that give it meaning. Fast campaign submissions say little about whether accepted work is reaching the provider in time.

When the provider throttles sends, reduced handoff capacity ages queued work and may force the API to refuse new campaigns. The estimate recorded with each capacity refusal makes that decision reviewable, and retry volume plus expired deliveries show the cost of delay as the system adjusts.

The merchant-facing status needs the same precision. A provider's acceptance does not prove that a device displayed the notification, and device delivery data may be incomplete. If the service reports a message as “sent,” it should mean the handoff it actually observed.

During overload, I would let the service do less work so the messages it accepts still matter. By the time the sale reminder reached the provider at 12:42, the chance to change the plan was gone.