Request Deadlines Need an End-to-End Budget

A checkout request has a one-second response target. The API client allows one second for inventory, another second for pricing, and another for payment. Every dependency obeys its configured timeout, yet the customer waits several seconds. Adding retries makes the gap larger. The bug is in the composition: each operation starts a new budget after earlier work has already spent time.

A deadline belongs to the logical request. Individual timeouts are allocations from that budget. This distinction becomes useful as soon as a request waits in a queue, crosses a service boundary, or retries an uncertain operation. The goal is to decide whether more work is still useful before starting it.

Account for the whole path

Start the clock at the boundary whose latency you promise. If the promise starts when a gateway accepts a request, measuring only handler execution excludes gateway queues. Record an absolute deadline using a local monotonic clock, then carry that deadline through functions in the same process. Read the clock again before every potentially blocking stage.

Reserve some time for constructing the response and releasing local resources. The reserve is a planning allowance, not a guarantee that cleanup will finish within it. A stuck driver or cancellation-resistant task can exceed the allowance. Give cleanup its own bounded policy and measure actual completion separately from the moment the caller stops waiting.

For sequential work, the available attempt timeout is the smaller of the dependency limit and the remaining request budget after the reserve. Queue waits and retry backoff consume the same budget. Parallel branches share the deadline; they do not each receive a fresh extension. A branch may receive a smaller allocation if the caller needs time to combine results.

Make budget arithmetic reproducible

This Python example uses a synthetic clock so the assertions do not depend on scheduler timing. It models admission and timeout allocation only. A real client must pass the returned duration into an API that enforces it, and the caller must still enforce the overall deadline.

from math import isclose, isfinite


def attempt_timeout(deadline, now, cap, reserve):
    values = (deadline, now, cap, reserve)
    if not all(isfinite(v) for v in values):
        raise ValueError("finite times required")
    if cap <= 0 or reserve < 0:
        raise ValueError("invalid policy")
    remaining = deadline - now - reserve
    if remaining <= 0:
        raise TimeoutError("no attempt budget remains")
    return min(cap, remaining)


# A deterministic clock: no network calls or sleeps.
now = 100.0
deadline = now + 1.0
reserve = 0.10
now += 0.30  # queue wait
assert isclose(attempt_timeout(deadline, now, .25, reserve), .25)
now += .25  # first attempt used its allowance
now += .20  # retry backoff consumes the same budget
assert isclose(attempt_timeout(deadline, now, .25, reserve), .15)
now += .20
try:
    attempt_timeout(deadline, now, .25, reserve)
except TimeoutError:
    pass
else:
    raise AssertionError("expired work was admitted")

assert isclose(attempt_timeout(2, 1, .25, 0), .25)
for bad in (float("nan"), float("inf")):
    try:
        attempt_timeout(2, bad, .25, 0)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid clock accepted")

The first call receives 250 milliseconds. After the queue, failed attempt, and retry delay, only 150 milliseconds remain for the next attempt while preserving the reserve. At the final check, the model refuses another attempt. Replacing that calculation with the original one-second timeout would silently extend the request whenever a stage starts late.

The test also rejects nonfinite clock values. In an actual service, policy values should be validated when configuration loads. Deadlines created by trusted code should not need repeated parsing at every call site; the arithmetic belongs in a small shared abstraction with units that are hard to confuse.

Propagate meaning across machines

A monotonic timestamp is useful within its clock domain. Do not serialize its numeric value to another host and assume both clocks have the same origin. A protocol can propagate a remaining duration, from which the receiver creates its own local deadline. The receiver must also cap that duration by its own policy rather than accepting an arbitrary client allowance.

A duration header alone does not subtract time spent in transit. The originating service must keep enforcing its own deadline, and a protocol that promises precise propagation needs documented treatment of transit time and clock skew. Absolute wall-clock deadlines can be appropriate in an established protocol with those assumptions. Inventing a timestamp field without defining its clock semantics leaves the hard part unresolved.

Async Python offers asyncio.timeout_at() for a deadline expressed in the event loop's clock. Use loop.time() when constructing it. Check the library's cancellation behavior: timeout helpers can request cancellation while still waiting for the underlying coroutine to unwind. Blocking synchronous code inside the event loop can prevent the timeout machinery from running promptly.

Timeouts do not undo remote work

If a payment call times out, the payment may already have committed. The caller has learned that it did not receive a result in time. That is different from learning that the operation failed. Retrying requires an idempotency contract or a reconciliation path that can recover the original result.

Cancellation is similarly a request to stop work, not a distributed rollback. Propagate it to cooperative children, but assume a remote service may continue after the connection closes. For expensive background operations, expose an operation identifier and an explicit status or cancellation interface. Decide which work is allowed to survive the initiating request.

Before a retry, check both the remaining time and whether another attempt can improve the outcome. Backoff should fit inside the deadline; jitter does not create more time. A retry budget also limits total attempts across callers, since a locally reasonable policy can still amplify a dependency outage.

Measure where the budget disappears

Record queue duration, attempt duration, backoff duration, and remaining budget at dependency admission. Keep timeout, cancellation, and remote failure as distinct outcomes. A histogram of dependency latency alone cannot explain requests that expire before a dependency call starts.

Choose the response contract for partial results explicitly. A search page might return available sections with a clear incomplete marker; a purchase should not claim success before required commitments finish. The deadline tells the system when to stop waiting. Product semantics determine what it may truthfully return at that point.

Sources and further reading

• Python monotonic clocks
• Python asyncio deadlines and cancellation