Configuration Changes Need a Rollout Protocol

Publish validated immutable snapshots, preserve one version within each request, and treat fleet adoption and rollback as observable state transitions.

Share
Glass configuration plates pass a brass validation aperture and branch toward five separate service modules.

A service reads its timeout setting, starts work, and reads its retry setting a moment later. An operator changes configuration between those reads. The request now uses a combination that never existed in any reviewed configuration. Both individual values are valid, yet together they exceed the intended latency allowance.

Configuration changes alter running behavior as surely as code deployments do. They need validation, version identity, an adoption boundary, and a rollback story. A shared dictionary updated one field at a time provides none of those properties by itself.

Publish a complete candidate

Build a candidate configuration separately from the active version. Parse types, reject unknown fields when appropriate, validate ranges, and check relationships between settings. An attempt count and timeout may each pass their own checks while their combined allowance violates an operational limit.

Distinguish syntactic validation from environmental validation. A URL can be well formed while pointing to the wrong service. A credential can be present while lacking permission. Some checks belong before rollout, while others need a controlled canary because they depend on real traffic. Avoid putting secrets or sensitive connection strings into validation error messages.

Give each accepted configuration a monotonically advancing version or an immutable content identity. Keep the reviewed candidate, its provenance, and the reason for activation. The running process should be able to report which version it adopted without exposing all configuration values.

Take one snapshot per unit of work

This executable Python model publishes immutable scalar settings under a lock. A request keeps the snapshot it acquired at entry. The lock protects publication and retrieval; the request does not hold it while performing business work. The version check prevents a stale operator update from silently overwriting a newer one.

from dataclasses import dataclass
from threading import Lock


@dataclass(frozen=True)
class Settings:
    version: int
    timeout_ms: int
    max_attempts: int

    def __post_init__(self):
        values = (self.version, self.timeout_ms, self.max_attempts)
        if any(type(value) is not int for value in values):
            raise ValueError("integer settings required")
        if self.version < 1 or not 50 <= self.timeout_ms <= 2000:
            raise ValueError("invalid version or timeout")
        if not 1 <= self.max_attempts <= 3:
            raise ValueError("invalid attempt limit")
        if self.timeout_ms * self.max_attempts > 3000:
            raise ValueError("combined attempt allowance is too large")


class Store:
    def __init__(self, initial):
        self.current, self.lock = initial, Lock()

    def snapshot(self):
        with self.lock:
            return self.current

    def publish(self, expected_version, replacement):
        with self.lock:
            if self.current.version != expected_version:
                raise ValueError("stale configuration update")
            if replacement.version != expected_version + 1:
                raise ValueError("version must advance by one")
            self.current = replacement


store = Store(Settings(1, 500, 2))
in_flight = store.snapshot()
store.publish(1, Settings(2, 800, 3))
assert in_flight == Settings(1, 500, 2)
assert store.snapshot() == Settings(2, 800, 3)
try:
    store.publish(1, Settings(2, 600, 2))
except ValueError:
    pass
else:
    raise AssertionError("stale writer overwrote configuration")
try:
    Settings(3, 2000, 3)
except ValueError:
    pass
else:
    raise AssertionError("invalid settings accepted")
assert store.snapshot().version == 2
store.publish(2, Settings(3, 500, 2))  # rollback values, new version
assert store.snapshot() == Settings(3, 500, 2)

The in-flight request continues using version one after version two is published. A new request sees version two. An invalid candidate cannot become active, and a stale update fails. Restoring the earlier values creates version three, preserving the historical fact that a rollback occurred.

A frozen dataclass prevents ordinary assignment to its fields; it is not a security boundary or a deep-freezing mechanism. This model contains only integers. If a configuration contains mutable lists or dictionaries, convert them to immutable representations or make defensive copies so a caller cannot mutate an allegedly stable snapshot through a nested reference.

A configuration candidate passes type and coupled-value validation, then a version comparison atomically publishes snapshot version 2. An in-flight request retains immutable version 1, while new requests acquire version 2. Invalid or stale candidates are rejected without replacing the active snapshot.
The lock protects local publication and retrieval. Each request uses one snapshot; fleet rollout, resource replacement, and emergency revocation require additional protocols.

Choose the adoption boundary

A request boundary is appropriate for many HTTP handlers. A long-running job may need one version for the whole job so its outputs remain reproducible, or a defined checkpoint where it can adopt changes. Re-reading arbitrary fields mid-operation makes incident reconstruction much harder.

Some settings cannot change safely in place. Connection-pool sizing, worker topology, or protocol negotiation may require constructing a replacement resource and draining the old one. Mark restart-required settings explicitly instead of pretending that updating a configuration object changes resources that were created at startup.

Emergency controls deserve their own semantics. A feature flag read once at request entry will not stop already-running work. If an operation must be revocable during execution, define a separate checked authorization or cancellation boundary. Mixing that requirement into every ordinary configuration read produces unpredictable partial adoption.

Roll out across the fleet

The local lock in the example does not coordinate multiple processes. A control plane needs durable version storage and a mechanism for distributing candidates. Each instance should acknowledge successful validation and adoption, and continue on an explicitly chosen last-known-good version if an update cannot be applied.

Start with a small, stable cohort and compare concrete signals such as error rate, latency, and resource usage. Stable assignment matters: randomly selecting a new cohort on every request can spread a risky configuration much further than its nominal percentage suggests. Keep the selected version visible in traces or logs with bounded-cardinality fields.

Define whether mixed versions are acceptable during rollout. Changes to a shared protocol, cache format, or durable data interpretation may require compatibility work first. A configuration toggle cannot make an incompatible storage transition safe merely because it can be switched quickly.

Rollback values without erasing history

Rollback should publish a new version containing known-good values, with the same validation and adoption checks as any other change. Reusing an old version number confuses caches, compare-and-publish checks, and incident timelines. It also hides whether a process observed the failed candidate before returning to earlier behavior.

Restoring settings does not undo side effects already produced under them. If a changed retention period deleted data, reverting the value cannot restore those records. Identify irreversible effects before activation and require a recovery plan appropriate to those effects, rather than treating every configuration change as trivially reversible.

Test invalid candidates, competing writers, in-flight work, delayed instances, failed resource replacement, and rollback after partial adoption. The useful operational question is not only which configuration is desired. It is which version each unit of work actually used, and whether that version was safe to combine with the surrounding system.

Sources and further reading