File Uploads Need a Quarantine State
Bind approval to an immutable object version so a slow scanner cannot approve replacement bytes, and keep access gated by explicit upload state.
An upload endpoint saves a file and immediately returns its public URL. A background worker scans it a few seconds later. Even if the scanner detects a problem, the file was already accessible during the interval. Moving the scan into the request helps with that interval but introduces another race: the bytes can change between inspection and publication.
The useful unit of approval is a particular object version, not a filename. Model the upload as a state transition from quarantine to an approved or rejected result. The serving path must enforce that state, and the scanner's result must identify exactly which bytes it examined.
Separate storage from permission to serve
Accept uploads into private storage under server-generated names. Associate the object with the authenticated owner, a size limit, an expected use, and an immutable version identifier. The original filename can be display metadata after normalization; it should not decide a storage path or an authorization boundary.
Client-supplied content types and file extensions are useful hints, not sufficient validation. Check the allowed formats, inspect content with appropriate parsers, and bound the resources those parsers can consume. Archives require limits on expansion and nesting as well as compressed size. A file that fits the transport limit can still be expensive to decode.
Quarantine must remain private even when metadata is temporarily unavailable. A storage URL that bypasses the application gate defeats the state model. If approved files receive signed download URLs, define their expiry and revocation behavior; changing a database row does not necessarily invalidate a URL already issued by another component.
Bind a scan result to a revision
This Python and SQLite model stores metadata only. The byte strings are fixtures used to calculate digests; it does not upload objects or detect malware. A real object store would keep each revision at an immutable key or version, and the scanner would read that exact object. Calls here are serialized on one connection.
import hashlib
import sqlite3
db = sqlite3.connect(":memory:")
db.execute("""CREATE TABLE uploads (
id TEXT PRIMARY KEY NOT NULL,
revision INTEGER NOT NULL,
digest TEXT NOT NULL,
state TEXT NOT NULL CHECK (state IN ('quarantine','ready','rejected'))
)""")
def stage(upload_id, content):
digest = hashlib.sha256(content).hexdigest()
with db:
db.execute("""INSERT INTO uploads VALUES (?, 1, ?, 'quarantine')
ON CONFLICT(id) DO UPDATE SET
revision = uploads.revision + 1,
digest = excluded.digest, state = 'quarantine'""",
(upload_id, digest))
return db.execute(
"SELECT revision, digest FROM uploads WHERE id = ?", (upload_id,)
).fetchone()
def finish_scan(upload_id, revision, digest, passed):
# Only a trusted scanner may invoke this transition.
state = "ready" if passed else "rejected"
with db:
changed = db.execute("""UPDATE uploads SET state = ?
WHERE id = ? AND revision = ? AND digest = ?
AND state = 'quarantine'""",
(state, upload_id, revision, digest)).rowcount
return changed == 1
old = stage("photo-1", b"old bytes")
new = stage("photo-1", b"replacement bytes")
assert not finish_scan("photo-1", *old, passed=True)
assert db.execute("SELECT state FROM uploads").fetchone()[0] == "quarantine"
assert finish_scan("photo-1", *new, passed=True)
assert not finish_scan("photo-1", *new, passed=False)
bad = stage("photo-2", b"scanner fixture")
assert finish_scan("photo-2", *bad, passed=False)
assert db.execute("SELECT state FROM uploads WHERE id='photo-2'").fetchone()[0] == "rejected"
The first scan result arrives after replacement bytes have been staged. Its revision and digest no longer match the current row, so it cannot promote the replacement. Only a matching result can move the current quarantine entry to ready or rejected. Repeated or contradictory reports for a completed revision do not silently change the decision.
The revision prevents confusion when a logical upload is replaced; the digest binds the report to content. A digest alone is not a clean bill of health. It says that two byte sequences match under the chosen hash, not that the bytes are safe. The trusted scanner's policy and execution still determine the meaning of approval.
Close the storage and database gap
An object store and a relational database usually do not share a transaction. Write a new private immutable object first, then commit metadata referring to its exact version. If the metadata commit fails, an orphan collector can remove unreferenced objects after a conservative grace period. Never reuse the same mutable object key for bytes that an earlier scan might still be reading.
The scanner should claim work with the object version, policy version, and upload revision. Its result should include those identities and an outcome. Publishing the result uses a conditional database update like the example. If an upload record can be deleted and recreated, use a new unpredictable identity or a generation that does not reset; reusing revision one can otherwise revive old work.
Serving approved bytes should resolve through the same immutable version that passed inspection. Copying an object to another location needs version-aware copy semantics and error handling. A failed copy must not leave metadata announcing an accessible artifact that never arrived. For many applications, retaining immutable storage and changing only the authorization pointer is simpler.
Treat scanning as fallible work
A scanner timeout is not a pass. Keep the upload quarantined with a retryable reason, and distinguish that operational failure from a content rejection. Set attempt and elapsed-time limits so unavailable scanning infrastructure cannot accumulate an invisible queue forever. Expose a useful pending status rather than making the client repeatedly upload the same bytes.
Inspection may include antivirus, format validation, metadata removal, or conversion to a safer representation. Each operation has different limits. Image re-encoding can reduce certain risks but is not a universal sanitizer, and the decoder itself must run with appropriate isolation and resource limits. Avoid uploading private files to public scanning services without a deliberate data-sharing decision.
When policy changes, decide whether existing approvals remain valid or require rescanning. Store the policy version so the choice is implementable. A new scanner result should not overwrite a newer approval policy accidentally; include policy identity in the conditional transition when several policy generations can coexist.
Test the transitions users cannot see
Exercise replacement during scanning, duplicate reports, scanner failure, metadata failure after object creation, and an attempted download while quarantined. Check that cleanup cannot delete an object still referenced by an approved revision. These cases test the upload lifecycle, which a successful happy-path upload does not cover.
Track time spent in quarantine, stale scan results, rejected content, retry exhaustion, and orphan cleanup. The state machine gives each component a narrow responsibility: storage preserves identified bytes, scanning evaluates those bytes, metadata records the accepted result, and the download path enforces it.