P21: Semantic Verification Layer
Proves an acknowledged write really happened: a bounded-poll read-back that checks existence, state, exact values and parent links within a per-backend budget.
- Silent corruptions caught
- 4 / 4
- vs 0 / 4 for `is not None`, 1 / 4 for ==
- Budget overruns
- 0
- Across 2,000 runs
- Tests passing
- 94
01
Problem
- Who
- Teams whose writes go through eventually-consistent stores, third-party APIs or LLM agent tool calls.
- Why it mattered
- A write acknowledgement is a claim, not evidence: ghost writes, wrong lifecycle states and silent type coercion all return success, while a naive read-back either misses them or false-alarms on honest replication lag.
- Before
- `assert result is not None` or a single immediate `==` read-back.
02
Constraints
Technical
Never exceed the latency budget declared for each backend.
Technical
One runtime dependency (pydantic): the layer sits on the critical path of every write.
03
Architecture
- Component
- Key decision (click)
Specify
- VerificationSpecBuilt from the request, never the response
- Consistency profilesWindow, backoff, stable reads, jitter
Poll
- ReadBackAdapterPostgres, DynamoDB, S3, 3rd-party APIs
Check
- Four pillarsExistence · state · value · topology
- Type fidelitySTRICT · NUMERIC · LOOSE
Report
- VerificationReportEvery attempt, every mismatch
- Verification ConsoleFastAPI + WebSocket, React 18
Each iteration reads the object back, evaluates all four pillars, and classifies the result: PASS counts towards stable reads, CONVERGING sleeps and re-reads, DIVERGENT aborts at once with a typed exception.
Sleeps are clamped to the time remaining, so the final read lands exactly on the deadline: the backend gets its whole window and not a millisecond more.
04
Key decisions
D1A three-valued verdict drives the loop, not a timer
- Alternatives considered
- Retry until timeout
- Why
- A missing record may still converge; a wrong value in an immutable field never will, so polling it only delays the alert.
D2No generic retry library
- Alternatives considered
- tenacity or backoff
- Why
- P21 retries on a semantic verdict, must abort on a specific one, and needs deadline-clamped attempts, an injectable clock and a forensic record: more code to wrap than the ~90-line loop.
D3Injectable clock
- Alternatives considered
- Real sleeps in tests
- Why
- Tests assert the literal sleep schedule, the only way to prove a bounded poll is bounded.
05
Build highlights
- 01
Why == is not enough
Each of these is True in Python, yet the stored data has changed. STRICT fidelity catches all four.
pythonTrue == 1 # bool persisted as int 1000 == 1000.0 # int persisted as float Decimal("10.00") == Decimal("10.0") # money lost a decimal place Decimal("10.00") == 10.0 # exact decimal became a float - 02
Fluent verification spec
Expectations come from what the caller asked for.
pythonspec = ( expect("invoice", ack.id) .in_state("OPEN", transient={"DRAFT"}, terminal={"VOID", "PAID"}) .with_values({"amount_due": Decimal("100.00"), "currency": "USD"}) .attached_to(ResourceRef("customer", customer_id), via="customer_id") .under(Profiles.THIRD_PARTY_API) .build() ) report = await SemanticVerifier(adapter).verify(spec) - 03
Live Verification Console
15 scenarios run the real engine against a genuinely misbehaving store, streaming every attempt over a WebSocket; on 8 of them plain == reports success on data that is provably wrong.
06
Results
0
Overruns
2,000 ghost-write runs, 4 jitter schedules, W = 3 s
0.00%
Budget used on divergent defects
Wrong state, Decimal → float, wrong parent
0
False alarms
200 runs with an honest 900 ms lag
0.043 ms
Healthy-path cost
1.00 read-back on average
How this was measured: experiments/benchmark.py: large-N runs on a virtual clock, re-checked on the real asyncio clock; detection matrix on four corruptions.
07
Lessons
What worked
- Classifying failures as converging or divergent returned the whole budget on defects that can never heal.
- Building expectations from the request, not the response, is what makes a ghost write detectable.
What I'd do differently
- Tune each window from the observed p99.9 of convergence time; set it too tight and it manufactures false ghost writes.
08
Links & media
- Not publicThe source code is private.
