Your Tests Pass and Your Verifier Is Broken: Designing CI for Asymmetric Failure

A green suite over a broken verifier

The bug was simple. Each SDK checked algorithm == "Ed25519" and had no branch for "Ed25519-batched", which is what every real proof carries. The check never ran, verified kept its initialised false, and the SDK reported that verification had failed.

Every unit test passed throughout. They tested that a tampered signature was rejected — it was. They tested that a malformed proof was rejected — it was. They tested that the client returned a well-formed result object — it did.

What no test asserted was that a real, valid, current proof returns true. The fixtures had been written when proofs were unbatched, so they were still valid inputs for a shape production had stopped emitting. The suite was faithfully testing a format that no longer existed.

Why verification fails asymmetrically

Most code fails loudly. A broken parser throws, a broken API returns 500, a broken query returns wrong rows you can eyeball. Verification is different: its failure mode is indistinguishable from its correct output.

verified: false is exactly what you want when a proof is genuinely bad. It is also what you get when the verifier never examined the proof at all. Same field, same value, same message — and the message is confidently negative:

A user seeing that assumes the proof is bad. They do not assume the verifier is. So the bug produces no bug reports, no error rate spike, no exception traces — nothing that monitoring is built to catch. It looks like the system working.

This generalises. Any code whose job is to say no will hide its own defects inside its expected output: authorisation checks, fraud rules, content filters, schema validators. If the only thing you assert is that bad input is rejected, a component that rejects everything passes every test you have.

The three gates that catch it

Assert the positive case with real data. The single most valuable test is that a genuine, current, production-captured artefact returns true. It is the only assertion the failure mode cannot satisfy. Capture it live rather than constructing it — a synthetic fixture tests your understanding of the format, not the format.

Include a case that only fails when a step is skipped. Our sharpest case keeps a real proof’s real, valid signature over its real Merkle root, then attaches sibling hashes that do not reconstruct to that root. Expected result: false. An implementation that checks the signature without first verifying inclusion returns true and fails — not because it computed something wrong, but because it skipped a step. Ordinary tests cannot see a missing step; they only see wrong answers.

Fail on exceptions, not just on wrong values. The contract is that verification returns false and never raises. So the harness counts three outcomes, and a raise is a failure in its own right:

PASS  conf_a_real_batched              verified=true
PASS  conf_f_siblings_wrong_root       verified=false
ERROR conf_c_bad_signature            threw TypeError

6/7 passed, 0 failed, 1 threw

Collapsing "threw" into "failed" loses the distinction between a verifier that judged wrongly and one that fell over — which are different bugs with different fixes.

One contract, not one per implementation

Twelve SDKs with twelve test suites gives you twelve opinions about what correct means. Each was internally consistent and all twelve were wrong in the same way, because each was written from the same mental model rather than from a shared executable definition.

The fix is that the definition lives outside every implementation and is derived from the server, not from any client. Ours is seven cases behind a dependency-free mock server, cloned at build time so there is one source of truth rather than twelve copies drifting apart:

- uses: actions/checkout@v4
  with:
    repository: forcedreamai/forcedream-sdk-conformance
    path: .conformance
- run: python3 .conformance/harness/mock_server.py &
- run: <language-specific harness>

Pointing each SDK’s api_base at localhost also means the test exercises the real transport — HTTP handling, JSON parsing, number coercion — rather than calling an inner function no user invokes. Several of the divergences we found lived in exactly that layer.

What it surfaced once it existed

The suite found seven remaining broken SDKs in one pass, and validated each fix as it was written rather than a session later. It also surfaced two things nobody had asked about.

A code path that had never executed anywhere. Every production proof to date carries a single Merkle leaf, so the sibling walk resolves trivially. Five fixes made before the suite existed were all validated against single-leaf proofs — correct for the case tested, unverified for the case written. The skipped-step case was the first thing to execute that branch in any language.

An implementation whose cryptography had never run. One SDK was written without package-manager access, so its key import and ASN.1 parsing had never touched a real signature. Both turned out correct — but "turned out" is doing the work in that sentence, and nobody knew until the suite ran.

If you maintain verification code, the question worth asking is not whether your tests pass. It is whether any of them would fail if the verifier simply returned false for everything.