One Bug, Twelve SDKs: Building a Cross-Language Conformance Suite for Cryptographic Verification

The bug that shipped twelve times

Every ForceDream agent execution produces an Ed25519-signed proof. The SDKs exist so a caller can verify that proof locally, without asking ForceDream whether it is valid — the signature math decides, in the caller’s own process.

Every SDK contained this shape:

if algorithm is None or algorithm == "Ed25519":
    verified = verify_signature(public_key, signature, digest)
# no else branch

Correct, as far as it goes. The problem is that the production server had moved to batched proofs, and every real proof now carries algorithm: "Ed25519-batched". That value matches neither condition. The branch never executed, verified retained its initialised false, and the SDK returned:

For a proof that was never checked at all. Not a false negative from a failed comparison — a verdict with no verification behind it. For a product whose entire proposition is trustless verification, this is the worst available failure mode: it reports the strongest possible negative claim on the strongest possible evidence, having examined nothing.

Why twelve implementations failed identically

The instinct is to look for a shared dependency. There isn’t one. These are twelve separate codebases using twelve different cryptographic libraries — cryptography in Python, ureq and ed25519-dalek in Rust, CryptoKit in Swift, NSec in C#, Erlang’s :crypto in Elixir, java.security in Java and Kotlin. No shared code path exists to carry a bug between them.

What was shared was the reasoning. Each SDK was written by porting the same mental model: fetch the proof, rebuild the signable payload, hash it, verify the signature. Each author checked their work against the same reference implementation. Each was correct on the day it was written. And every one of them encoded the same assumption — that algorithm would be Ed25519 — which quietly stopped being true when the server introduced batching.

This is the failure mode that matters for anyone shipping a multi-language SDK family. Code review catches implementation defects. It does not catch twelve people independently making the same correct-at-the-time assumption, because each individual review passes. The only thing that catches it is an executable definition of correct that lives outside every implementation.

There is a second-order lesson too. The bug was found in one SDK and fixed there. Then found in a second and fixed there. Five SDKs were fixed one at a time, each fix validated by hand against two real proofs, before anyone asked whether the other seven had the same problem. They did. Fixing sequentially without a shared oracle is how a systemic defect gets mistaken for a series of individual ones.

The contract, read from the server rather than inferred

A conformance suite is only as good as its definition of correct. Deriving that definition from any SDK would have encoded the bug into the test. So the contract was read directly from the production verification path — wfVerifyProofDigest and verifyMerkleInclusion in the server source — and reduced to four rules every SDK must satisfy:

1. Reject unless algorithm is exactly Ed25519 or Ed25519-batched.
2. Batched: verify Merkle inclusion of the digest against the claimed root before checking the signature, then verify the signature over the root, not the digest.
3. Plain: verify the signature directly over the digest.
4. Failure returns false. It never raises.

Rule 2 is the security property, and the ordering within it is not stylistic. Verifying the signature first would establish only that ForceDream signed some root. It would say nothing about whether the digest in front of you is a leaf of that tree. An implementation that checks the signature and skips inclusion will happily accept a valid signature over an unrelated root as proof of an arbitrary digest.

Merkle reconstruction walks inclusion_proof.siblings in order. Each sibling carries its own position, so ordering is never derived from index arithmetic:

current = leaf_digest
for step in siblings:
    if step["position"] == "right":
        current = sha256_hex(current + step["hash"])
    else:
        current = sha256_hex(step["hash"] + current)
return current == expected_root

Two details cause silent divergence across languages. Hashing is over concatenated hex strings, not raw bytes — a natural mistake in typed languages where byte concatenation is the idiomatic choice. And an empty sibling list means the root is the leaf digest, unchanged, which several implementations would otherwise treat as an error case.

Seven cases, two of them real

The suite is deliberately small. Seven cases, of which two are real proofs captured live from the public proof endpoint and five are derived from those by local mutation. Nothing is synthesised from scratch, because a synthetic proof tests your understanding of the format rather than the format itself.

Case F is the one that earns its place. It keeps the real proof’s real, valid signature over its real Merkle root, then attaches siblings that do not reconstruct to that root. An implementation that verifies the signature without first checking inclusion returns true — and is wrong. It is the only case that executes the sibling walk at all.

Case C matters for a subtler reason: it flips one byte in the middle of the signature, keeping it a structurally valid 64-byte Ed25519 signature. An implementation that rejects it on length or encoding passes for the wrong reason. This forces the rejection to come from the cryptography.

The harness is a mock server, not a fixture loader

The obvious design is to load fixtures from disk and call each SDK’s verification function directly. That was rejected for two reasons.

First, most SDKs expose verification only through a network-fetching entry point — verify_by_task_id() or equivalent — because that is how callers actually use them. Testing a private inner function tests something no user invokes.

Second, a fixture loader skips the parts that genuinely differ across languages: HTTP handling, JSON parsing, and number coercion. That last one is a real hazard here. In the signed payload, started_at arrives as a JSON number while completed_at arrives as a string containing the same kind of value. Any language that normalises those differently produces a different canonical payload, a different digest, and a verification failure with nothing to do with the cryptography.

So the harness is a mock server implementing the two real routes, in dependency-free standard-library Python:

python3 harness/mock_server.py        # serves on 127.0.0.1:8787
python3 harness/mock_server.py --list # print cases without serving

Each SDK points its api_base at localhost and runs its complete real path. Twelve harnesses, one per repo, each invoking that language’s genuine public interface:

cargo run --example conformance                 # Rust
dart run tool/conformance.dart                 # Dart
mix run examples/conformance.exs               # Elixir
php examples/conformance.php                   # PHP
mvn exec:java -Dexec.mainClass=...Conformance  # Java, Kotlin

What it found once it existed

The suite was validated first against an SDK already believed correct. If the harness disagreed with a known-good implementation, the harness was wrong. Python returned 7/7, so the oracle was trustworthy before it was allowed to judge anything else.

It then found the remaining seven broken SDKs in a single grep, and validated each fix the moment it was written rather than a session later. Fixes that had previously taken hours of manual reasoning each took one pass.

Two findings were more interesting than the bug itself.

The sibling walk had never executed in any language. Every real proof the platform has emitted carries batch_size: 1 with an empty sibling list, so the Merkle walk resolves trivially. The five fixes made before the suite existed were each validated against those same two single-leaf proofs. They were correct for the case they were tested on, and entirely unverified for the case they were written for. Case F was the first thing to execute that code path in any implementation.

One SDK’s cryptography had never run at all. The C# implementation was written in an environment without NuGet access, so its Ed25519 path — the NSec key import and the ASN.1 SubjectPublicKeyInfo extraction — had never been executed against a real signature. The file said so in a header comment. Both turned out to be correct, but nobody knew that until the suite ran.

Running it yourself

The suite is public. Cloning it and reproducing every claim in this post takes about two minutes:

git clone https://github.com/forcedreamai/forcedream-sdk-conformance
cd forcedream-sdk-conformance
python3 harness/mock_server.py &
python3 harness/harness_python.py /path/to/forcedream-sdk-python

Every official SDK repo runs the same seven cases in CI on every push, cloning the specification at build time so there is one definition of correct rather than twelve copies drifting apart:

- uses: actions/checkout@v4
  with:
    repository: forcedreamai/forcedream-sdk-conformance
    path: .conformance
- run: python3 .conformance/harness/mock_server.py &
- run: <language-specific harness>

If you are building your own multi-language SDK family, the transferable parts are: derive the contract from the server rather than from any client; capture real fixtures rather than synthesising them; test through the public interface over the real transport; and include at least one case that fails only if a step is skipped rather than done wrong.

What the suite still cannot prove

Case F proves each SDK performs the Merkle walk and correctly rejects an inclusion path that does not reconstruct. No case proves an SDK accepts a correct multi-sibling path, because constructing one requires the private signing key.

Production proofs currently carry batch_size: 1, so no live proof exercises multi-sibling acceptance. The batch ceiling is 200 and larger batches are structurally supported; they simply have not occurred yet. Acceptance testing will be added when the platform emits such a proof. Simulating one would test our own understanding rather than the system, which is precisely the failure this suite exists to prevent.

That gap is documented in the repository README rather than left for someone to discover. A conformance suite that claims complete coverage is making exactly the kind of unverified assertion it was built to catch.