The reason most multi-language SDKs drift is not that the implementations diverge, it is that the test suites do, because each port gets its own tests written by a different person on a different week, and from that moment on “we test the same thing in all three languages” is a claim nobody is actually checking. I watched this happen repeatedly in the years when every server framework promised a PHP port and a Ruby port with identical behavior, and the promise held for exactly one release, after which the ported tests quietly stopped covering the cases that were inconvenient in the second language. When we built the conformance layer for MongrelDB Kit, the TypeScript, Rust, and Python SDKs over the shared Rust core, the design rule was that no human would ever write the same test three times, because a test written three times is three tests, and three tests drift.

The fixture is the test

The suite lives in tests/conformance/ in the mongreldb-kit repo, and its center of gravity is a directory of language-neutral JSON fixtures, currently 23 of them, covering schema definition, inserts, updates, deletes, queries, aggregates, joins, CTEs, subqueries, FM-index contains filters, ANN and sparse vector search, encrypted tables, key encoding, views, and migration failure handling. Every scenario in every fixture has a name, and for every name there is an expected result under fixtures/expected/, so the entire contract between the engine and the SDKs is data, not code. A slice of inserts.json makes the shape obvious:

{"name": "insert_user_invalid_age", "table": "users",
 "row": {"id": 3, "email": "charlie@example.com", "role": "user",
         "age": -1, "handle": "charlieh", "active": true, "meta": null}}
{"name": "insert_post_missing_user", "table": "posts",
 "row": {"id": 3, "user_id": 99, "title": "No user",
         "slug": "no-user", "published": false}}

Both of those rows are supposed to fail, and the expected side says so in the same vocabulary every runner speaks:

"insert_user_invalid_age": {"error": "VALIDATION"},
"insert_post_missing_user": {"error": "FOREIGN_KEY"}

A runner’s whole job is load the fixture, drive the scenario through the public SDK surface, serialize what came back, and compare it against the expected entry by name, which means the interesting engineering is not in any runner but in the corpus itself, and adding coverage is editing JSON rather than writing three more test functions.

Errors are part of the contract

The part of the suite I would defend first is that failure behavior is pinned down with the same rigor as the happy path, because an SDK that returns the right rows but rejects the wrong ones is still a different database depending on which language you called it from. The runners never assert on message text, since messages are allowed to be idiomatic per language and should be; what they assert is a shared error vocabulary, a small set of codes like VALIDATION, DUPLICATE, FOREIGN_KEY, RESTRICT, MIGRATION, CONFLICT, STORAGE, and INTEGRITY, and each runner maps its own exception shapes onto that vocabulary before comparing. The Python runner’s version of the mapping is short enough to quote:

def error_code(exc):
    if hasattr(exc, "code"):
        return exc.code
    name = type(exc).__name__
    return {
        "ValidationError": "VALIDATION",
        "DuplicateError": "DUPLICATE",
        "ForeignKeyError": "FOREIGN_KEY",
        "RestrictError": "RESTRICT",
        "MigrationError": "MIGRATION",
        "ConflictError": "CONFLICT",
        "StorageError": "STORAGE",
        "IntegrityError": "INTEGRITY",
    }.get(name, "UNKNOWN")

So when insert_post_missing_user runs, the Rust runner sees its ForeignKeyError, the Python runner maps its exception to FOREIGN_KEY, and the TypeScript runner maps its thrown error to the same code, and what is being tested is that all three languages reject the same row for the same reason, not that they phrase it identically, which is exactly the distinction that matters in production and exactly the one that per-language test suites tend to lose first.

Three runners, one corpus, and the seams are written down

The Rust runner is a workspace member called conformance-runner that also builds as a standalone binary, which is handy when you want to reproduce a failure outside of cargo test; the TypeScript side is a vitest file that npm test picks up automatically; the Python side is a pytest module that loads the same JSON from the same directory. None of the runners reimplement assertions about semantics, they only adapt, and the adaptations that genuinely differ per language are documented in the suite’s README rather than hidden in helper code: the TypeScript runner normalizes the storage representation of NULL for nullable columns so logical behavior matches, and the Rust runner commits each mutating scenario before verifying state so its transactional isolation matches the other two. Those two lines in the README are doing real work, because a conformance suite with undocumented normalizations is just drift wearing a lab coat; the moment a runner quietly patches over a difference, the suite stops measuring the SDKs and starts measuring the patch.

The fixtures/schema.json file earns its keep here too, since the users, posts, and comments tables it defines carry unique constraints, check constraints, and foreign keys with cascade, set-null, and restrict actions, so deletes.json can pin down behavior most SDK test suites never touch: delete a user and the posts cascade, delete a post and its comments follow, try to delete through a restrict edge and every language has to refuse with RESTRICT, and all of that is verified against expected rows afterward rather than against a vague sense that the delete probably worked.

How a new behavior actually ships

The workflow the suite enforces is the part that pays rent over years: when the engine gains a behavior, say a new filter operator or a new index type, the behavior is not considered supported in the SDKs until there is a fixture for it and all three runners pass that fixture in the same CI run, which inverts the usual sequence where the TypeScript SDK ships the feature, Python follows a month later, and Rust gets it when someone files an issue. Because the corpus is shared, the cost of adding the scenario is one JSON edit plus one expected entry, and because the runners are thin, the failure output when one language lags is a named scenario with a named diff, not a stack trace through three layers of test helpers. That is the whole trick, and it is not really a trick: put the contract in data, make the runners stupid, let CI embarrass whichever SDK drifts first.

What the suite cannot prove

The honest tradeoff is that conformance proves semantics and nothing else, so it will not tell you that the Python API feels un-Pythonic, that the TypeScript types are awkward to compose, or that one binding is slower than the others on the same query, and those are real properties that still need per-language tests, benchmarks, and taste. The suite also only covers what someone bothered to write a fixture for, so the 23 files are a floor rather than a ceiling, and the discipline that keeps them honest is the same one that keeps any test suite honest: when a bug report comes in, the reproducer becomes a named scenario in the corpus before the fix lands, so the suite grows along the exact axes where reality has already hurt someone once. What you get in exchange for that narrowness is the one guarantee a three-language SDK actually needs, which is that the same schema, the same rows, and the same query produce the same result and the same error no matter which side of the stack asked, and that guarantee is mechanical, diffable, and rerun on every commit rather than restated in a README and hoped about.