Why middleware auth is the wrong layer for a database file

The classic shape, and I wrote this shape myself in the mysql_* era more times than I want to admit, is that the application holds one shared database password, every request funnels through application code that checks the user’s session before it touches the connection, and the database itself trusts anyone who shows up with the one password it knows. That model survives contact with production exactly as long as every path to the data goes through the application, which is true right up until someone opens a REPL against the production file, or a second service gets added with its own connection logic, or a backup script starts reading raw pages, and at that point your permission model is a comment in a wiki page rather than a property of the system.

Middleware auth also has a layering problem that gets worse the more surfaces a database exposes, because MongrelDB is not just a daemon; it is an embedded library, a CLI, a set of SDKs in a dozen languages, and an HTTP server, all over the same on-disk format, and if auth lives in the HTTP layer then the embedded open path has no auth at all unless you build a second auth system for it, and now you have two systems to keep honest instead of one.

Users and roles live in the catalog

MongrelDB stores users, roles, and grants inside the database catalog itself, the same _meta blob that holds table schemas, procedures, and triggers, so there is no separate auth file to back up, no external user database to keep in sync, and no drift between “who the app thinks exists” and “who the engine thinks exists.” Passwords are hashed with Argon2id at the OWASP-recommended parameters (19 MiB memory, time cost 2, parallelism 1), which is the same construction the engine uses to derive the key-encryption key for encrypted databases, and plaintext passwords are never stored or logged anywhere in the system.

The permission model is deliberately small: All, Admin, Ddl, and per-table Select, Insert, Update, and Delete, matched literally with no wildcard syntax, so select:* grants access to a table actually named * and nothing else. Two details are worth calling out because they are the ones people get wrong when they skim: a principal with the is_admin flag short-circuits every check, which is how the bootstrap admin works, and Permission::All explicitly does not imply Admin, so granting someone everything on every table still does not let them create users or hand out grants, which is the separation you want between “can touch all the data” and “can mint new identities.”

The SQL frontend speaks the standard vocabulary, so nothing here requires learning a bespoke control language:

CREATE USER alice WITH PASSWORD 's3cret-pw';

CREATE ROLE analyst;
GRANT SELECT ON orders TO analyst;
GRANT INSERT ON orders TO analyst;
GRANT analyst TO alice;

SHOW USERS;
SHOW ROLES;

What require_auth actually enforces

By default the catalog auth is advisory, meaning the engine stores users and permissions and your application can consult them with check_permission, but reads and writes are not gated, which keeps MongrelDB drop-in compatible with the SQLite-style “open the file and go” workflow. The interesting mode is opt-in per database: set require_auth = true in the catalog and the storage layer itself starts enforcing, which changes the failure semantics in a way middleware never can.

use mongreldb_core::Database;

// Create a credentialed database with a bootstrap admin.
let db = Database::create_with_credentials("./secure_db", "admin", "s3cret-pw")?;

// Later: plain open fails with AuthRequired.
// Database::open("./secure_db")?  -> Err(AuthRequired)

let db = Database::open_with_credentials("./secure_db", "alice", "alice-pw")?;
// Every Table, Transaction, and SQL operation on this handle is now
// checked against alice's permissions. A missing grant returns
// PermissionDenied from the engine, not from a wrapper.

With enforcement on, a Table::put requires Insert on that table, an SQL UPDATE requires both Update and Select, DDL and compaction require Ddl, user and role management requires Admin, and the daemon’s HTTP auth middleware stops being the only line of defense, because the storage layer re-checks the resolved principal on every operation even after the middleware has already accepted the request. That is the defense-in-depth shape you actually want: the HTTP layer asserts who you are, and the engine enforces what that identity may do, so a bug in either layer alone does not silently open the whole file.

The error surface maps cleanly onto HTTP when the daemon is involved, which keeps client code boring: AuthRequired and InvalidCredentials come back as 401, PermissionDenied comes back as 403, and AuthNotRequired is a 400 for the case where you passed credentials to a database that does not want them.

What it costs

The honest cost numbers are small and they sit in the right places. One Argon2id verification is tuned to land around 50 milliseconds on current server hardware, and you pay it once per open or once per authenticated HTTP request, after which the engine caches the resolved principal and every per-operation check is a cheap permission comparison against an Arc-cloned AuthState handle rather than another hash. The cached principal is worth one operational caveat: a revoked grant does not bite on an already-open handle until the next authentication, which is why the engine exposes refresh_principal() for the cases where you cannot wait for a reconnect, and why the daemon re-resolves the principal per request rather than per connection. The deliberate expense of the hash is the feature, not the overhead, because it caps online password guessing at roughly 20 attempts per second per core, though the v1 implementation does not add lockouts or throttling on top of that, so weak passwords remain weak passwords and no storage layer can fix that for you.

The composability story is the part I would have killed for in 2009: credential enforcement and page-level encryption are orthogonal, so create_encrypted_with_credentials gives you a database where the bytes on disk are AES-256-GCM ciphertext and the logical operations still require a catalog identity, and the two systems share nothing beyond the Argon2id primitive and the password that feeds it, which means a mistake in one does not weaken the other.

What it does not stop

This is the section most vendor posts skip, and skipping it is how you end up with a CVE and a sheepish blog post. Credential enforcement is a logical access control, not a cryptographic one, so an attacker with raw disk access who parses the file format directly can read the data without ever calling the API, and an attacker who can write to the _meta directory can flip require_auth off or rewrite the catalog, which is why the docs tell you plainly that filesystem permissions on the database directory are the real boundary and why the recovery path for lost credentials is “edit the catalog offline,” a path that grants no power an attacker with disk access did not already have. If your threat model includes the disk getting pulled, you want the encryption layer too, and if your threat model includes the machine itself, you want full-disk encryption or hardware keys underneath all of it, because a database file, no matter how carefully it hashes its passwords, cannot defend the host it runs on.

The modern-equivalent close here is simple: we spent twenty years putting auth in front of databases because databases were big shared servers with one front door, and the embedded engine breaks that assumption by having as many doors as there are processes that can open the file, so the auth has to move to the one place every door leads through, which is the storage layer itself.

Why most embedded engines stop at single-process

SQLite has WAL mode and it is genuinely good at what it does, but if you want change data capture you reach for the SQLite Update Hook or you parse the WAL file yourself, and neither of those gives you a durable, resumable stream with stable event identifiers that survive a restart; LevelDB and RocksDB expose iterators and you can tail the log, but the log is an internal artifact subject to compaction and the ordering guarantees are the iterator’s problem, not yours, and the moment you want transactional semantics, where a batch of writes commits atomically and your downstream consumer sees the whole batch or nothing, you are back to application-level bookkeeping. The pattern across the embedded-database landscape is the same: the engine optimizes for local read and write throughput, the replication story is delegated to external tooling, and the application developer who needed a simple embedded database ends up running a three-component pipeline to get changes out of it.

CDC derived from the WAL, not a second log

MongrelDB’s change data capture is not a separate log that the engine maintains alongside its data; it is a reconstruction of committed transactions from the same WAL that the engine uses for crash recovery, which means there is exactly one source of truth for what committed and in what order, and the CDC stream cannot diverge from the data because it is the data, just read differently. The API surface is a single function, change_events_since(last_event_id), which replays the WAL records, correlates each Op::TxnCommit with its staged operations, groups them by transaction, and returns a CdcBatch containing one ChangeEvent per committed transaction with a stable ID shaped as <commit_epoch>:<operation_index>, so a consumer can persist the last ID it processed and resume from exactly there on the next poll without missing or duplicating a transaction.

The design choice that matters here is that aborted transactions never appear in the CDC stream, because the reconstruction logic keys on Op::TxnCommit records, and a transaction that rolled back never wrote one, which means your downstream consumer never sees a write that did not durably commit; this is the guarantee that external WAL tailers have to reconstruct for themselves, and they get it wrong in the gap between the WAL entry and the commit marker, where a half-applied transaction looks like data until you cross-reference the commit log and discover it was never finalized. The engine test suite exercises this by staging a put, rolling it back, and confirming the CDC stream contains no put events for that transaction:

let mut transaction = db.begin();
transaction.put("items", vec![(1, Value::Int64(1))]).unwrap();
transaction.rollback();

assert!(db
    .change_events_since(None)
    .unwrap()
    .events
    .iter()
    .all(|event| event.op != "put"));

A rolled-back put produces no CDC event, full stop, and that is not a filter applied after the fact but a consequence of the reconstruction logic skipping any WAL records that lack a matching commit.

The gap signal, and why it matters

A derived CDC stream has one honest limitation that a separate log does not: the WAL segments are finite, and once the engine checkpoints and purges old segments to reclaim space, the records needed to reconstruct older events are gone. MongrelDB handles this by returning gap = true in the CdcBatch when the consumer’s resume point falls before the oldest retained commit, which is the engine saying “I cannot give you a continuous stream from where you left off, so do not pretend you have one”; the correct consumer response is to rebootstrap, not to silently skip ahead, because a gap in a change stream is a data integrity problem, not a performance optimization. The retention is tunable via set_replication_wal_retention_segments, and the test suite verifies the gap behavior by writing data, checkpointing to purge segments, then resuming from a now-evicted position:

db.set_replication_wal_retention_segments(1);
// ... write and checkpoint multiple transactions ...
let resumed = db.change_events_since(Some(&first_id)).unwrap();
assert!(resumed.gap);

The bounds are generous but real: at most one million WAL records replayed, 256 MiB of replay bytes, 100,000 events, one million rows, and 256 MiB of total retained bytes, after which the engine fails closed rather than silently truncating, because a change stream that drops events without telling you is worse than no stream at all.

Replication as WAL streaming

The replication surface is the same idea at a different granularity, because instead of reconstructing logical change events, the follower receives raw committed WAL records over HTTP and replays them into its own local database directory, which means the follower’s logical state at any given commit epoch matches the leader’s, not a best-effort approximation; the physical bytes on disk may diverge once the follower independently checkpoints and compacts on its own schedule, but the data you can read at a shared epoch is identical. The daemon exposes GET /wal/stream?since=<seq>, the follower polls it, and the ReplicationFollower client in mongreldb-client wraps the whole loop:

use mongreldb_client::ReplicationFollower;

let mut follower = ReplicationFollower::new("http://leader:8453", "/local/copy")?;
follower.sync()?;  // fetch + apply all new records since last sync
println!("caught up to epoch {}", follower.last_epoch());

The follower bootstraps from a snapshot if the local copy does not exist yet, refuses to overwrite a non-replica database directory, and advances its watermark epoch only after the records are durably appended and the local database has recovered, so a crash mid-sync does not corrupt the follower or leave it in a half-applied state; it simply resumes from the last confirmed epoch on the next sync() call. This is asynchronous leader-to-follower replication designed for read scaling and for warm standby copies, distinct from the synchronous Raft consensus replication that MongrelDB also supports in cluster mode, and the two coexist because they solve different problems: Raft gives you strong consistency across a quorum for write availability and zero-data-loss failover, while WAL streaming gives you a geographically distant read copy or an analytics replica without paying the latency of a cross-region quorum ack. The honest tradeoff is that async streaming has an RPO greater than zero, because writes the leader committed but had not yet streamed are lost if the leader dies before the next poll, and if you need RPO 0 that is what the Raft quorum path is for; the async follower is the right tool for read scaling and for a recovery target that tolerates a small amount of tail loss, not a substitute for consensus-backed durability.

What changes and what stays the same

The thing that changes when an embedded engine is also a replicating engine is that the WAL stops being a purely internal recovery artifact and becomes a public contract, because external consumers depend on its ordering, its commit semantics, and its retention window, and those properties now have to hold across versions and across restarts. The thing that stays the same is the commit path itself, because the engine does not do extra work on the write side to produce the CDC stream or feed the replication follower; it writes the same WAL records it would write anyway, and the consumers read them after the fact, which means there is no throughput tax on writes for the benefit of change data capture until somebody actually calls change_events_since or polls /wal/stream. The cost moves to the read side, where WAL reconstruction and network transfer do real work, and to the storage side, where WAL retention has to be wide enough to cover your consumers’ polling interval, and those are knobs you turn based on your actual workload rather than overhead you pay whether you use the feature or not.

The modern equivalent of an old pattern

In the MySQL world, if you wanted change data capture in 2010, you parsed the binlog, and if you wanted replication, you configured a replica and hoped the binlog position tracking held up across schema changes and version upgrades, and the binlog was a separate log from the InnoDB redo log, which meant there were two sources of truth for what committed and they could disagree in edge cases around crash recovery. MongrelDB collapses that to one log: the WAL is the redo log and the replication log and the CDC source, and there is no second log to keep in sync, no binlog position to lose, and no external connector to reverse-engineer your commit semantics, because the engine that writes the data is the same engine that tells you what changed.

Why middleware idempotency is not enough

The middleware version of this pattern is well understood, and it is also where the pattern quietly breaks. A reverse proxy or an application-level interceptor can dedupe retries that pass through it, but it cannot dedupe a retry that arrives after the process crashed, because its dedupe table is usually in-memory or in a cache with weaker durability than the database itself, and a cache entry that vanishes on restart is a promise that expires at exactly the wrong moment. The worst case is the gap between commit and response, where the database has durably applied the write and the middleware never learned that fact; the client retries, the middleware has no record, the database executes again, and you are back to two rows. You can paper over this with a dedupe table inside the same database, and people do, but then you are hand-rolling transactional semantics around a unique constraint and hoping you got the race conditions right, which is precisely the kind of work a storage engine should be doing for you.

What MongrelDB does instead

The core transaction API takes an optional idempotency request at commit time, carrying three things: the key itself, an owner string so two tenants can never collide on the same key, and a fingerprint of the request so the engine can tell a genuine retry apart from a key being reused for different work. When the transaction commits, the engine writes the idempotency record into a durable ledger that is anchored to the commit log, which means the key-to-receipt binding survives a restart under the same authority as the data pages, and an unkeyed commit never creates the ledger file at all, so you pay nothing for the feature when you do not use it.

The behavior on retry is the interesting part. The idempotency check runs inside the commit path before the transaction’s writes are ever proposed to the log, so if a commit arrives with a key and fingerprint that already have a completed record, the engine does not re-apply the staged writes or append anything new; it replays the original commit receipt, the client sees the same transaction identity it would have seen the first time, and the row count stays exactly where the first commit left it. If a commit arrives with a key that exists but a different fingerprint, the engine aborts with a conflict error, because reusing a key for different work is a bug in the caller and the correct response is to refuse, not to guess. If the key has a time-to-live and the record has expired, the record is swept and the key becomes available again, which keeps the ledger bounded for workloads like checkout flows where the retry window is minutes, not days.

In the Rust core that looks like this, lifted from the engine’s own test suite:

let mut first = db.begin();
first.put("t", vec![(1, Value::Int64(1))])?;
let epoch = first.commit_idempotent(IdempotencyRequest {
    key: "checkout-42".into(),
    owner: "store-api".into(),
    fingerprint: request_fingerprint,
    ttl: None,
})?;

// Network drops, client retries the same logical request.
let mut retry = db.begin();
retry.put("t", vec![(1, Value::Int64(2))])?;
let retry_epoch = retry.commit_idempotent(IdempotencyRequest {
    key: "checkout-42".into(),
    owner: "store-api".into(),
    fingerprint: request_fingerprint,
    ttl: None,
})?;

assert_eq!(retry_epoch, epoch);      // same receipt, no re-execution
assert_eq!(row_count(&db, "t"), 1);  // one row, not two

The daemon layer builds the same guarantee into the HTTP surface, because a write that goes through mongreldb-server deserves the same crash safety as a write through the embedded API. Buffered SQL writes accept an Idempotency-Key header or an idempotency_key field in the JSON body, the daemon durably publishes an intent before it executes anything, and after a known terminal outcome it publishes a receipt, with a file fsync and atomic rename on Unix and write-through replacement on Windows. The awkward truth of that design is that the commit and the receipt cannot be one atomic operation across a filesystem and a database, so if the process dies in the narrow gap between them, the daemon refuses to guess; a retry with the same key gets a non-retryable QUERY_OUTCOME_UNKNOWN, the intent stays on disk as a tombstone, and an operator verifies the outcome before recovery, because auto-expiring that tombstone into a fresh execution is how you charge a credit card twice. Receipt and intent files hold HMAC-authenticated hashes, never the raw key or the SQL, and on encrypted databases the HMAC key is derived from the database’s own key-encryption key, which means the idempotency store is authenticated by the same root of trust as the data itself and does not become a side channel.

From the client side

None of this matters if using it is painful, so the PHP client threads the key straight through the transaction surface, and the whole retry-safety story collapses into one extra argument at commit:

$txn = $db->beginTransaction();
$txn->put('orders', ['id' => 1042, 'total' => 9900, 'status' => 'paid']);
$txn->put('inventory', ['sku' => 'WIDGET-1', 'on_hand' => 41]);

// Safe to call again on timeout or connection drop:
$results = $txn->commit(idempotencyKey: 'order-1042-payment');

The Transaction::commit() signature takes the key as an optional parameter and drops it into the idempotency_key field of the batch payload, so the retry path in your application is just “catch the connection exception and call the same code again,” and the dedupe work happens where the fsyncs happen. The batch itself is atomic, so the order row and the inventory decrement commit together or not at all, and the idempotency record commits with them, which closes the gap that middleware cannot close.

What it costs

The honest tradeoff is capacity and bookkeeping. The embedded engine’s ledger is a single checksum-framed file, optionally sealed with the metadata key on encrypted databases, and it is bounded by two hard limits: at most 65,536 records, with the oldest committed records evicted first and in-flight reservations never evicted, and a 64 MiB cap on the file itself, beyond which the engine fails closed rather than risk a corrupted or replayed record. A key with no explicit TTL is kept until that bounded-size eviction takes it, so an embedded application that keys every write will eventually cycle the oldest records out, while a keyed write with a TTL sweeps on schedule. The daemon layer adds its own bounds on top, because one server may speak for many databases and many tenants: MONGRELDB_SQL_IDEMPOTENCY_MAX_ENTRIES (4,096 entries by default) caps the store, completed receipts expire after MONGRELDB_SQL_IDEMPOTENCY_TTL_SECS (24 hours by default, a safety bound rather than the expected retry window, which for a checkout flow is minutes), and each owner is capped at a quarter of the total capacity so one noisy tenant cannot starve the rest; when the store is full the daemon fails closed with a capacity error rather than evicting a receipt someone might still retry against. Crash-left intents consume capacity until an operator resolves them, which is deliberate, because an unresolved intent is an unresolved question about your data and the correct behavior is to keep asking it. There is also a small amount of extra fsync work per keyed write for the ledger mutation, which is real but bounded, and it is the price of making the retry guarantee survive power loss instead of merely surviving a TCP reset.

The modern equivalent of an old habit

In the mysql_* era the standard advice was “make your writes idempotent by design,” which in practice meant a unique key on a natural column and an INSERT ... ON DUPLICATE KEY UPDATE, and it worked, but only for the shapes of write that fit a single-row upsert, and only if you remembered to design the table that way up front. Storage-layer idempotency keys are the same instinct generalized: instead of contorting your schema so duplicates are harmless, you name the logical request once and let the engine recognize the retry, whatever shape the write takes, whether it is one row or a forty-operation batch. The guarantee moved down the stack, which is where guarantees like this have always belonged.

What one fsync buys you

A transaction in MongrelDB commits when its records are appended to the WAL and that WAL segment is fsync’d to disk, full stop. The data pages, index pages, and column metadata are updated in memory first and flushed to disk afterwards by a background writer, but they do not block the commit. If the process crashes right after the fsync, recovery replays the WAL from the last checkpoint and reconstructs the same pages; if the process crashes before the fsync, the transaction never happened. There is no intermediate state where the database is confused, because the WAL is the only source of truth that matters at crash time.

The latency win is straightforward. A single fsync on a sequential WAL append is one disk seek on a rotating drive or one flush on an SSD, and on modern NVMe it is typically sub-millisecond. Writing and fsyncing the actual data pages would mean one or more random writes per modified page, each with its own rotational latency or flash write-amplification cost, and those costs multiply quickly on a transaction that touches many rows or many indexes. By separating “must survive crash” from “nice to have on disk,” the commit path stays flat even when the transaction is large.

Throughput benefits too, because group commit can share one fsync across multiple transactions. When several transactions are ready to commit at the same time, MongrelDB batches their WAL records into the same segment and pays one fsync for the batch instead of one per transaction. The published cargo bench -p mongreldb-core --bench write_path numbers show group commit at roughly 686 µs for 1,000 rows, which is the fsync cost amortized across many writers; the single-row no-fsync put is around 618 ns, and the durable commit is around 6.79 µs, because that measurement is the fsync-bound path. The difference between a single-row commit and a group commit is how many transactions ride inside that single disk barrier.

What the rest of the storage layer does asynchronously

Data pages are written out by a background thread that watches dirty-page counts and checkpoint intervals, and the rule it follows is simple: keep the WAL from growing forever, but never let a page reach disk before the WAL record that describes its change. That ordering is what makes recovery possible; if a data page were written before its WAL record, a crash between the two would leave an updated page with no log to explain it. The background writer respects the log-sequence-number ordering, so pages are flushed only after their covering WAL records are durable.

Checkpoints are the mechanism that bounds recovery time. A checkpoint marks a point in the WAL before which all dirty pages are known to be on disk, which means recovery only has to replay from the last checkpoint rather than from the beginning of time. MongrelDB checkpoints on a configurable interval and on dirty-page thresholds, and the checkpoint itself is a lightweight metadata write; the heavy work of flushing pages happens continuously in the background, so checkpoint stalls are small.

The tradeoffs that come with this design

Recovery time is the obvious one. Because data pages are not necessarily on disk at commit time, a crash means replaying every WAL record since the last checkpoint, and the length of that replay is the product of your write rate and your checkpoint interval. On a write-heavy workload with a long checkpoint interval, recovery can take seconds or minutes, which is fine for most applications but unacceptable for a system that needs sub-second restart. The fix is shorter checkpoints, but shorter checkpoints mean more foreground disruption and more write amplification; that is the standard WAL checkpoint tradeoff and MongrelDB does not pretend to have escaped it.

The second tradeoff is that the single fsync is still the latency floor for synchronous commits, and on a filesystem or storage stack with slow fsync behavior, that floor is visible. Some filesystems stall fsyncs while other writes drain, some virtualized disks lie about fsync completion, and some network-attached storage introduces multi-millisecond latency on every barrier. MongrelDB cannot make a bad disk fast, but it can make sure you only pay that cost once per transaction rather than once per page.

The third tradeoff is that group commit adds jitter. A transaction that is ready to commit just after a group has been dispatched waits for the next fsync window, and that wait is bounded by the group-commit timeout. For latency-sensitive single-row writes, the default timeout is short enough that the jitter is small; for throughput-bound workloads, the batching is the whole point. The settings are exposed if you need to tune them, but the default is biased toward the common case where one slow fsync is worse than a few microseconds of scheduling delay.

Why this is the modern equivalent of an old pattern

In the simpler embedded engines of that era, the way to get durability was often to open the data file with O_SYNC or O_DIRECT and pay the synchronous write cost on every page, because the code path was simpler and the gap between WAL and direct-write was smaller on the hardware available then. Modern equivalent is WAL fast-commit: the same crash-recovery guarantee, but the synchronous work is concentrated in one sequential append instead of scattered across random writes, and the storage layer can schedule the rest of the work when the disk is least busy. The guarantee did not change; the shape of the work did.

Asymmetric visibility and the shape of the client

The Database class is the front door of the client, and it carries internal state that callers should read but never set: the base URL, the default transport, the current session token, and a few cached metadata objects. Before PHP 8.4, the honest choices were either public readonly properties, which expose the value for reading but still let anyone assign it during construction if the constructor is inside the class, or a pile of private properties with boilerplate getters that exist only to say “this is not for you.” Asymmetric visibility fixes this by letting the property be public private(set), which means the world can read it and only the class can write it after construction, so internal mutation in methods like withToken still works while external callers are frozen out. This is exactly what a Database object needs.

final class Database
{
    public function __construct(
        public private(set) string $baseUrl,
        public private(set) TransportInterface $transport,
        private ?string $sessionToken = null,
    ) {}

    public function withToken(string $token): static
    {
        $clone = clone $this;
        $clone->sessionToken = $token;       // private property, fine here
        return $clone;
    }
}

This is not a cosmetic win; it is a stability win, because the public surface of the object now matches the contract we actually want, and there is no private getter a maintainer can forget to update when they rename a field. The client has a lot of value objects, Column, Table, ResultSet, QueryPlan, and asymmetric visibility keeps them honest without turning the source code into Java.

Readonly properties and the transport configuration

PHP 8.1 gave us readonly properties, PHP 8.2 added readonly classes, and by PHP 8.4 they are mature enough that we use them as the default for any object that is effectively a record. A transport configuration, for example, is a bundle of timeouts, retry counts, and header defaults that should be immutable once constructed, because a live request is already in flight and changing its timeout halfway through is a great way to get a bug nobody can reproduce. The combination of a readonly class and constructor property promotion means the entire configuration object collapses to a few lines.

readonly class TransportConfig
{
    public function __construct(
        public int $connectTimeoutMs = 5000,
        public int $requestTimeoutMs = 30000,
        public int $maxRetries = 2,
        public array $defaultHeaders = [],
    ) {}
}

Readonly classes are not a free lunch, because clone becomes awkward and you need a with method or clone plus reassignment if you want a modified copy, but for transport config that is exactly what we want: the object is a snapshot, not a living document, and the database client treats it that way.

Array find functions and query-plan navigation

PHP 8.4 adds array_find, array_find_key, array_any, and array_all, and these are the kind of functions that sound trivial until you write the code that used to do the same thing with a loop and a flag. We hit them constantly when walking query-plan metadata, because a plan node is a nested array of operators, and we need to know whether any operator touches a vector index, or whether all operators are point lookups, or which node is the first one that references a JSON column. The old way was array_filter(...)[0] ?? null for find, and a hand-rolled foreach with return false for any/all, which is fine once but becomes noise when it appears twenty times in a client.

// Before: array_filter gives you all matches, then you take the first.
$vectorNode = array_filter($planNodes, fn($n) => $n['op'] === 'ann')[0] ?? null;

// After: intent is explicit, no allocation of the full filtered array.
$vectorNode = array_find($planNodes, fn($n) => $n['op'] === 'ann');

$hasVectorScan = array_any($planNodes, fn($n) => $n['op'] === 'ann');
$allPointLookups = array_all($planNodes, fn($n) => $n['op'] === 'point');

The performance difference is small for most client workloads, but the readability difference is large, and a readable client is one that gets more contributions and fewer misuses.

Property hooks and lazy hydration

PHP 8.4 introduces property hooks, which let you attach get and set logic directly to a property declaration without writing a backing field and a separate accessor. We use them for lazy hydration on the ResultSet rows, where each row is a thin wrapper over a raw JSON array until the caller actually asks for a typed value. The hook is not magic; it is just a place to put the conversion logic so that the object remains a plain object while still doing the work on demand.

final class Row
{
    public function __construct(private array $data) {}

    public string $name {
        get => $this->data['name'] ?? ''
    }

    public int $id {
        get => (int) ($this->data['id'] ?? 0)
    }

    public object $document {
        get => json_decode($this->data['document'] ?? '{}')
    }
}

The tradeoff is that property hooks can hide work, and a getter that does JSON decoding on every access is a trap if you call it in a loop. We keep the hooks thin and cache the heavier conversions in the constructor, so the hook is mostly a type-safe doorway, not a compute sink.

ext-curl in PHP 8.4: the HTTP foundation

The default transport is cURL, because cURL is the only HTTP client that is present on enough hosts that we can assume it exists, and PHP 8.0 gave us the OO CurlHandle and typed exceptions we use in the wrapper, with PHP 8.4 refinements to default encoding behavior keeping the implementation small. The client detects cURL at runtime, falls back to streams if it is missing, and lets you inject a PSR-18 or Symfony HttpClient adapter if you want something else, but the cURL path is the one we optimize for because it is the one that exists everywhere.

final class CurlTransport implements TransportInterface
{
    private \CurlHandle $handle;

    public function __construct(private TransportConfig $config)
    {
        $this->handle = curl_init();
        curl_setopt($this->handle, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($this->handle, CURLOPT_FOLLOWLOCATION, false);
        curl_setopt($this->handle, CURLOPT_CONNECTTIMEOUT_MS, $config->connectTimeoutMs);
        curl_setopt($this->handle, CURLOPT_TIMEOUT_MS, $config->requestTimeoutMs);
    }

    public function request(string $method, string $url, string $body = ''): Response
    {
        curl_setopt($this->handle, CURLOPT_CUSTOMREQUEST, $method);
        curl_setopt($this->handle, CURLOPT_URL, $url);
        curl_setopt($this->handle, CURLOPT_POSTFIELDS, $body);

        $raw = curl_exec($this->handle);
        if ($raw === false) {
            throw new ConnectionException(curl_error($this->handle));
        }

        return Response::fromCurl($this->handle, $raw);
    }
}

The cURL handle is reused for the lifetime of the transport object, which is better than creating a new handle per request, but it is not yet true connection persistence across the whole process, which is where PHP 8.5 comes in.

PHP 8.5: the pipe operator and persistent cURL sharing

PHP 8.5 shipped in November 2025 with the pipe operator, persistent cURL share handles, and clone with modifications. These are not theoretical wins for the client; they are the features that let us simplify the code we already have.

The pipe operator lets you write $value |> $this->normalize(...) |> $this->send(...) instead of nesting calls right-to-left or assigning to a temporary variable. For a client that chains encoding, validation, and request building, the pipe operator makes the middle of the call stack readable without turning it into a fluent builder.

// PHP 8.5
$response = $requestBody
    |> json_encode(...)
    |> $this->transport->request('POST', $this->baseUrl . '/sql', ...)
    |> Response::fromRaw(...);

Persistent cURL share handles are the bigger deal. In PHP 8.4, the CurlTransport reuses a single CurlHandle for the lifetime of the transport object, but each transport object still starts a fresh TCP handshake. PHP 8.5 lets share handles persist across multiple PHP requests, which means an FPM or FrankenPHP worker can keep a warm connection to the database server instead of reopening it. We are wiring this into the transport layer so that the same CurlTransport can attach to a process-wide share handle when the runtime offers one, and the fallback for PHP 8.4 stays exactly the same.

clone with modifications is the last piece. Readonly classes are great for value objects, but changing one field used to mean either a constructor spread with get_object_vars or a hand-written with method. PHP 8.5’s clone($this, ['field' => $value]) collapses that to a single expression, so the withAlpha style of method becomes small enough that we no longer need helper traits to keep it readable.

// PHP 8.5
public function withAlpha(int $alpha): self
{
    return clone($this, ['alpha' => $alpha]);
}

The backward-compatibility tradeoff

The client currently supports PHP 8.4 as its minimum, and PHP 8.5 is now stable and widely available in the distributions we track. Runtime features like persistent cURL share handles can be guarded with function_exists and curl_share_init, so the same package can run faster on PHP 8.5 without breaking PHP 8.4, but syntax features like the pipe operator cannot be runtime-guarded. That means the pipe operator and clone with modifications will arrive as part of a major version bump that raises the minimum to PHP 8.5, while the cURL sharing lands as a backward-compatible performance improvement for anyone already on 8.5.

The modern equivalent

In 2010, a PHP database client would have been a thin wrapper around mysql_connect and the main challenge was escaping strings correctly. In 2026, the challenge is building a client that is safe, typed, and deployable without native code, and PHP 8.4 is the first version where that feels natural rather than defensive. PHP 8.5 is now the version that removes the remaining reasons a database client would need to touch anything outside Composer, not by adding one big feature, but by making the small workarounds unnecessary. That is the goal, and the language is finally moving in the same direction.

Why the split usually happens

The conventional wisdom, which is correct for most projects, is that an embedded database and a server database serve different enough audiences that splitting the crate reduces complexity for each audience. SQLite is embedded-only; it runs in your process, touches your memory space, and the only IPC is the filesystem. PostgreSQL is server-only; it runs as a daemon, manages its own process tree, and the only way to talk to it is through the wire protocol. Both are extremely good at what they do, and part of the reason is that neither one carries the baggage of the other mode.

When you try to do both in one codebase, the pressure to split shows up in the storage layer first, because an embedded engine wants direct memory access to pages and a server engine wants a page cache that is isolated from client memory, and those are different interfaces even if the underlying page format is the same. Then the pressure moves to transactions: an embedded engine can use thread-local state for the current transaction, while a server needs a session-scoped transaction context that survives across requests. By the time you reach the query executor, you are passing around enough context objects that the embedded path is paying a tax it does not need, and the server path is calling through indirections that an embedded call would skip.

The seam we picked instead

The decision we made early was that the engine crate, the thing that owns pages, WAL, transactions, indexes, and the query executor, is mode-agnostic, and the mode is applied at the boundary by the caller. When you load MongrelDB as an embedded library, you call the engine directly through a Rust function interface, and the transaction context lives in your thread. When you run mongreldb-server, the same engine crate is linked into the daemon binary, but there is a session layer between the HTTP handler and the engine call, and that session layer is the only code that knows about connections, authentication, and wire serialization.

The seam is narrow on purpose. It is roughly 800 lines of Rust that translate between “an HTTP request arrived with a SQL body and a session cookie” and “call engine::execute(sql, tx_context) with the right transaction handle.” That translation layer is the entire server-specific code path; everything below it, from the parser down to the page cache, is shared without branching on mode.

// The engine trait that both modes call. No mode flag, no feature gate.
pub trait Engine {
    fn execute(&self, sql: &str, ctx: &TxContext) -> Result<ResultSet>;
    fn begin(&self, opts: TxOptions) -> Result<TxContext>;
    fn commit(&self, ctx: TxContext) -> Result<()>;
}

// Embedded: call directly on the concrete type. Monomorphized, no vtable.
let engine: MongrelDB = MongrelDB::open(path)?;
let tx = engine.begin(TxOptions::default())?;
let result = engine.execute("SELECT * FROM users WHERE id = 1", &tx)?;

// Server: the session layer holds the engine as dyn Engine behind a
// trait object, because the session does not know the concrete type;
// it only knows it has something that implements Engine.
async fn handle_sql(req: HttpRequest, engine: &dyn Engine) -> HttpResponse {
    let session = req.session();
    let tx = session.current_tx_or_begin(engine)?;
    // execute is blocking; the server wraps it in spawn_blocking so the
    // async runtime is not stalled while the query runs.
    let result = tokio::task::spawn_blocking(move || {
        engine.execute(&req.body(), &tx)
    }).await??;
    session.maybe_commit(tx)?;
    HttpResponse::json(result)
}

The embedded caller never touches the session layer, the session layer never touches the filesystem directly, and neither side has a feature flag that conditionally compiles the other side out. Both paths exist in the same binary, and the optimizer handles dead-code elimination if you link the crate as a library and never spin up the HTTP listener.

What this buys you in practice

The first thing it buys you is testability, because every test you write against the engine trait runs identically in embedded mode and in server mode, and the only difference is whether there is an HTTP round-trip in the middle. Cross-language conformance tests for Kit, the SDK layer, all hit the server endpoint, but the expected results are generated by the embedded engine running in the test harness, which means a mismatch between the two is a real bug, not a test artifact. This is the kind of thing that sounds like a nice-to-have until you have shipped a database where the embedded path and the server path disagree about, say, how NULL sorts in a composite index, and then it becomes the thing you wish you had built from day one.

The second thing it buys you is deployment flexibility, because the engine crate that the daemon binary links can also be linked as a library into a CLI tool, a test runner, or an edge function, with no recompilation and no feature flags. The MongrelDB CLI is the server binary with a different entry point; it loads the engine, runs a query, prints the result, and exits, without ever binding a socket. The migration runner in Kit does the same thing, because it needs to apply migrations to a database file that might be local or remote, and the difference is just whether the engine handle came from MongrelDB::open(path) or from MongrelDB::connect(url).

The cost, which is real

The cost is that the server path routes every engine call through a trait object, which means a vtable lookup on every query, while the embedded path calls the concrete type directly and monomorphizes to static dispatch with no indirection. In the server path, where you are doing thousands of point lookups per second across concurrent sessions, that indirection is measurable at roughly 2 to 3 nanoseconds per call on modern hardware, which is below the noise floor for anything except a synthetic benchmark but is there, and a server that inlined its dispatch would not pay it. We judged this acceptable because the query parser, planner, and executor dominate the cost of any non-trivial query, and the vtable indirection vanishes into the overhead of actual work; the embedded path, meanwhile, pays nothing because the compiler sees the concrete type and devirtualizes the call.

The second cost is that the engine crate has to be disciplined about not reaching for session-level concepts, because once the storage layer knows about sessions, the embedded path starts carrying server baggage. This is a code review constraint, not a runtime cost, but it is the kind of architectural discipline that erodes if you are not watching for it; every PR that tries to sneak a session reference into the engine module is a PR that needs to be pushed back on, and the pushback is worth it because the seam is the entire reason the design works.

The modern equivalent

In 2008 the equivalent decision was choosing between linking against libsqlite3 and running mysqld, and the two worlds did not overlap because the engineering cost of supporting both modes in one codebase was higher than the benefit, given the hardware and the languages available at the time. Rust changes the math, because trait objects, zero-cost abstractions, and a borrow checker that enforces lifetime discipline make it possible to share code across modes without the runtime cost that a C or C++ codebase would pay, and the result is a database that can be embedded in your application today and deployed as a server tomorrow without changing a line of application code. That is the goal, and so far the seam is holding.

The shape of the problem

SQLite SEE works by presenting the database file as encrypted pages on disk and decrypting them into the page cache on read, which means the WAL, the shared memory region, and every tempfile that hits the filesystem are all covered by the same symmetric key. The encryption boundary is the entire file-level storage surface, and the key is held in memory for the lifetime of the process. That is the right model. Any encryption that does not cover the WAL is not encryption at rest, it is encryption at rest for the parts of the database that are not being actively written, and anyone who has watched a production incident knows that the interesting data is usually in the WAL during the incident.

The hard part is not the encryption itself; AES-256 in an appropriate mode is solved, and you reach for a library rather than rolling your own. The hard part is the key lifecycle, the page-level IO pattern that a database engine uses, and the throughput cost when every read and write hits the crypto layer before anything else.

What we built instead

MongrelDB uses a two-layer encryption surface: the page layer and the WAL layer share a derived key that is derived from a master key via HKDF-SHA256 with a per-table salt, and the master key itself is injected at startup from an environment variable or a plugin hook rather than stored on disk anywhere. The page encryption operates on 4KB page boundaries with AES-256-GCM, which gives us authenticated encryption so that a corrupted or tampered page on disk is detectable rather than silently misread, and page-layer key rotation is handled by re-encrypting pages in-place on a background thread during a controlled maintenance window, with the next-write rewrite path covering any pages that were not caught by the re-encryption pass. The WAL gets its own per-segment key derivation so that a WAL segment written at time T is not decryptable with a key derived after time T if the master key has been rotated in the interim.

The reason for the plugin hook rather than a fixed key storage mechanism is that different deployment shapes have different constraints around secret distribution, and the plugin hook keeps the engine honest about the abstraction boundary rather than baking in assumptions about where keys come from.

What it costs

The measured number is between 8 and 15 percent throughput reduction on write-heavy workloads at the storage layer, measured on the same hardware and same fsync profile as the unencrypted baseline. Read-heavy workloads see less, closer to 3 to 7 percent, because the page cache absorbs a portion of the read path and encrypted pages in cache cost nothing extra to serve. The GCM authentication tag adds 16 bytes per page, which increases WAL segment size by a small percentage on write-heavy workloads, and the per-segment WAL key derivation adds a small constant cost to each WAL segment open.

The number that matters more than the percentage is the shape of the cost curve. Encryption at rest is a constant factor on every IO operation, which means the overhead is proportionally largest on small transactions with many fsync calls and proportionally smallest on large sequential writes where the crypto cost is amortized over many rows per fsync. If your workload is dominated by bulk import, the measured overhead will be lower than if your workload is dominated by single-row commits with synchronous durability requirements.

The comparison that gets asked for

SQLite SEE uses a similar page-level encryption model with AES-128-CBC by default, and the performance profile is close enough that the real differentiator is not the crypto algorithm but the authentication property and the key management surface. CBC mode is unauthenticated, which means a bit-flip attack on an encrypted page produces modified plaintext with no checksum to catch it; GCM is authenticated, which means MongrelDB encrypted pages detect tampering rather than returning silently corrupted data, and that property is the engineering argument worth making explicitly rather than leaving it as an implicit difference. We did not build our own crypto to beat SEE on throughput, we built it because the closed-source distribution model does not fit a database that ships as a single open-source binary with no vendor relationship, and if your threat model includes the database vendor as a potential adversary, you should be using hardware-enforced encryption anyway, but if your threat model is “the disk gets pulled or the machine gets stolen,” a software encryption layer at the page level covers the practical attack surface for most threat models short of nation-state adversaries with physical disk access and a budget for hardware forensics.

The tradeoff is what it always is: you are trading a known-cost constant factor on every IO operation for protection against a class of incidents that are rare, expensive when they happen, and impossible to retrofit after the fact. Whether that tradeoff is worth it for your workload is not a question we can answer for you, but the numbers above are the inputs to that calculation.

What a learned-range index is, briefly

A learned-range index is a small neural network, typically an MLP of one or two hidden layers, that learns the cumulative distribution function (CDF) of sorted keys and predicts the leaf-page offset of any given key from that CDF. The reference is Kraska et al. (2017), “The Case for Learned Index Structures,” and the headline result, measured on specific in-memory integer-key benchmarks with the paper’s own caveats, was up to an order of magnitude faster than a B-tree on point lookups and up to 70% smaller in memory footprint; later work (Ferragina et al., 2020) showed those gains were workload-sensitive and that the win was real but smaller for general-purpose data, so we treat the headline numbers as upper bounds rather than the floor. The model replaced the tree with two or three multiply-adds per lookup; the B-tree existed only as a fallback when the model’s error term exceeded the leaf size.

In MongrelDB the learned-range index is implemented as a recency-weighted CDF: incremental updates nudge the model on hot inserts, full retraining happens at segment and compaction boundaries rather than per row, and the loss is weighted toward recent keys so the high-traffic part of the keyspace stays accurately modelled even as the tail grows. The shape is the same MLP, the loss and the update cadence are what changed.

What it bought us on the path we cared about

The path that mattered was sorted range scans on data larger than memory, and the query shape that kept appearing in our traces was WHERE event_at BETWEEN ? AND ? against an events table that had outgrown the host’s RAM. A B-tree over a 200 GB cold column on a 64 GB host pays roughly log(N) / log(fanout) random reads per query, where N is the leaf-page count, and that random-read pattern is what dominates p99 on the workloads we kept getting bug reports for, where the column was the company’s events table and the query was “everything between Tuesday and Wednesday last week.” A learned-range index over the same data collapses most of the random reads into sequential reads, because the model’s prediction lands the seek in the right column block on the first try more often than not, and the cold read degrades to “load the leaf and scan” rather than “walk the tree and load the leaf.”

We measured this on a customer trace with the kind of WHERE event_at BETWEEN ? AND ? query above against an 180 GB events table on a 64 GB host at roughly a 3-4x speedup on p99 for the cached-leaf case and a 1.5-2x speedup on p99 for the cold-leaf case; those numbers are honest, they hold over a multi-day window, and a benchmark fixture in the kit/tests directory reproduces them so anyone can re-run them on their own hardware without trusting ours.

The second thing the learned-range index bought us was a smaller working set, and that part mattered more than the latency number in the traces we cared about: a learned index holds each key distribution in a few kilobytes of model weights, where a B-tree holds the same key distribution in a few megabytes of pointers and page metadata, and the difference in cache pressure showed up as a higher hit rate on the rest of the workload, because there was more cache left over for the leaf pages the model was pointing at.

What we had to give up to get there

Unconditional correctness at the index boundary. A B-tree places every key at a known offset, so WHERE k BETWEEN 100 AND 200 returns exactly the keys whose value is in [100, 200], every time, with no error term to reason about. A learned-range index has a model error that defines a “delta band” around each prediction, and a query that lands in that band has to fall back to a leaf scan; on data with sharp CDF cliffs (compressed keys, monotonic timestamps) the delta band is narrow and the scan is short, but on data with heavy tail skew (event sources, sparse UUIDs) the delta band is wider and the scan turns into the worst case the model was supposed to prevent. The default IndexBuildPolicy::Learned configures the model for the common recency-heavy case; on data that is not recency-heavy, you fall back to IndexBuildPolicy::BTree and the B-tree beats the learned index. We chose that on purpose rather than papering over it.

Engineering cost. Training a model per index is not a one-line addition to a B-tree codepath; it is a separate subsystem with its own CI fixtures, its own drift detection (model error as measured against held-out data is a real signal we monitor in the catalog), its own rebuild policy on schema change, and its own failure mode when the loss function does not converge on a degenerate distribution (uniform UUIDs, for example). That cost is real, and it is the kind of cost that makes a database team ask whether the speedup justifies the cognitive load; we think it does, but we are not going to pretend that a B-tree was simpler.

Where it shows up in the wire format

WHERE k BETWEEN ? AND ? against a typed numeric, decimal, timestamp, or date column dispatches through index_plan = LearnedRange(model_id = X, delta = 8KB) to /kit/query, and the fallback path is index_plan = BTree. The choice is per-index at build time, not per-query at runtime, which means a single table can hold a B-tree on its primary key and a learned-range index on its events column, and you pick the right tool per column rather than per query. The wire format is the same either way:

CREATE TABLE events (
  id            INT64,
  event_at      TIMESTAMP,
  payload       JSON,
  INDEX event_at_idx USING LEARNED ON event_at,
  INDEX id_pk     USING BTREE   ON id
);

SELECT count(*), avg(latency_ms)
FROM events
WHERE event_at BETWEEN '2026-07-01 00:00:00' AND '2026-07-08 00:00:00';

Same WHERE clause you would write for a B-tree, same planner, same transactional commit at the end of the statement, same WAL coverage for both index and row update on any INSERT INTO events. The learned-range index is the right shape for event_at, the B-tree is the right shape for id, and we ship both because they are the right tool for each column rather than because we had an academic result to justify.

When this is the wrong tradeoff

If your column is small enough to fit in cache, the B-tree wins anyway, because the random-read cost of the B-tree is small relative to the cache-hit latency of the learned-range index. If your column is wider than memory and your queries are equality lookups (WHERE k = ?) rather than range scans, the B-tree wins by a wider margin, because the learned-range index’s prediction error buys you nothing when you are not sweeping across a key range. If your data has no recency structure at all (a uniformly-distributed synthetic key, an auto-increment surrogate id), the learned-range model’s loss never improves past the prior, and the B-tree is the right answer with no hesitation. The learned-range index is a tool for one specific shape of query on one specific shape of data; the B-tree is the universal tool that covers the rest, and we ship both as first-class index options in the same engine, on the same wire format, with the same transaction model.

What pure PHP actually means in 2026

The phrase “pure PHP” has accumulated some baggage over the years, partly because PHP’s early reputation as a slow interpreted language made “pure PHP” sound like “slow PHP,” and partly because the PHP ecosystem spent a decade leaning on C extensions as proof of seriousness. Neither of those associations is accurate anymore, and the PHP client does not trade on either of them.

What pure PHP means here is concrete: the entire client is written in PHP 8.4 syntax, it ships as a Composer package, it has no native code dependencies, and it does not call any function that requires a C extension. HTTP is handled by curl via ext-curl when PHP 8.5 persistent handle sharing is available (PHP 8.5 being the newer of the two runtimes, so the stream-context path remains the common case for now), and by regular stream contexts when it is not; both are in the standard PHP distribution and neither requires a compile step. JSON is handled by json_decode and json_encode, which have been built into PHP since 5.2 and have been reliably fast since 7.0. There is no binary .so to load, no phpize to run, no ABI version to match against the server’s architecture, and no risk that pecl install mongrel fails because your host has disabled exec() in php.ini.

The install experience

composer require visorcraft/mongreldb-php

That is the entire install. The package lands in vendor/, autoloading is configured, and you can instantiate the client in one line:

use Visorcraft\MongrelDB\Client;

$db = new Client('https://db.example.com', token: getenv('MONGRELDB_TOKEN'));

No bootstrap scripts, no extension loading in php.ini, no running mongreldb-php-setup as a post-install script. The Composer package ships the client, the type stubs, and the error hierarchy; you require the package and you are talking to the database.

Why HTTP is the right transport for a pure PHP client

The PHP client could have used a Unix socket or a raw TCP protocol with a binary framing layer, and in a pure-server context those choices would make sense for latency. But PHP is not a long-running process by default; PHP-FPM spawns a process per request or per few requests, and the typical PHP application opens a fresh database connection on every web request, closes it at the end, and does not reuse it unless you have gone out of your way to implement connection pooling. An HTTP client reuses keep-alive connections across requests in the same process lifetime, and when PHP-FPM recycles a worker the next request starts fresh, which is the same behavior the database would have expected anyway.

HTTP also means the PHP client works through every HTTP-aware proxy, load balancer, and API gateway that already exists in your infrastructure, and it means TLS termination happens at the normal layer rather than requiring a special database-facing certificate setup. You are already doing HTTP for every other external service your application calls; the database does not need to be special.

What you give up and what you do not

You give up the single-digit-microsecond in-process path, because HTTP adds a header processing layer that a raw TCP protocol would not have (a few hundred microseconds of framing overhead per operation, not a category shift), and you give up the ability to operate without a network path to the server, which is the embedded-database use case that MongrelDB handles differently through its Kit SDK. For a PHP web application that talks to a mongreldb-server over a LAN or a VPC, those tradeoffs are not on the critical path; the query execution time inside the database dominates the HTTP overhead by two orders of magnitude.

You do not give up type safety, because the client speaks a typed wire format and deserializes responses into PHP native types with a full exception hierarchy that maps HTTP status codes to domain exceptions, so a 404 from the server becomes a NotFoundException in your application code rather than a generic runtime error. You do not give up connection pooling in the PHP-FPM context, because the client uses persistent curl handles when PHP 8.5 is available, which means the underlying TCP connection is reused across requests in the same worker process without the TLS handshake cost on every page load. And you do not give up SQL surface area, because the wire format carries the same query plans and constraint semantics as the Kit SDK and the Rust core, so the PHP client is not a reduced-surface wrapper; it is the full client in a different transport.

Who benefits from this in practice

The obvious answer is the shared-hosting developer who is on a cPanel machine with no root access and no ability to run phpize, and that answer is correct but incomplete. The more interesting case is the CI pipeline: a composer require in a Dockerfile that uses php:cli as its base image does not need any system package installed beyond the PHP runtime and Composer itself, which means your test matrix can cover PHP 8.4 and PHP 8.5 on the same Docker image without an extension compilation step slowing down every build. The pure PHP client makes your database client a first-class Composer dependency rather than a system-level install, and that distinction matters when your CI minutes are metered and your container build time is on the critical path.

What an FM-index is, briefly

A Burrows-Wheeler transform turns text into something that compresses as well as the same text runs through gzip, because the BWT groups repeated substrings the same way. The FM-index adds enough suffix-sampling metadata on top of the BWT that you can test whether a pattern is a substring by walking the BWT’s first-column matrix one character at a time, without scanning the original text at all. The asymptotic shape is O(|pattern|) per query against a corpus that is highly compressed on disk.

Inverted indexes, by contrast, store the original tokens uncompressed, hash each token to a postings list, and answer a query by intersecting those postings lists in constant time per term. They are the right choice when your data is largely read-only, your queries are repeated, and you can afford to rebuild the index out of band after writes. They are not the right choice for a transactional engine that wants the index to participate in the same commit as the rows.

What the FM-index bought us at the storage layer

Two things specifically.

The first is write-path simplicity. An inverted index wants to be immutable between commits, which is why transactional inverted indexes need separate WAL coordination, deferred rebuilds, or row-level deltas stitched onto an immutable postings structure. An FM-index in MongrelDB is materialized as a column-level rewrite, batched with the rows it covers, fsync’d through the same WAL, and rolled back by not rewriting it; the index has no separate consistency model because there is no separate index state to coordinate.

The second is the readability of the data on disk. The same BWT that makes the FM-index queryable is the same representation that compresses text well, and MongrelDB’s storage layer already pays for compressed column layouts in the rest of the engine; the search side gets that compression benefit alongside the rest of the engine’s columnar storage without a separate index cost.

That is the trade we wanted: one source of truth, one consistency model, one wire format. The FM-index was the data structure that gave us all three; the inverted index would have given us one and a half and required a separate subsystem for the half.

What we had to give up to get there

Build cost. Inverted indexes get built incrementally as documents are inserted, and the index is correct after every INSERT. The FM-index is meaningful only over a corpus; rebuilding it across a single row is wasteful. MongrelDB batches FM-index rewrites across table-rewrite checkpoints, with the same IndexBuildPolicy::Deferred default the SQL indexes use; this means write amplification on full-text-heavy workloads is higher than on inverted-index engines, and first-query-after-batch latency can spike when a checkpoint has not been built yet. The benchmark numbers are honest about this; the architecture is honest about why.

Pure-pattern ranking. The FM-index gives you containment (does the pattern appear?), not relevance (how much does it matter?). BM25 ranking is available via the mongreldb_fts_rank(text, query) UDF, which computes simplified BM25 from per-row term frequency and approximated corpus statistics at query time; accurate BM25-with-IDF lives in the separate fts_docs virtual table, which is the inverted-index path. That is a hybrid shape, not pure FM-index purity, and we note it because a search-engine purist would.

Where it shows up in the wire format

WHERE body LIKE '%phrase%' in raw SQL, ->whereFts(...) in the Kit query builder, and where('fm', ...) in the PHP fluent builder all dispatch to the same FM-index path through /kit/query. There is no separate search endpoint to point your app at, no separate consistency model to reason about, and no eventual-index lag to defend against in application code. A search query joined against the same WHERE clauses you would write for a regular typed column looks like:

SELECT id, title
FROM articles
WHERE body LIKE '%rust bwt fts%'
  AND published_at > '2026-01-01'
ORDER BY mongreldb_fts_rank(body, 'rust bwt fts') DESC
LIMIT 20;

Same transactional commit at the end of the statement, same WAL coverage for the index update on any INSERT to articles, and the same single-engine consistency model for both the row and the search result.

When this is the wrong tradeoff

If you are running a pure search-engine workload with no transactional semantics on the indexed data, with hot-write patterns the FM-index chokes on, and with strict sub-millisecond p99 on very short queries, an inverted-index engine tuned for that workload is still faster than what an FM-index on a transactional engine gives you; we did not pretend otherwise when we built it. What we wanted was transactional full-text search on the same engine as the rest of your data, on the same wire format, and the FM-index was the right way to spend the storage budget to get it.

What “in-process” actually buys you

Two things specifically, and they are the same two things every in-process database call gets when the network is gone.

The first is a round trip. A single HTTP request from any Tier 2 client to a mongreldb-server over loopback measures p50 ~0.22 ms / p99 ~0.62 ms at ~4,000 ops/s for PUT /tables/{name}/put and POST /kit/txn commit on the published BENCHMARKS.md; the equivalent call into the engine through the in-process Tier 1 binding (Rust, PyO3 for Python, NAPI for TypeScript / Node) drops the per-call cost into single-digit microseconds, with the same 6.79 µs commit (group-committed, fsync’d) bound from the write-path numbers in the same file. The network is the dominant cost in the Tier 2 path; axum handler dispatch, JSON encode/decode, and the TCP round trip add up, while the engine work underneath is sub-10 µs.

The second is transactional consistency. The HNSW graph and the row it covers get the same atomic-write semantics; either both are durable, or both are not. There is no “the database committed but the vector index lagged” state for the engine to defend against because the engine is one engine. CDC pipelines, outbox consumers, and dual-write coordinators exist in application code only because two systems need to look like one; they stop existing when the two systems collapse back into one.

The actual numbers

The HNSW graph is a column-level index in the same engine files as the rows it indexes. A vector insert inside a transaction rides the same single-digit-microsecond put path the published cargo bench -p mongreldb-core --bench write_path numbers report (put no-fsync at 618 ns, commit fsync at 6.79 µs, group commit at 686 µs for 1,000 rows). A k-nearest-neighbour query in the same transaction rides the same sub-10 µs filter path the cold-SQL-filter number (8.7 µs at N=1M) tracks. The HTTP loopback path costs the same 0.22 ms p50 / 0.62 ms p99 as every other Tier 2 op, because the dominant cost in that path is the network, not the index; HNSW-specific benchmark numbers will land in BENCHMARKS.md when the HNSW build policy is committed, and we will not invent them in the meantime.

What this looks like from a where('ann', ...) call

The fluent query builder in mongreldb-php pushes where('ann', ...) to the same HNSW index through the same /kit/query path as a regular typed column. A query for ten nearest neighbours of a 384-dimensional embedding against a documents table, filtered by tags = 'engineering', looks like:

$rows = $db->table('documents')
    ->where('tags', 'engineering')
    ->whereAnn('embedding', $vector, k: 10)
    ->get();

That call, against a mongreldb-server running on the same host, is one HTTP round trip; against an in-process Tier 1 binding it is one function call into mongreldb-core. Both paths return the same rows with the same transaction isolation; the only difference is the round trip.

When this is the wrong shape

If your workload has a single vector index that already exceeds what fits in the memory of one server, and you genuinely need sharded vector search across dozens of machines, an HNSW graph that lives inside one engine process is the wrong shape; you want a vector service that is built to shard, and we will not pretend otherwise. One engine process means one HNSW graph in memory, and that is the limit on a single-machine deployment. What you get by staying inside one process is that the row that gets updated with a new title is the same transaction that updates the row’s vector, the same WHERE clause that filters by tags = 'engineering' also filters by approximate-nearest-neighbour distance, and the same /healthz endpoint that tells you the database is up also tells you the vector index is up; for most applications that is a much better set of defaults than another microservice to operate, and when a workload outgrows it the same client code paths through the same wire format keep working against a vector service with no application-side rewrite.

It reflects an architectural decision we made before writing the eighth client, and it has visible tradeoffs.

The two-tier model

MongrelDB clients come in two flavors:

1. Tier 1 (in-process native): the engine runs in your process. No daemon, no serialization, no network hop. You get the same sub-10µs single-row write path as the embedded Rust API itself.
2. Tier 2 (HTTP): a pure-language client connects to a running mongreldb-server over HTTP. There is one round-trip per query, in the low-millisecond range.

We have three Tier 1 entry points today: Rust (direct, native), Python (PyO3, via MongrelDB Kit), and TypeScript / Node (NAPI addon, via either Kit or crates/mongreldb-node). The other nine are Tier 2: PHP, Go, Java/Kotlin, C#/.NET, Ruby, Swift, Zig, D, and Nim.

This is not because we could not write native bindings for the other nine. It is because we chose not to.

Why two tiers exist

Twenty-five years of SQLite have produced one clear lesson: the embedded path beats the network path every time, when you have a choice. Native bindings for an embedded database are the better-sqlite3 model, and it is hard to argue with. We picked that model for Rust, TypeScript, and Python.

The other nine do not fit cleanly. The C toolchain story in each of them has, at best, one platform with first-class support and a long tail of platforms with rough edges:

• Ruby has gem build and Fiddle. Native extensions work on MRI/YARV, less cleanly on JRuby and TruffleRuby.
• PHP has PECL. PECL extensions work everywhere PHP installs, but every install means a separate Makefile, an phpize, and a php-config check. The pure-PHP client we shipped this month exists precisely because that build matrix is painful.
• Swift has SwiftPM on macOS. On Linux, the Swift toolchain is real but second-tier. The cost is per-platform build scripts.
• Java has JNI. JNI still works in 2026, but it is a generation-old API, and every JVM in the install path is another moving part.
• C# has P/Invoke. That part is fine. The complications come from cross-platform packaging, RID resolution, and what to do about self-contained versus framework-dependent deployments.
• Go, Zig, D, Nim all have C FFI stories that work, but each brings an extra build-time dependency on a Rust compiler or C toolchain that the language’s users did not sign up for.

Native bindings for those nine would mean nine more CI matrices, nine more sets of release artifacts, and a permanent maintenance cost on every engine change. We chose to ship pure-language HTTP clients instead, with a real latency tradeoff we will not pretend away.

What the HTTP tier looks like

A Tier 2 client speaks a JSON-over-HTTP protocol to mongreldb-server. The protocol mirrors the native API surface: writes land at PUT /tables/{name}/put (or batch through POST /kit/txn), hybrid queries at POST /kit/query. Encoding is plain UTF-8 JSON for most requests; binary values go over base64. It is not exotic.

Latency for a single-row write over HTTP on loopback is dominated by axum’s handler dispatch, JSON parse/serialize, and the TCP round trip, not the engine.

To put a number on it: on a release build over loopback HTTP, single-row PUT /tables/bench/put measured p50 ~0.22 ms, p99 ~0.62 ms, with ~4,000 ops/s sustained. Single-row commit was nearly identical, p50 ~0.21 ms, p99 ~0.53 ms. fsync is in the commit path but does not dominate on this hardware.

That is fine for most PHP, Ruby, Java, and Go workloads, where the unit of work is rarely one row. It is not fine for Python and Node code that wants the sub-10µs native path, which is exactly why Tier 1 exists for those two.

The C ABI and why it exists separately

The newest crate in the repository is mongreldb-ffi. It exposes a stable C ABI over the engine: a fixed set of extern "C" functions for opening a database, beginning a transaction, putting a row, committing, and a handful of query helpers. Public structs and enums are #[repr(C)]. Opaque handles (mongreldb_database_t, mongreldb_transaction_t) wrap Arc<Database> and friends so language runtimes can hold them by pointer.

It is not yet what the Tier 1 clients call into. Rust and PyO3 call into mongreldb-core directly. The NAPI addon also calls into mongreldb-core directly. The C ABI exists so that future Tier 1 clients can land on top of it.

Three reasons we did this:

1. Stability. A C ABI is a contract you can hold to across major versions of the engine. Rust API surfaces drift; the C ABI does not.
2. Reuse across FFI stacks. Swift (@_cdecl), Ruby (Fiddle on MRI), Go (cgo), Java (JNI), Zig (@cImport), Nim (direct C import), and every other language with a working C FFI all consume C headers. One crate, many targets.
3. Decoupling. The C ABI can change without the engine changing, and vice versa. That means we can ship and version the FFI surface independently of the engine, which we will need to do.

The current mongreldb-ffi is small on purpose. It exposes the minimum surface needed for an “open, write a few thousand rows, commit, query, close” workflow, plus error and panic-safety plumbing (mongreldb_last_error, mongreldb_last_error_code, mongreldb_free_error_string, free functions for every allocated handle). We will extend it as we add Tier 1 bindings for more languages.

Picking the right tier

If you are writing Rust, Python, or TypeScript / Node against MongrelDB, use the Tier 1 binding. You get in-process speed, no daemon to run, and the same mongreldb-core API surface used by our own server. The NAPI and PyO3 clients are not wrappers; they are first-class Rust compiled to a shared library and bound through standard FFI machinery. For Node specifically, the raw mongreldb-node NAPI addon (the better-sqlite3 model, cd crates/mongreldb-node && npm install && npm run build) is the lowest-overhead path. Kit wraps the same engine behind a typed object API if you would rather skip the build step.

If you are writing PHP, Go, Java/Kotlin, C#/.NET, Ruby, Swift, Zig, D, or Nim, use the Tier 2 HTTP client. Run mongreldb-server alongside your app (systemd, container, sidecar), point the client at it, and you get the same engine underneath with a network in the middle.

If your language is not on the list, the C ABI is the on-ramp. Either we already have a Tier 1 prototype and we can wire you up, or you can ship your own binding on top of mongreldb-ffi and we will help you land it on the org.

What we are shipping next

We will keep extending Tier 2 to languages where maintaining a pure-language HTTP client is the right call, including maintenance and small feature work on the eight new clients.

We have no specific Tier 1 language conversion committed. mongreldb-ffi is the on-ramp; what we land on top of it will be picked per language when the engineering case is clear.

Where to find each client

The wire protocol for Tier 2 is documented at docs/08-daemon. The C ABI surface lives in crates/mongreldb-ffi. For each client’s install command and examples, see its repository.

If you maintain a language we do not yet support, file an issue with the language name. The C ABI makes the answer shorter than it used to be. For the release announcement and install commands for the eight new HTTP clients, see the companion post.