Why most embedded engines stop at single-process

SQLite has WAL mode and it is genuinely good at what it does, but if you want change data capture you reach for the SQLite Update Hook or you parse the WAL file yourself, and neither of those gives you a durable, resumable stream with stable event identifiers that survive a restart; LevelDB and RocksDB expose iterators and you can tail the log, but the log is an internal artifact subject to compaction and the ordering guarantees are the iterator’s problem, not yours, and the moment you want transactional semantics, where a batch of writes commits atomically and your downstream consumer sees the whole batch or nothing, you are back to application-level bookkeeping. The pattern across the embedded-database landscape is the same: the engine optimizes for local read and write throughput, the replication story is delegated to external tooling, and the application developer who needed a simple embedded database ends up running a three-component pipeline to get changes out of it.

CDC derived from the WAL, not a second log

MongrelDB’s change data capture is not a separate log that the engine maintains alongside its data; it is a reconstruction of committed transactions from the same WAL that the engine uses for crash recovery, which means there is exactly one source of truth for what committed and in what order, and the CDC stream cannot diverge from the data because it is the data, just read differently. The API surface is a single function, change_events_since(last_event_id), which replays the WAL records, correlates each Op::TxnCommit with its staged operations, groups them by transaction, and returns a CdcBatch containing one ChangeEvent per committed transaction with a stable ID shaped as <commit_epoch>:<operation_index>, so a consumer can persist the last ID it processed and resume from exactly there on the next poll without missing or duplicating a transaction.

The design choice that matters here is that aborted transactions never appear in the CDC stream, because the reconstruction logic keys on Op::TxnCommit records, and a transaction that rolled back never wrote one, which means your downstream consumer never sees a write that did not durably commit; this is the guarantee that external WAL tailers have to reconstruct for themselves, and they get it wrong in the gap between the WAL entry and the commit marker, where a half-applied transaction looks like data until you cross-reference the commit log and discover it was never finalized. The engine test suite exercises this by staging a put, rolling it back, and confirming the CDC stream contains no put events for that transaction:

let mut transaction = db.begin();
transaction.put("items", vec![(1, Value::Int64(1))]).unwrap();
transaction.rollback();

assert!(db
    .change_events_since(None)
    .unwrap()
    .events
    .iter()
    .all(|event| event.op != "put"));

A rolled-back put produces no CDC event, full stop, and that is not a filter applied after the fact but a consequence of the reconstruction logic skipping any WAL records that lack a matching commit.

The gap signal, and why it matters

A derived CDC stream has one honest limitation that a separate log does not: the WAL segments are finite, and once the engine checkpoints and purges old segments to reclaim space, the records needed to reconstruct older events are gone. MongrelDB handles this by returning gap = true in the CdcBatch when the consumer’s resume point falls before the oldest retained commit, which is the engine saying “I cannot give you a continuous stream from where you left off, so do not pretend you have one”; the correct consumer response is to rebootstrap, not to silently skip ahead, because a gap in a change stream is a data integrity problem, not a performance optimization. The retention is tunable via set_replication_wal_retention_segments, and the test suite verifies the gap behavior by writing data, checkpointing to purge segments, then resuming from a now-evicted position:

db.set_replication_wal_retention_segments(1);
// ... write and checkpoint multiple transactions ...
let resumed = db.change_events_since(Some(&first_id)).unwrap();
assert!(resumed.gap);

The bounds are generous but real: at most one million WAL records replayed, 256 MiB of replay bytes, 100,000 events, one million rows, and 256 MiB of total retained bytes, after which the engine fails closed rather than silently truncating, because a change stream that drops events without telling you is worse than no stream at all.

Replication as WAL streaming

The replication surface is the same idea at a different granularity, because instead of reconstructing logical change events, the follower receives raw committed WAL records over HTTP and replays them into its own local database directory, which means the follower’s logical state at any given commit epoch matches the leader’s, not a best-effort approximation; the physical bytes on disk may diverge once the follower independently checkpoints and compacts on its own schedule, but the data you can read at a shared epoch is identical. The daemon exposes GET /wal/stream?since=<seq>, the follower polls it, and the ReplicationFollower client in mongreldb-client wraps the whole loop:

use mongreldb_client::ReplicationFollower;

let mut follower = ReplicationFollower::new("http://leader:8453", "/local/copy")?;
follower.sync()?;  // fetch + apply all new records since last sync
println!("caught up to epoch {}", follower.last_epoch());

The follower bootstraps from a snapshot if the local copy does not exist yet, refuses to overwrite a non-replica database directory, and advances its watermark epoch only after the records are durably appended and the local database has recovered, so a crash mid-sync does not corrupt the follower or leave it in a half-applied state; it simply resumes from the last confirmed epoch on the next sync() call. This is asynchronous leader-to-follower replication designed for read scaling and for warm standby copies, distinct from the synchronous Raft consensus replication that MongrelDB also supports in cluster mode, and the two coexist because they solve different problems: Raft gives you strong consistency across a quorum for write availability and zero-data-loss failover, while WAL streaming gives you a geographically distant read copy or an analytics replica without paying the latency of a cross-region quorum ack. The honest tradeoff is that async streaming has an RPO greater than zero, because writes the leader committed but had not yet streamed are lost if the leader dies before the next poll, and if you need RPO 0 that is what the Raft quorum path is for; the async follower is the right tool for read scaling and for a recovery target that tolerates a small amount of tail loss, not a substitute for consensus-backed durability.

What changes and what stays the same

The thing that changes when an embedded engine is also a replicating engine is that the WAL stops being a purely internal recovery artifact and becomes a public contract, because external consumers depend on its ordering, its commit semantics, and its retention window, and those properties now have to hold across versions and across restarts. The thing that stays the same is the commit path itself, because the engine does not do extra work on the write side to produce the CDC stream or feed the replication follower; it writes the same WAL records it would write anyway, and the consumers read them after the fact, which means there is no throughput tax on writes for the benefit of change data capture until somebody actually calls change_events_since or polls /wal/stream. The cost moves to the read side, where WAL reconstruction and network transfer do real work, and to the storage side, where WAL retention has to be wide enough to cover your consumers’ polling interval, and those are knobs you turn based on your actual workload rather than overhead you pay whether you use the feature or not.

The modern equivalent of an old pattern

In the MySQL world, if you wanted change data capture in 2010, you parsed the binlog, and if you wanted replication, you configured a replica and hoped the binlog position tracking held up across schema changes and version upgrades, and the binlog was a separate log from the InnoDB redo log, which meant there were two sources of truth for what committed and they could disagree in edge cases around crash recovery. MongrelDB collapses that to one log: the WAL is the redo log and the replication log and the CDC source, and there is no second log to keep in sync, no binlog position to lose, and no external connector to reverse-engineer your commit semantics, because the engine that writes the data is the same engine that tells you what changed.

Why the split usually happens

The conventional wisdom, which is correct for most projects, is that an embedded database and a server database serve different enough audiences that splitting the crate reduces complexity for each audience. SQLite is embedded-only; it runs in your process, touches your memory space, and the only IPC is the filesystem. PostgreSQL is server-only; it runs as a daemon, manages its own process tree, and the only way to talk to it is through the wire protocol. Both are extremely good at what they do, and part of the reason is that neither one carries the baggage of the other mode.

When you try to do both in one codebase, the pressure to split shows up in the storage layer first, because an embedded engine wants direct memory access to pages and a server engine wants a page cache that is isolated from client memory, and those are different interfaces even if the underlying page format is the same. Then the pressure moves to transactions: an embedded engine can use thread-local state for the current transaction, while a server needs a session-scoped transaction context that survives across requests. By the time you reach the query executor, you are passing around enough context objects that the embedded path is paying a tax it does not need, and the server path is calling through indirections that an embedded call would skip.

The seam we picked instead

The decision we made early was that the engine crate, the thing that owns pages, WAL, transactions, indexes, and the query executor, is mode-agnostic, and the mode is applied at the boundary by the caller. When you load MongrelDB as an embedded library, you call the engine directly through a Rust function interface, and the transaction context lives in your thread. When you run mongreldb-server, the same engine crate is linked into the daemon binary, but there is a session layer between the HTTP handler and the engine call, and that session layer is the only code that knows about connections, authentication, and wire serialization.

The seam is narrow on purpose. It is roughly 800 lines of Rust that translate between “an HTTP request arrived with a SQL body and a session cookie” and “call engine::execute(sql, tx_context) with the right transaction handle.” That translation layer is the entire server-specific code path; everything below it, from the parser down to the page cache, is shared without branching on mode.

// The engine trait that both modes call. No mode flag, no feature gate.
pub trait Engine {
    fn execute(&self, sql: &str, ctx: &TxContext) -> Result<ResultSet>;
    fn begin(&self, opts: TxOptions) -> Result<TxContext>;
    fn commit(&self, ctx: TxContext) -> Result<()>;
}

// Embedded: call directly on the concrete type. Monomorphized, no vtable.
let engine: MongrelDB = MongrelDB::open(path)?;
let tx = engine.begin(TxOptions::default())?;
let result = engine.execute("SELECT * FROM users WHERE id = 1", &tx)?;

// Server: the session layer holds the engine as dyn Engine behind a
// trait object, because the session does not know the concrete type;
// it only knows it has something that implements Engine.
async fn handle_sql(req: HttpRequest, engine: &dyn Engine) -> HttpResponse {
    let session = req.session();
    let tx = session.current_tx_or_begin(engine)?;
    // execute is blocking; the server wraps it in spawn_blocking so the
    // async runtime is not stalled while the query runs.
    let result = tokio::task::spawn_blocking(move || {
        engine.execute(&req.body(), &tx)
    }).await??;
    session.maybe_commit(tx)?;
    HttpResponse::json(result)
}

The embedded caller never touches the session layer, the session layer never touches the filesystem directly, and neither side has a feature flag that conditionally compiles the other side out. Both paths exist in the same binary, and the optimizer handles dead-code elimination if you link the crate as a library and never spin up the HTTP listener.

What this buys you in practice

The first thing it buys you is testability, because every test you write against the engine trait runs identically in embedded mode and in server mode, and the only difference is whether there is an HTTP round-trip in the middle. Cross-language conformance tests for Kit, the SDK layer, all hit the server endpoint, but the expected results are generated by the embedded engine running in the test harness, which means a mismatch between the two is a real bug, not a test artifact. This is the kind of thing that sounds like a nice-to-have until you have shipped a database where the embedded path and the server path disagree about, say, how NULL sorts in a composite index, and then it becomes the thing you wish you had built from day one.

The second thing it buys you is deployment flexibility, because the engine crate that the daemon binary links can also be linked as a library into a CLI tool, a test runner, or an edge function, with no recompilation and no feature flags. The MongrelDB CLI is the server binary with a different entry point; it loads the engine, runs a query, prints the result, and exits, without ever binding a socket. The migration runner in Kit does the same thing, because it needs to apply migrations to a database file that might be local or remote, and the difference is just whether the engine handle came from MongrelDB::open(path) or from MongrelDB::connect(url).

The cost, which is real

The cost is that the server path routes every engine call through a trait object, which means a vtable lookup on every query, while the embedded path calls the concrete type directly and monomorphizes to static dispatch with no indirection. In the server path, where you are doing thousands of point lookups per second across concurrent sessions, that indirection is measurable at roughly 2 to 3 nanoseconds per call on modern hardware, which is below the noise floor for anything except a synthetic benchmark but is there, and a server that inlined its dispatch would not pay it. We judged this acceptable because the query parser, planner, and executor dominate the cost of any non-trivial query, and the vtable indirection vanishes into the overhead of actual work; the embedded path, meanwhile, pays nothing because the compiler sees the concrete type and devirtualizes the call.

The second cost is that the engine crate has to be disciplined about not reaching for session-level concepts, because once the storage layer knows about sessions, the embedded path starts carrying server baggage. This is a code review constraint, not a runtime cost, but it is the kind of architectural discipline that erodes if you are not watching for it; every PR that tries to sneak a session reference into the engine module is a PR that needs to be pushed back on, and the pushback is worth it because the seam is the entire reason the design works.

The modern equivalent

In 2008 the equivalent decision was choosing between linking against libsqlite3 and running mysqld, and the two worlds did not overlap because the engineering cost of supporting both modes in one codebase was higher than the benefit, given the hardware and the languages available at the time. Rust changes the math, because trait objects, zero-cost abstractions, and a borrow checker that enforces lifetime discipline make it possible to share code across modes without the runtime cost that a C or C++ codebase would pay, and the result is a database that can be embedded in your application today and deployed as a server tomorrow without changing a line of application code. That is the goal, and so far the seam is holding.