Why most embedded engines stop at single-process

SQLite has WAL mode and it is genuinely good at what it does, but if you want change data capture you reach for the SQLite Update Hook or you parse the WAL file yourself, and neither of those gives you a durable, resumable stream with stable event identifiers that survive a restart; LevelDB and RocksDB expose iterators and you can tail the log, but the log is an internal artifact subject to compaction and the ordering guarantees are the iterator’s problem, not yours, and the moment you want transactional semantics, where a batch of writes commits atomically and your downstream consumer sees the whole batch or nothing, you are back to application-level bookkeeping. The pattern across the embedded-database landscape is the same: the engine optimizes for local read and write throughput, the replication story is delegated to external tooling, and the application developer who needed a simple embedded database ends up running a three-component pipeline to get changes out of it.

CDC derived from the WAL, not a second log

MongrelDB’s change data capture is not a separate log that the engine maintains alongside its data; it is a reconstruction of committed transactions from the same WAL that the engine uses for crash recovery, which means there is exactly one source of truth for what committed and in what order, and the CDC stream cannot diverge from the data because it is the data, just read differently. The API surface is a single function, change_events_since(last_event_id), which replays the WAL records, correlates each Op::TxnCommit with its staged operations, groups them by transaction, and returns a CdcBatch containing one ChangeEvent per committed transaction with a stable ID shaped as <commit_epoch>:<operation_index>, so a consumer can persist the last ID it processed and resume from exactly there on the next poll without missing or duplicating a transaction.

The design choice that matters here is that aborted transactions never appear in the CDC stream, because the reconstruction logic keys on Op::TxnCommit records, and a transaction that rolled back never wrote one, which means your downstream consumer never sees a write that did not durably commit; this is the guarantee that external WAL tailers have to reconstruct for themselves, and they get it wrong in the gap between the WAL entry and the commit marker, where a half-applied transaction looks like data until you cross-reference the commit log and discover it was never finalized. The engine test suite exercises this by staging a put, rolling it back, and confirming the CDC stream contains no put events for that transaction:

let mut transaction = db.begin();
transaction.put("items", vec![(1, Value::Int64(1))]).unwrap();
transaction.rollback();

assert!(db
    .change_events_since(None)
    .unwrap()
    .events
    .iter()
    .all(|event| event.op != "put"));

A rolled-back put produces no CDC event, full stop, and that is not a filter applied after the fact but a consequence of the reconstruction logic skipping any WAL records that lack a matching commit.

The gap signal, and why it matters

A derived CDC stream has one honest limitation that a separate log does not: the WAL segments are finite, and once the engine checkpoints and purges old segments to reclaim space, the records needed to reconstruct older events are gone. MongrelDB handles this by returning gap = true in the CdcBatch when the consumer’s resume point falls before the oldest retained commit, which is the engine saying “I cannot give you a continuous stream from where you left off, so do not pretend you have one”; the correct consumer response is to rebootstrap, not to silently skip ahead, because a gap in a change stream is a data integrity problem, not a performance optimization. The retention is tunable via set_replication_wal_retention_segments, and the test suite verifies the gap behavior by writing data, checkpointing to purge segments, then resuming from a now-evicted position:

db.set_replication_wal_retention_segments(1);
// ... write and checkpoint multiple transactions ...
let resumed = db.change_events_since(Some(&first_id)).unwrap();
assert!(resumed.gap);

The bounds are generous but real: at most one million WAL records replayed, 256 MiB of replay bytes, 100,000 events, one million rows, and 256 MiB of total retained bytes, after which the engine fails closed rather than silently truncating, because a change stream that drops events without telling you is worse than no stream at all.

Replication as WAL streaming

The replication surface is the same idea at a different granularity, because instead of reconstructing logical change events, the follower receives raw committed WAL records over HTTP and replays them into its own local database directory, which means the follower’s logical state at any given commit epoch matches the leader’s, not a best-effort approximation; the physical bytes on disk may diverge once the follower independently checkpoints and compacts on its own schedule, but the data you can read at a shared epoch is identical. The daemon exposes GET /wal/stream?since=<seq>, the follower polls it, and the ReplicationFollower client in mongreldb-client wraps the whole loop:

use mongreldb_client::ReplicationFollower;

let mut follower = ReplicationFollower::new("http://leader:8453", "/local/copy")?;
follower.sync()?;  // fetch + apply all new records since last sync
println!("caught up to epoch {}", follower.last_epoch());

The follower bootstraps from a snapshot if the local copy does not exist yet, refuses to overwrite a non-replica database directory, and advances its watermark epoch only after the records are durably appended and the local database has recovered, so a crash mid-sync does not corrupt the follower or leave it in a half-applied state; it simply resumes from the last confirmed epoch on the next sync() call. This is asynchronous leader-to-follower replication designed for read scaling and for warm standby copies, distinct from the synchronous Raft consensus replication that MongrelDB also supports in cluster mode, and the two coexist because they solve different problems: Raft gives you strong consistency across a quorum for write availability and zero-data-loss failover, while WAL streaming gives you a geographically distant read copy or an analytics replica without paying the latency of a cross-region quorum ack. The honest tradeoff is that async streaming has an RPO greater than zero, because writes the leader committed but had not yet streamed are lost if the leader dies before the next poll, and if you need RPO 0 that is what the Raft quorum path is for; the async follower is the right tool for read scaling and for a recovery target that tolerates a small amount of tail loss, not a substitute for consensus-backed durability.

What changes and what stays the same

The thing that changes when an embedded engine is also a replicating engine is that the WAL stops being a purely internal recovery artifact and becomes a public contract, because external consumers depend on its ordering, its commit semantics, and its retention window, and those properties now have to hold across versions and across restarts. The thing that stays the same is the commit path itself, because the engine does not do extra work on the write side to produce the CDC stream or feed the replication follower; it writes the same WAL records it would write anyway, and the consumers read them after the fact, which means there is no throughput tax on writes for the benefit of change data capture until somebody actually calls change_events_since or polls /wal/stream. The cost moves to the read side, where WAL reconstruction and network transfer do real work, and to the storage side, where WAL retention has to be wide enough to cover your consumers’ polling interval, and those are knobs you turn based on your actual workload rather than overhead you pay whether you use the feature or not.

The modern equivalent of an old pattern

In the MySQL world, if you wanted change data capture in 2010, you parsed the binlog, and if you wanted replication, you configured a replica and hoped the binlog position tracking held up across schema changes and version upgrades, and the binlog was a separate log from the InnoDB redo log, which meant there were two sources of truth for what committed and they could disagree in edge cases around crash recovery. MongrelDB collapses that to one log: the WAL is the redo log and the replication log and the CDC source, and there is no second log to keep in sync, no binlog position to lose, and no external connector to reverse-engineer your commit semantics, because the engine that writes the data is the same engine that tells you what changed.