MongrelDB Architecture: From WAL to Hybrid Search

A database architecture diagram can make almost anything look coherent, because arrows do not wait for fsync, boxes do not retain old row versions for a reader that started twelve seconds ago, and the neat cylinder labeled “index” never has to explain whether its results are exact, approximate, stale, encrypted, rebuildable, or even part of the same transaction as the row it claims to describe; the useful way to understand MongrelDB is to follow one row from the moment an application presents it to the engine, through the durability and visibility boundaries, into immutable columnar storage, and then back out through a query that combines ordinary predicates with dense and sparse retrieval.

That route exposes the reason MongrelDB exists. It is not an attempt to make one data structure win every argument, and it is not MongoDB with an extra syllable. It is an open-source Rust database that keeps operational writes, analytical SQL, exact text constraints, approximate vector retrieval, sparse ranking, encryption, and optional server access inside one storage and recovery model, which is a useful shape when an application needs several of those capabilities together and an unnecessary shape when SQLite, DuckDB, a key-value engine, or a dedicated vector service already matches the workload.

This article describes the current implementation, not an imaginary finished state. Where a subsystem is integrated but not qualified against a clean exact-SHA release artifact, the distinction is stated; where an index is approximate, it stays approximate in the prose; where a benchmark measures accepted work rather than durable work, the storage barrier does not disappear because the smaller number looks better in a table.

Start with ownership, because every later guarantee depends on it

Before a row can be written, one runtime has to own the database root. An embedded Database opens the directory and takes an exclusive lease, so threads inside one process can share the same core while a second independent process, including one reaching the directory through a path alias, receives DatabaseLocked rather than creating a second transaction history against the same files; the shared-handle path allows multiple identity-bearing handles inside one process to attach to one DatabaseCore, but those handles still share one storage owner.

That rule is deliberately boring. File locking is not a clustering protocol, and two processes appending to their own idea of a local WAL will not converge because both filenames happen to live under the same directory. When several processes need one database, mongreldb-server becomes the owner and clients cross a protocol boundary; when one application owns the lifecycle, embedded mode avoids that boundary and pays the corresponding cost in native packaging, process-level failure, and application-managed maintenance.

The durable root also carries a storage-mode marker identifying standalone, server-owned standalone, or cluster-replica ownership. Current cluster and replicated paths are present in the codebase, but the public implementation-status matrix keeps them at Integrated until exact-SHA packaged qualification succeeds, so the conservative production center remains embedded or single-node server operation.

A row is an identity plus versions, not a mutable slot

MongrelDB gives each logical row a RowId, then stores committed versions against epochs. An update does not seek to a fixed disk offset and overwrite a record in place; it creates a newer version, while snapshots determine which version a reader may see. That choice is the thread connecting the write path, MVCC, compaction, secondary indexes, change capture, backup, and hybrid search, because every one of those systems needs a stable way to refer to the row while its physical representation changes.

The schema is typed. Core values include integers, floating-point numbers, bytes, booleans, decimals, JSON, arrays, intervals, embeddings, and other documented types, and indexes attach to columns through explicit definitions rather than appearing through a schemaless side channel. Native JSON exists for data that is genuinely irregular, but frequently filtered fields still belong in typed columns when the application expects predictable indexing and query behavior; storing JSON does not make MongrelDB MongoDB-compatible, and a native document value does not imply that every nested path receives an index.

Primary-key lookup is automatic, although one implementation detail deserves plain language: the current primary-key surface uses an ordered-map stand-in, not the completed HOT trie described by some older architectural material. PMA exists as an internal mutable-run tier, not a seventh user-creatable secondary index. The accurate public count is six secondary index families, and making the count larger by promoting internal machinery would only make the explanation less useful.

The write path begins before the WAL

An application write reaches more than a serializer. The transaction path resolves defaults and generated values, applies permissions and row-level policy where the credentialed surface requires them, expands relevant trigger and constraint actions, checks unique and foreign-key rules, and stages the final row version; only after that work succeeds can the engine prepare authoritative commit commands.

Generated embeddings make this ordering visible. MongrelDB does not hard-code an external model vendor into core storage, and applications may supply vectors directly, but a configured generated column can invoke a registered embedding provider from final source cells, validate count, dimension, normalization, finiteness, deadline, and cancellation, then stage the materialized vector and provenance with the source row. Under the currently exposed synchronous AbortWrite policy, provider failure aborts the whole source write before any WAL append, while replication carries the resulting vector instead of calling the provider again on each follower.

That is the only defensible transaction boundary for a generated value that the application treats as part of the row. Committing text first and hoping an asynchronous vector job catches up later is a valid architecture, but it is a two-state architecture and has to expose pending, failed, stale, and retry semantics; MongrelDB’s current synchronous path chooses the narrower contract, which costs provider latency during the write but avoids presenting a committed row whose required generated embedding never existed.

For an ordinary embedded write, the API remains direct:

use mongreldb_core::Value;

let row_id = db.put(vec![
    (1, Value::Int64(42)),
    (2, Value::Bytes(b"published".to_vec())),
    (3, Value::Bytes(b"wal recovery and hybrid search".to_vec())),
    (4, Value::Embedding(query_vector)),
])?;

db.commit()?;

The important line is the last one. put admits work into the write path; commit crosses the durability boundary.

Commit means the log reached stable storage

MongrelDB routes every commit through a commit-log contract whose standalone implementation wraps the shared write-ahead log and group commit. The engine appends authoritative commands to the WAL, obtains a commit receipt after the durability operation, applies committed state, and only then publishes visibility to readers; the WAL is the recovery authority for committed changes that have not yet reached immutable sorted runs.

This is where benchmark language matters. On the published Intel Core Ultra 9 386H Linux host with local NVMe storage, an accepted put without fsync measured 4.4828 microseconds, while a durable commit including fsync measured 4.6721 milliseconds. Both measurements are real and they answer different questions. The first says how quickly the engine accepted a row into its write path; the second includes the filesystem, kernel, controller, and storage device agreeing that the log reached stable media.

Group commit does not make an individual storage barrier vanish. It lets concurrent committers share one barrier, so several transactions can ride in the same WAL flush instead of each forcing an independent one. A batch of 1,000 puts plus one commit measured 7.7071 milliseconds, or 129.75 thousand rows per second on that machine, which is useful evidence for transaction shaping and not permission to describe every row as independently durable in 7.7 microseconds.

Recovery follows the same contract in reverse. A process reopens the root, validates durable metadata, replays committed WAL records not represented in stable runs, reconstructs the latest visible state, and refuses to turn an incomplete transaction into a committed one merely because a partial frame reached disk. The WAL article goes deeper into group commit and the difference between accepted and durable work.

The mutable layers buy time for columnar storage

A durable WAL is excellent for recovery and a poor final table format. Scanning a long history of row-level log records to answer every query would move the complexity from commit to read, so committed versions feed a B-epsilon-tree memtable keyed by (RowId, Epoch), with the internal PMA mutable-run tier helping bridge active state and immutable runs.

The B-epsilon-tree choice keeps the write side append-friendly while preserving ordered access to versions. It does not mean readers look only in one tree, because the latest visible row may live in the memtable, a mutable run, or one of several immutable sorted runs; a read merges those layers under a snapshot and chooses the newest version whose commit epoch is visible to that reader.

That merge is the price paid for deferring rewrite work. An LSM-style design makes writes cheaper by allowing several generations to coexist, then spends read work consulting generations and maintenance work combining them later; the architecture is attractive when writes and flushes are shaped well, and it becomes unpleasant when a short-lived process repeatedly opens, writes, exits, and never flushes or compacts.

MVCC keeps old truth alive long enough

A snapshot pins an epoch. A reader sees versions with committed epochs at or before that snapshot and ignores newer commits, which gives the query a stable view while writers continue. An update can therefore create a new row version without waiting for every older reader to finish, but the engine cannot reclaim the old version until no active subsystem still needs it.

MongrelDB tracks retention through several pin sources, including transaction snapshots, configured history retention, backup and point-in-time recovery, replication, immutable read generations, and online index builds. Garbage collection computes the oldest required version across those sources rather than assuming that the oldest SQL transaction is the only consumer of history; a backup copying one generation and an index build catching up from another are readers too, even if neither looks like SELECT in an application log.

This is the detail architecture pictures omit because “old files remain until all pins release” does not fit in a tidy arrow. It is also the detail that prevents compaction from deleting a run beneath a cursor, backup, or index generation that still references it.

Flush turns row versions into PAX columnar runs

When mutable state flushes, MongrelDB writes immutable .sr sorted-run files. The format uses PAX-style pages, keeping columns together within a page so a query can decode the requested fields without reconstructing every value in every row; adaptive per-column encoding can select delta encoding for ordered integers, dictionary encoding for low-cardinality values, Zstd for high-cardinality data, or passthrough storage when another codec would add cost without enough benefit.

Columns are divided into pages of up to 65,536 rows, and page statistics carry min, max, and null-count information used for pruning. A range predicate can reject a page whose bounds cannot contain a match before decoding its payload, while projection pushdown avoids decoding columns the query never requested. Sorted runs are memory-mapped, so readers can operate on mapped regions instead of issuing one system call per page.

PAX is a compromise, and the reason for the compromise matters. A pure row layout keeps one record convenient for point access but wastes bandwidth when an analytical query wants three columns from a million rows; a pure column store makes large scans natural but can turn small operational writes and row reconstruction into the dominant cost. MongrelDB accepts writes through the WAL and mutable layers, then gives settled data a column-friendly representation because its target workload needs both operational changes and local scans.

Bulk load takes a different path because importing a prepared dataset is not the same operation as accepting independent transactions. The typed bulk loader can write columnar storage directly and bypass ordinary per-row WAL work; that is why the benchmark for one million typed rows, 58.471 milliseconds on the published fixture, must not be presented as one million separately durable transactions.

Compaction is a version-preserving rewrite

Several immutable runs eventually make reads consult too many generations, so compaction reads live versions, writes a new clean run, syncs it, atomically publishes the new run list, and retires the old files. Readers holding older snapshots continue against retained generations, and garbage collection reclaims retired runs only when version pins and backup file pins permit it.

A crash during compaction must leave either the old published topology or the new published topology, never a manifest pointing at half a run. The old run set remains authoritative until the new run and manifest transition are durable, which makes compaction a storage transaction rather than a housekeeping script that happens to rewrite files.

Long-running daemons can compact and collect garbage periodically. Short-lived embedded and CLI processes should close with a flush and schedule maintenance appropriate to their write pattern, because a WAL that can recover every commit is still not a reason to retain every segment and run forever.

Secondary indexes are rebuildable views over committed rows

The committed row version is authoritative. Secondary indexes accelerate access and can be checkpointed, validated, rebuilt, or replaced without becoming a second database that the application has to reconcile. Every public secondary family resolves candidates back to the same RowId domain:

The exact versus approximate column is not documentation decoration. Bitmap, LearnedRange, FmIndex, and Sparse paths must complete or return an explicit work-budget error rather than silently truncate; ANN and MinHash can miss candidates by design, so query traces expose cap hits and underfill reasons, and the deterministic churn oracles enforce documented recall floors for specific test corpora and configurations rather than pretending one recall number applies to every production dataset.

ANN also separates graph algorithm from vector representation. HNSW supports binary-sign, dense, and product-quantized modes; DiskANN and IVF currently pair with dense vectors; product quantization uses a flat PQ backend even though the compatibility selector remains hnsw, so calling that combination an HNSW graph would be wrong. Approximate candidates can be reranked against stored full-precision vectors when an application needs a bounded exact-distance stage after candidate generation.

Online index creation and replacement build a hidden generation from a pinned snapshot, catch up committed deltas, validate, and publish with a short barrier. The source table does not become unavailable for the entire build, and an algorithm change never silently rewrites the meaning of an existing index in place.

Native conditions make the RowId model explicit

The typed Condition API shows the simplest query shape: each hard condition produces RowIds, the engine intersects those sets, and only surviving rows and requested columns are materialized.

let q = Query::new()
    .and(Condition::BitmapEq {
        column_id: 2,
        value: b"published".to_vec(),
    })
    .and(Condition::RangeF64 {
        column_id: 5,
        lo: 50.0,
        lo_inclusive: true,
        hi: 100.0,
        hi_inclusive: true,
    })
    .and(Condition::FmContains {
        column_id: 3,
        pattern: b"wal".to_vec(),
    })
    .and(Condition::Ann {
        column_id: 4,
        query: query_embedding,
        k: 20,
    });

let rows = db.query(&q)?;

This query means all four conditions, not “blend four relevance scores.” The Bitmap, range, and FM paths are exact hard filters; ANN contributes an approximate candidate set; intersection keeps only RowIds present in every set. A highly selective hard filter can leave fewer than k ANN results, which is more honest than quietly returning unauthorized or out-of-scope rows to fill the requested count.

Projection can return typed native columns instead of row objects, and the Arrow bridge can construct all-non-null integer and floating-point arrays directly from typed buffers. The point is not that every query should use the native API; the point is that the index contract remains visible when an application needs precise control over query construction.

DataFusion turns the same storage into SQL

MongrelDB registers its tables with DataFusion 54 for SQL planning and execution. Recognized predicates push into engine access paths before the scan: equality can use primary-key or Bitmap lookup, ranges can use LearnedRange candidates and page pruning, and LIKE '%text%' can use FM candidates before DataFusion rechecks SQL pattern semantics. Projection pushdown asks the storage layer for only the columns needed by the plan.

Predicates the engine cannot translate remain DataFusion filters. That fallback is a correctness rule: pushdown may reduce work, but failure to push a complex expression must not change the rows returned. SQL also supplies joins across tables in one database, recursive CTEs, window functions, materialized views, CREATE TABLE AS SELECT, JSON functions, planner inspection, and scored retrieval table functions.

Trusted embedded SQL may use Boolean ANN and sparse predicates. Remote SQL requires scored functions such as ann_search_scored, sparse_search_scored, and hybrid_search_scored, because a remote request needs explicit deadlines, work budgets, candidate limits, result ceilings, and concurrency admission; carrying an unbounded local assumption across a network boundary is how one authenticated query turns into a resource-exhaustion bug.

Hybrid search is a different operation from condition intersection

MongrelDB has two related query shapes that should not be collapsed into one slogan. Native conditions are strict conjunctions. The scored retrieval and SearchRequest surfaces apply hard filters, run one or more named retrievers, union their candidates, and combine rank positions through deterministic reciprocal-rank fusion; an optional exact-vector stage can then rerank a bounded window while preserving component scores, fused score, exact score, final score, and final rank.

That distinction lets each signal do the job it understands. Dense ANN finds semantic neighbours; sparse vectors preserve weighted lexical evidence and rare terms; FM catches literal fragments such as an incident code; Bitmap enforces tenant, status, or category; LearnedRange constrains time and numeric windows; MinHash proposes near-duplicate sets. The useful property is not that MongrelDB invented those structures, because it did not, but that every candidate returns to one RowId and one snapshot instead of becoming a foreign document identifier that the application joins back to a separately committed source row.

Reciprocal-rank fusion also avoids pretending that cosine distance, sparse dot product, exact containment, and set similarity share one raw score scale. The method has parameters and can still rank poorly for a particular corpus, so relevance needs labelled queries and regression fixtures, but rank fusion is a defensible baseline where adding incomparable numbers is not.

The embedded hybrid search guide separates dense, sparse, literal, and metadata signals in more detail, while the local RAG guide covers the source identifiers, model versions, access policy, and provenance that belong beside an embedding.

Encryption follows the data through each durable layer

Encryption at rest is not one checkbox applied to a directory after the engine finishes writing it. MongrelDB derives a key-encryption key from a passphrase through Argon2id and HKDF, accepts high-entropy raw keys, or unwraps a root key through HashiCorp Vault Transit; per-run data-encryption keys protect page payloads with AES-256-GCM, while separate derived keys protect WAL frames, the persistent result cache, and the global index checkpoint.

Per-page min and max statistics for encrypted columns live in an encrypted statistics envelope so page pruning does not require plaintext zone-map values. Run headers and directories remain structurally readable, but a keyed HMAC authenticates them, while manifests and schema remain unencrypted metadata. Those boundaries matter: encryption protects dormant payloads from a copied disk or backup when keys are separate, but it does not hide file sizes, all schema information, query access patterns, or plaintext inside a trusted process that has loaded the key.

ENCRYPTED_INDEXABLE scalar columns add deterministic equality tokens or order-preserving range tokens so selected predicates can narrow candidates without decrypting every row first. Equality tokens reveal repetition and frequency; range tokens reveal order; neither mechanism is encrypted ANN, homomorphic vector search, or protection from a compromised application process. The encrypted vector database guide treats embeddings, index artifacts, metadata, backups, and query-time exposure as separate parts of the threat model.

Memory pressure and temporary data need the same discipline. The single-node core includes a memory governor, resource groups, and per-query spill management; encrypted databases seal spill-frame payloads with AES-256-GCM, spill files are checksummed and bounded, and dropped sessions clean their query-specific directories. These subsystems are integrated in the current architecture program, but the implementation-status matrix remains the authority for whether a particular release artifact has completed exact-SHA qualification.

Caches shorten reads without becoming the source of truth

MongrelDB has an in-memory and persistent result-cache path with query-footprint and condition-column invalidation, plus an Arrow IPC shadow for clean single-run scans. These are derived accelerators. A cache entry can disappear without losing committed data, and a stale or invalid index checkpoint can be rebuilt from authoritative row versions; this separation is what lets recovery reason about the WAL and runs first, then restore faster paths without inventing another source of truth.

Caching still has correctness obligations. A commit affecting the columns or footprint of a cached query has to invalidate that result, encryption has to cover a persistent cache derived from encrypted tables, and resource governance has to keep cache growth from starving foreground queries or recovery. “It is only a cache” is an explanation of rebuildability, not permission to return an old answer.

The benchmark profile says where work happens

The current published measurements were collected from release builds on Linux x86-64 with an Intel Core Ultra 9 386H, 62 GiB of RAM, local NVMe storage, and the toolchain recorded in BENCHMARKS.md. Selected results show the shape of the engine rather than a cross-database contest:

The embedded and HTTP point numbers are not the same operation with TCP subtracted from one. One is a native begin/get/rollback path; the other includes request handling, session lookup, SQL planning, and serialization. Likewise, the typed bulk loader is not a million independently durable inserts, and the accepted put is not a committed transaction.

Concurrent mixed work changes the profile again. Four writers and four readers at one-million-row scale recorded 33.012 milliseconds commit p50 and 176.885 milliseconds p99 in the published qualification fixture, which is far above the single-committer durable result because readers, writers, snapshots, and shared hardware were competing at once. Production evaluation needs that mixed shape, not only the smallest isolated median.

What this architecture does not promise

MongrelDB does not open SQLite files, preserve SQLite APIs, or inherit SQLite’s decades of deployment history. It does not implement the MongoDB wire protocol or MongoDB query model. Its FM-index answers exact substring containment and should not be described as a complete BM25 search engine; sparse retrieval supplies a different ranked lexical signal. ANN and MinHash remain approximate. The current primary-key implementation is not the planned HOT trie. PMA is internal. Six secondary index families are user-creatable, not seven, nine, or whatever number results from counting every internal map and cache.

The engine also does not turn one machine into a distributed service by having server and cluster modules in the repository. Replicated and sharded paths are Integrated according to the current status matrix and remain subject to exact-SHA packaged qualification. A team that needs managed multi-region availability, independent vector scaling, or an established warehouse ecosystem should choose a system centered on those requirements rather than asking an embedded engine to become one through configuration.

This is why the architecture is most credible at its narrow center: one application or one server owns a database root; operational rows, analytical projections, and mixed retrieval need one recovery model; specialized indexes save operating separate local services; and the team accepts the responsibility of evaluating a younger engine against its own workload.

Reproduce the claims before adopting the engine

The shortest useful evaluation is not a feature tour. Build one table shaped like production, execute one durable transaction, restart, query through the most important hard predicates and retrieval signals, compact, back up, restore, and compare the restored results; then run the largest analytical query while writes continue and record commit p50 and p99, memory, temporary disk, candidate-cap events, and cancellation behavior.

The source exposes the relevant checks:

git clone https://github.com/visorcraft/MongrelDB.git
cd MongrelDB

# Core tests and the documented benchmark families.
cargo test -p mongreldb-core
cargo bench -p mongreldb-core --bench write_path -- --noplot
cargo bench -p mongreldb-core --bench scale -- --noplot
cargo bench -p mongreldb-core --bench filtered_query -- --noplot

# Inspect current release status before repeating an architecture claim.
cat docs/architecture/implementation-status.md

Run those commands on deployment-class hardware, pin the exact commit, keep the fixture and configuration beside the results, and distinguish a test threshold from a service-level objective. If the application uses encryption, enable it for the evaluation. If it uses generated embeddings, test provider failure and model-version migration. If it uses the daemon, measure the daemon rather than extrapolating from an embedded call. If it relies on ANN, build brute-force ground truth and report recall beside latency.

One recovery model is the point

The enduring idea in MongrelDB is not HNSW, DataFusion, PAX, B-epsilon trees, or reciprocal-rank fusion in isolation; all are established ideas with their own literature and mature implementations elsewhere. The idea is to make them answer to one committed row identity, one visibility model, and one ownership boundary, so an application can update operational state, query it analytically, retrieve it semantically, constrain it literally, and recover it after a crash without reconciling three independently committed local systems.

That consolidation is valuable only when the workload needs it. For ordinary relational state, use SQLite until there is a concrete reason not to. For scan-first analytics, start with DuckDB. For an independently operated vector workload, use a vector service. When the application genuinely needs durable local writes, column-friendly scans, SQL, dense and sparse retrieval, exact substring constraints, metadata filters, and encryption under one process or one single-node owner, MongrelDB is a coherent engine to put through a serious fixture, and the fixture, not the architecture diagram, gets the final word.