That route exposes the reason MongrelDB exists. It is not an attempt to make one data structure win every argument, and it is not MongoDB with an extra syllable. It is an open-source Rust database that keeps operational writes, analytical SQL, exact text constraints, approximate vector retrieval, sparse ranking, encryption, and optional server access inside one storage and recovery model, which is a useful shape when an application needs several of those capabilities together and an unnecessary shape when SQLite, DuckDB, a key-value engine, or a dedicated vector service already matches the workload.

This article describes the current implementation, not an imaginary finished state. Where a subsystem is integrated but not qualified against a clean exact-SHA release artifact, the distinction is stated; where an index is approximate, it stays approximate in the prose; where a benchmark measures accepted work rather than durable work, the storage barrier does not disappear because the smaller number looks better in a table.

Start with ownership, because every later guarantee depends on it

Before a row can be written, one runtime has to own the database root. An embedded Database opens the directory and takes an exclusive lease, so threads inside one process can share the same core while a second independent process, including one reaching the directory through a path alias, receives DatabaseLocked rather than creating a second transaction history against the same files; the shared-handle path allows multiple identity-bearing handles inside one process to attach to one DatabaseCore, but those handles still share one storage owner.

That rule is deliberately boring. File locking is not a clustering protocol, and two processes appending to their own idea of a local WAL will not converge because both filenames happen to live under the same directory. When several processes need one database, mongreldb-server becomes the owner and clients cross a protocol boundary; when one application owns the lifecycle, embedded mode avoids that boundary and pays the corresponding cost in native packaging, process-level failure, and application-managed maintenance.

The durable root also carries a storage-mode marker identifying standalone, server-owned standalone, or cluster-replica ownership. Current cluster and replicated paths are present in the codebase, but the public implementation-status matrix keeps them at Integrated until exact-SHA packaged qualification succeeds, so the conservative production center remains embedded or single-node server operation.

A row is an identity plus versions, not a mutable slot

MongrelDB gives each logical row a RowId, then stores committed versions against epochs. An update does not seek to a fixed disk offset and overwrite a record in place; it creates a newer version, while snapshots determine which version a reader may see. That choice is the thread connecting the write path, MVCC, compaction, secondary indexes, change capture, backup, and hybrid search, because every one of those systems needs a stable way to refer to the row while its physical representation changes.

The schema is typed. Core values include integers, floating-point numbers, bytes, booleans, decimals, JSON, arrays, intervals, embeddings, and other documented types, and indexes attach to columns through explicit definitions rather than appearing through a schemaless side channel. Native JSON exists for data that is genuinely irregular, but frequently filtered fields still belong in typed columns when the application expects predictable indexing and query behavior; storing JSON does not make MongrelDB MongoDB-compatible, and a native document value does not imply that every nested path receives an index.

Primary-key lookup is automatic, although one implementation detail deserves plain language: the current primary-key surface uses an ordered-map stand-in, not the completed HOT trie described by some older architectural material. PMA exists as an internal mutable-run tier, not a seventh user-creatable secondary index. The accurate public count is six secondary index families, and making the count larger by promoting internal machinery would only make the explanation less useful.

The write path begins before the WAL

An application write reaches more than a serializer. The transaction path resolves defaults and generated values, applies permissions and row-level policy where the credentialed surface requires them, expands relevant trigger and constraint actions, checks unique and foreign-key rules, and stages the final row version; only after that work succeeds can the engine prepare authoritative commit commands.

Generated embeddings make this ordering visible. MongrelDB does not hard-code an external model vendor into core storage, and applications may supply vectors directly, but a configured generated column can invoke a registered embedding provider from final source cells, validate count, dimension, normalization, finiteness, deadline, and cancellation, then stage the materialized vector and provenance with the source row. Under the currently exposed synchronous AbortWrite policy, provider failure aborts the whole source write before any WAL append, while replication carries the resulting vector instead of calling the provider again on each follower.

That is the only defensible transaction boundary for a generated value that the application treats as part of the row. Committing text first and hoping an asynchronous vector job catches up later is a valid architecture, but it is a two-state architecture and has to expose pending, failed, stale, and retry semantics; MongrelDB’s current synchronous path chooses the narrower contract, which costs provider latency during the write but avoids presenting a committed row whose required generated embedding never existed.

For an ordinary embedded write, the API remains direct:

use mongreldb_core::Value;

let row_id = db.put(vec![
    (1, Value::Int64(42)),
    (2, Value::Bytes(b"published".to_vec())),
    (3, Value::Bytes(b"wal recovery and hybrid search".to_vec())),
    (4, Value::Embedding(query_vector)),
])?;

db.commit()?;

The important line is the last one. put admits work into the write path; commit crosses the durability boundary.

Commit means the log reached stable storage

MongrelDB routes every commit through a commit-log contract whose standalone implementation wraps the shared write-ahead log and group commit. The engine appends authoritative commands to the WAL, obtains a commit receipt after the durability operation, applies committed state, and only then publishes visibility to readers; the WAL is the recovery authority for committed changes that have not yet reached immutable sorted runs.

This is where benchmark language matters. On the published Intel Core Ultra 9 386H Linux host with local NVMe storage, an accepted put without fsync measured 4.4828 microseconds, while a durable commit including fsync measured 4.6721 milliseconds. Both measurements are real and they answer different questions. The first says how quickly the engine accepted a row into its write path; the second includes the filesystem, kernel, controller, and storage device agreeing that the log reached stable media.

Group commit does not make an individual storage barrier vanish. It lets concurrent committers share one barrier, so several transactions can ride in the same WAL flush instead of each forcing an independent one. A batch of 1,000 puts plus one commit measured 7.7071 milliseconds, or 129.75 thousand rows per second on that machine, which is useful evidence for transaction shaping and not permission to describe every row as independently durable in 7.7 microseconds.

Recovery follows the same contract in reverse. A process reopens the root, validates durable metadata, replays committed WAL records not represented in stable runs, reconstructs the latest visible state, and refuses to turn an incomplete transaction into a committed one merely because a partial frame reached disk. The WAL article goes deeper into group commit and the difference between accepted and durable work.

The mutable layers buy time for columnar storage

A durable WAL is excellent for recovery and a poor final table format. Scanning a long history of row-level log records to answer every query would move the complexity from commit to read, so committed versions feed a B-epsilon-tree memtable keyed by (RowId, Epoch), with the internal PMA mutable-run tier helping bridge active state and immutable runs.

The B-epsilon-tree choice keeps the write side append-friendly while preserving ordered access to versions. It does not mean readers look only in one tree, because the latest visible row may live in the memtable, a mutable run, or one of several immutable sorted runs; a read merges those layers under a snapshot and chooses the newest version whose commit epoch is visible to that reader.

That merge is the price paid for deferring rewrite work. An LSM-style design makes writes cheaper by allowing several generations to coexist, then spends read work consulting generations and maintenance work combining them later; the architecture is attractive when writes and flushes are shaped well, and it becomes unpleasant when a short-lived process repeatedly opens, writes, exits, and never flushes or compacts.

MVCC keeps old truth alive long enough

A snapshot pins an epoch. A reader sees versions with committed epochs at or before that snapshot and ignores newer commits, which gives the query a stable view while writers continue. An update can therefore create a new row version without waiting for every older reader to finish, but the engine cannot reclaim the old version until no active subsystem still needs it.

MongrelDB tracks retention through several pin sources, including transaction snapshots, configured history retention, backup and point-in-time recovery, replication, immutable read generations, and online index builds. Garbage collection computes the oldest required version across those sources rather than assuming that the oldest SQL transaction is the only consumer of history; a backup copying one generation and an index build catching up from another are readers too, even if neither looks like SELECT in an application log.

This is the detail architecture pictures omit because “old files remain until all pins release” does not fit in a tidy arrow. It is also the detail that prevents compaction from deleting a run beneath a cursor, backup, or index generation that still references it.

Flush turns row versions into PAX columnar runs

When mutable state flushes, MongrelDB writes immutable .sr sorted-run files. The format uses PAX-style pages, keeping columns together within a page so a query can decode the requested fields without reconstructing every value in every row; adaptive per-column encoding can select delta encoding for ordered integers, dictionary encoding for low-cardinality values, Zstd for high-cardinality data, or passthrough storage when another codec would add cost without enough benefit.

Columns are divided into pages of up to 65,536 rows, and page statistics carry min, max, and null-count information used for pruning. A range predicate can reject a page whose bounds cannot contain a match before decoding its payload, while projection pushdown avoids decoding columns the query never requested. Sorted runs are memory-mapped, so readers can operate on mapped regions instead of issuing one system call per page.

PAX is a compromise, and the reason for the compromise matters. A pure row layout keeps one record convenient for point access but wastes bandwidth when an analytical query wants three columns from a million rows; a pure column store makes large scans natural but can turn small operational writes and row reconstruction into the dominant cost. MongrelDB accepts writes through the WAL and mutable layers, then gives settled data a column-friendly representation because its target workload needs both operational changes and local scans.

Bulk load takes a different path because importing a prepared dataset is not the same operation as accepting independent transactions. The typed bulk loader can write columnar storage directly and bypass ordinary per-row WAL work; that is why the benchmark for one million typed rows, 58.471 milliseconds on the published fixture, must not be presented as one million separately durable transactions.

Compaction is a version-preserving rewrite

Several immutable runs eventually make reads consult too many generations, so compaction reads live versions, writes a new clean run, syncs it, atomically publishes the new run list, and retires the old files. Readers holding older snapshots continue against retained generations, and garbage collection reclaims retired runs only when version pins and backup file pins permit it.

A crash during compaction must leave either the old published topology or the new published topology, never a manifest pointing at half a run. The old run set remains authoritative until the new run and manifest transition are durable, which makes compaction a storage transaction rather than a housekeeping script that happens to rewrite files.

Long-running daemons can compact and collect garbage periodically. Short-lived embedded and CLI processes should close with a flush and schedule maintenance appropriate to their write pattern, because a WAL that can recover every commit is still not a reason to retain every segment and run forever.

Secondary indexes are rebuildable views over committed rows

The committed row version is authoritative. Secondary indexes accelerate access and can be checkpointed, validated, rebuilt, or replaced without becoming a second database that the application has to reconcile. Every public secondary family resolves candidates back to the same RowId domain:

The exact versus approximate column is not documentation decoration. Bitmap, LearnedRange, FmIndex, and Sparse paths must complete or return an explicit work-budget error rather than silently truncate; ANN and MinHash can miss candidates by design, so query traces expose cap hits and underfill reasons, and the deterministic churn oracles enforce documented recall floors for specific test corpora and configurations rather than pretending one recall number applies to every production dataset.

ANN also separates graph algorithm from vector representation. HNSW supports binary-sign, dense, and product-quantized modes; DiskANN and IVF currently pair with dense vectors; product quantization uses a flat PQ backend even though the compatibility selector remains hnsw, so calling that combination an HNSW graph would be wrong. Approximate candidates can be reranked against stored full-precision vectors when an application needs a bounded exact-distance stage after candidate generation.

Online index creation and replacement build a hidden generation from a pinned snapshot, catch up committed deltas, validate, and publish with a short barrier. The source table does not become unavailable for the entire build, and an algorithm change never silently rewrites the meaning of an existing index in place.

Native conditions make the RowId model explicit

The typed Condition API shows the simplest query shape: each hard condition produces RowIds, the engine intersects those sets, and only surviving rows and requested columns are materialized.

let q = Query::new()
    .and(Condition::BitmapEq {
        column_id: 2,
        value: b"published".to_vec(),
    })
    .and(Condition::RangeF64 {
        column_id: 5,
        lo: 50.0,
        lo_inclusive: true,
        hi: 100.0,
        hi_inclusive: true,
    })
    .and(Condition::FmContains {
        column_id: 3,
        pattern: b"wal".to_vec(),
    })
    .and(Condition::Ann {
        column_id: 4,
        query: query_embedding,
        k: 20,
    });

let rows = db.query(&q)?;

This query means all four conditions, not “blend four relevance scores.” The Bitmap, range, and FM paths are exact hard filters; ANN contributes an approximate candidate set; intersection keeps only RowIds present in every set. A highly selective hard filter can leave fewer than k ANN results, which is more honest than quietly returning unauthorized or out-of-scope rows to fill the requested count.

Projection can return typed native columns instead of row objects, and the Arrow bridge can construct all-non-null integer and floating-point arrays directly from typed buffers. The point is not that every query should use the native API; the point is that the index contract remains visible when an application needs precise control over query construction.

DataFusion turns the same storage into SQL

MongrelDB registers its tables with DataFusion 54 for SQL planning and execution. Recognized predicates push into engine access paths before the scan: equality can use primary-key or Bitmap lookup, ranges can use LearnedRange candidates and page pruning, and LIKE '%text%' can use FM candidates before DataFusion rechecks SQL pattern semantics. Projection pushdown asks the storage layer for only the columns needed by the plan.

Predicates the engine cannot translate remain DataFusion filters. That fallback is a correctness rule: pushdown may reduce work, but failure to push a complex expression must not change the rows returned. SQL also supplies joins across tables in one database, recursive CTEs, window functions, materialized views, CREATE TABLE AS SELECT, JSON functions, planner inspection, and scored retrieval table functions.

Trusted embedded SQL may use Boolean ANN and sparse predicates. Remote SQL requires scored functions such as ann_search_scored, sparse_search_scored, and hybrid_search_scored, because a remote request needs explicit deadlines, work budgets, candidate limits, result ceilings, and concurrency admission; carrying an unbounded local assumption across a network boundary is how one authenticated query turns into a resource-exhaustion bug.

Hybrid search is a different operation from condition intersection

MongrelDB has two related query shapes that should not be collapsed into one slogan. Native conditions are strict conjunctions. The scored retrieval and SearchRequest surfaces apply hard filters, run one or more named retrievers, union their candidates, and combine rank positions through deterministic reciprocal-rank fusion; an optional exact-vector stage can then rerank a bounded window while preserving component scores, fused score, exact score, final score, and final rank.

That distinction lets each signal do the job it understands. Dense ANN finds semantic neighbours; sparse vectors preserve weighted lexical evidence and rare terms; FM catches literal fragments such as an incident code; Bitmap enforces tenant, status, or category; LearnedRange constrains time and numeric windows; MinHash proposes near-duplicate sets. The useful property is not that MongrelDB invented those structures, because it did not, but that every candidate returns to one RowId and one snapshot instead of becoming a foreign document identifier that the application joins back to a separately committed source row.

Reciprocal-rank fusion also avoids pretending that cosine distance, sparse dot product, exact containment, and set similarity share one raw score scale. The method has parameters and can still rank poorly for a particular corpus, so relevance needs labelled queries and regression fixtures, but rank fusion is a defensible baseline where adding incomparable numbers is not.

The embedded hybrid search guide separates dense, sparse, literal, and metadata signals in more detail, while the local RAG guide covers the source identifiers, model versions, access policy, and provenance that belong beside an embedding.

Encryption follows the data through each durable layer

Encryption at rest is not one checkbox applied to a directory after the engine finishes writing it. MongrelDB derives a key-encryption key from a passphrase through Argon2id and HKDF, accepts high-entropy raw keys, or unwraps a root key through HashiCorp Vault Transit; per-run data-encryption keys protect page payloads with AES-256-GCM, while separate derived keys protect WAL frames, the persistent result cache, and the global index checkpoint.

Per-page min and max statistics for encrypted columns live in an encrypted statistics envelope so page pruning does not require plaintext zone-map values. Run headers and directories remain structurally readable, but a keyed HMAC authenticates them, while manifests and schema remain unencrypted metadata. Those boundaries matter: encryption protects dormant payloads from a copied disk or backup when keys are separate, but it does not hide file sizes, all schema information, query access patterns, or plaintext inside a trusted process that has loaded the key.

ENCRYPTED_INDEXABLE scalar columns add deterministic equality tokens or order-preserving range tokens so selected predicates can narrow candidates without decrypting every row first. Equality tokens reveal repetition and frequency; range tokens reveal order; neither mechanism is encrypted ANN, homomorphic vector search, or protection from a compromised application process. The encrypted vector database guide treats embeddings, index artifacts, metadata, backups, and query-time exposure as separate parts of the threat model.

Memory pressure and temporary data need the same discipline. The single-node core includes a memory governor, resource groups, and per-query spill management; encrypted databases seal spill-frame payloads with AES-256-GCM, spill files are checksummed and bounded, and dropped sessions clean their query-specific directories. These subsystems are integrated in the current architecture program, but the implementation-status matrix remains the authority for whether a particular release artifact has completed exact-SHA qualification.

Caches shorten reads without becoming the source of truth

MongrelDB has an in-memory and persistent result-cache path with query-footprint and condition-column invalidation, plus an Arrow IPC shadow for clean single-run scans. These are derived accelerators. A cache entry can disappear without losing committed data, and a stale or invalid index checkpoint can be rebuilt from authoritative row versions; this separation is what lets recovery reason about the WAL and runs first, then restore faster paths without inventing another source of truth.

Caching still has correctness obligations. A commit affecting the columns or footprint of a cached query has to invalidate that result, encryption has to cover a persistent cache derived from encrypted tables, and resource governance has to keep cache growth from starving foreground queries or recovery. “It is only a cache” is an explanation of rebuildability, not permission to return an old answer.

The benchmark profile says where work happens

The current published measurements were collected from release builds on Linux x86-64 with an Intel Core Ultra 9 386H, 62 GiB of RAM, local NVMe storage, and the toolchain recorded in BENCHMARKS.md. Selected results show the shape of the engine rather than a cross-database contest:

The embedded and HTTP point numbers are not the same operation with TCP subtracted from one. One is a native begin/get/rollback path; the other includes request handling, session lookup, SQL planning, and serialization. Likewise, the typed bulk loader is not a million independently durable inserts, and the accepted put is not a committed transaction.

Concurrent mixed work changes the profile again. Four writers and four readers at one-million-row scale recorded 33.012 milliseconds commit p50 and 176.885 milliseconds p99 in the published qualification fixture, which is far above the single-committer durable result because readers, writers, snapshots, and shared hardware were competing at once. Production evaluation needs that mixed shape, not only the smallest isolated median.

What this architecture does not promise

MongrelDB does not open SQLite files, preserve SQLite APIs, or inherit SQLite’s decades of deployment history. It does not implement the MongoDB wire protocol or MongoDB query model. Its FM-index answers exact substring containment and should not be described as a complete BM25 search engine; sparse retrieval supplies a different ranked lexical signal. ANN and MinHash remain approximate. The current primary-key implementation is not the planned HOT trie. PMA is internal. Six secondary index families are user-creatable, not seven, nine, or whatever number results from counting every internal map and cache.

The engine also does not turn one machine into a distributed service by having server and cluster modules in the repository. Replicated and sharded paths are Integrated according to the current status matrix and remain subject to exact-SHA packaged qualification. A team that needs managed multi-region availability, independent vector scaling, or an established warehouse ecosystem should choose a system centered on those requirements rather than asking an embedded engine to become one through configuration.

This is why the architecture is most credible at its narrow center: one application or one server owns a database root; operational rows, analytical projections, and mixed retrieval need one recovery model; specialized indexes save operating separate local services; and the team accepts the responsibility of evaluating a younger engine against its own workload.

Reproduce the claims before adopting the engine

The shortest useful evaluation is not a feature tour. Build one table shaped like production, execute one durable transaction, restart, query through the most important hard predicates and retrieval signals, compact, back up, restore, and compare the restored results; then run the largest analytical query while writes continue and record commit p50 and p99, memory, temporary disk, candidate-cap events, and cancellation behavior.

The source exposes the relevant checks:

git clone https://github.com/visorcraft/MongrelDB.git
cd MongrelDB

# Core tests and the documented benchmark families.
cargo test -p mongreldb-core
cargo bench -p mongreldb-core --bench write_path -- --noplot
cargo bench -p mongreldb-core --bench scale -- --noplot
cargo bench -p mongreldb-core --bench filtered_query -- --noplot

# Inspect current release status before repeating an architecture claim.
cat docs/architecture/implementation-status.md

Run those commands on deployment-class hardware, pin the exact commit, keep the fixture and configuration beside the results, and distinguish a test threshold from a service-level objective. If the application uses encryption, enable it for the evaluation. If it uses generated embeddings, test provider failure and model-version migration. If it uses the daemon, measure the daemon rather than extrapolating from an embedded call. If it relies on ANN, build brute-force ground truth and report recall beside latency.

One recovery model is the point

The enduring idea in MongrelDB is not HNSW, DataFusion, PAX, B-epsilon trees, or reciprocal-rank fusion in isolation; all are established ideas with their own literature and mature implementations elsewhere. The idea is to make them answer to one committed row identity, one visibility model, and one ownership boundary, so an application can update operational state, query it analytically, retrieve it semantically, constrain it literally, and recover it after a crash without reconciling three independently committed local systems.

That consolidation is valuable only when the workload needs it. For ordinary relational state, use SQLite until there is a concrete reason not to. For scan-first analytics, start with DuckDB. For an independently operated vector workload, use a vector service. When the application genuinely needs durable local writes, column-friendly scans, SQL, dense and sparse retrieval, exact substring constraints, metadata filters, and encryption under one process or one single-node owner, MongrelDB is a coherent engine to put through a serious fixture, and the fixture, not the architecture diagram, gets the final word.

Calling this general full-text search is convenient but imprecise. The FM-index finds containment candidates for a literal pattern. MongrelDB also has a separate FTS document surface and ranking function, while sparse retrieval handles learned weighted terms. Those paths can work together, but an FM-index does not turn a substring into a tokenized relevance model by itself.

Why ordinary indexes miss %needle%

A sorted B-tree can seek a known prefix because values beginning with user: occupy one ordered interval. It cannot seek an arbitrary middle fragment without another representation, because strings containing needle may appear anywhere in lexical order.

A full scan solves the problem by checking every value. That is correct and often good enough for a small table, but cost grows with the bytes examined. The FM-index stores a transformed view of the corpus that supports backward search over the pattern, narrowing the suffix interval one symbol at a time through rank operations.

The search work depends primarily on pattern length plus the cost of locating and materializing matches. Output still matters: a two-byte pattern matching half the corpus cannot return half the corpus for free.

What the Burrows-Wheeler transform contributes

The Burrows-Wheeler transform rearranges text so repeated contexts cluster together. A wavelet tree supplies compact rank queries over that transformed sequence, and the FM-index combines those pieces with sampled location metadata so a matching interval can resolve back to candidate rows.

That structure is valuable when the application needs literal containment over stored text and does not want a separate search service. It is not automatically the best answer for hot write-heavy token search, fuzzy matching, typo tolerance, language stemming, or relevance-ranked document retrieval; those are different query models with different indexes.

How it appears in MongrelDB

The schema declares an FM secondary index on a bytes/text column:

IndexDef {
    name: "body_fm".into(),
    column_id: 4,
    kind: IndexKind::FmIndex,
}

A native condition requests containment:

Condition::FmContains {
    column_id: 4,
    pattern: b"error:42".to_vec(),
}

SQL can push down a recognized containment shape:

SELECT id, body
FROM logs
WHERE body LIKE '%error:42%';

The FM path produces candidate RowIds. DataFusion rechecks SQL pattern semantics where the index result is a superset, which keeps pushdown an optimization rather than a change in query truth. Native FmContains expresses the exact literal containment condition directly.

For anchored prefixes such as LIKE 'user:%', a Bitmap index on a bytes column can enumerate distinct keys and union matching prefixes exactly, which is usually a tighter path than asking the FM-index to solve a query whose anchor already gives the ordered key domain enough information.

Why the shared RowId matters

The substring result can intersect with a tenant Bitmap, a learned time range, an ANN candidate set, or any other hard condition before rows and requested columns are decoded. This is the useful part of keeping text search in the operational engine: the text match does not become a detached document identifier that the application must join back to a row stored somewhere else.

A hybrid query can ask for semantic neighbours whose body contains an exact code and whose timestamp falls inside an incident window. HNSW, FM, and PGM each do the work they were built for, then the engine combines their RowIds under one snapshot.

Index maintenance is derived state

The authoritative data is the committed row version. Secondary index generations can be checkpointed and rebuilt from runs plus mutable state. Online create, replace, and drop operations build or publish generations without rewriting the table, with a short barrier at publication.

That distinction matters for crash reasoning. The WAL protects committed changes to source data; the FM structure is an acceleration path that can be reconstructed, not a second authoritative database whose divergence the application must reconcile.

Maintenance cost still exists. Building a transformed corpus is more work than appending one posting to a small in-memory map, and frequent updates to large indexed text deserve a representative benchmark. MongrelDB exposes REINDEX and rebuild paths because no immutable text structure remains cheap under every update pattern.

When to choose another search path

Use a tokenized inverted index when the product needs word analysis, stemming, fuzzy terms, facets, mature BM25 tuning, or a search-only workload large enough to justify a dedicated engine. Use sparse retrieval when learned lexical expansion and weighted terms are the signal. Use ANN when semantic similarity matters. Use a Bitmap prefix path when the query is anchored and the distinct-key set is manageable.

Use the FM-index when exact substring containment is the requirement, the data belongs transactionally with the rest of the row, and combining that containment with ordinary database filters is worth more than operating a separate text-search service.

The current index behavior is documented in docs/06-indexes.md, and the broader embedded vector database guide shows how the text path combines with dense and sparse retrieval.

MongrelDB takes the embedded route. The operational row, its dense embedding, its sparse representation, and the metadata used to filter it all live under one database transaction model, while HNSW, DiskANN, IVF, Bitmap, FM, Sparse, MinHash, and learned-range indexes resolve candidates through the same RowId space. That does not make a single-node engine the right answer for every vector workload, but it changes the failure model in a way that matters more than shaving another fraction from an isolated nearest-neighbour benchmark.

For a broader selection checklist, including when a separate vector service is the better answer, start with the embedded vector database guide.

What in-process actually buys

The first gain is the obvious one: no network request sits between the application and the index. A native Rust call, the Node NAPI addon, and the Python binding enter the engine directly; HTTP clients still exist for applications that need a warm multi-process owner, but local embedding does not pay for TCP, request routing, JSON encoding, authentication middleware, and response serialization on each operation.

The second gain is more important: source data and model-derived data share one transaction boundary. An application can update a document, its tenant, its timestamp, and its embedding as one commit, then query against a snapshot that sees either the old row or the new row rather than a row from now and an embedding from thirty seconds ago.

The ANN graph itself is derived state, not the authoritative copy of the row. MongrelDB can checkpoint and rebuild secondary indexes from durable table data, which is the right boundary for a graph whose shape can depend on insertion order and implementation details; the WAL protects authoritative changes, index generations publish atomically, and a damaged or obsolete generation can be rebuilt without inventing missing source vectors.

HNSW is one choice, not the schema

MongrelDB separates the ANN algorithm from the stored representation, because those are different decisions even when libraries like to bundle them into one checkbox.

The current supported combinations are:

Unsupported combinations fail at index creation rather than silently falling back to a different algorithm. Replacing Dense HNSW with another supported generation is an online build followed by a short publication barrier; the table is not rewritten, and the schema does not lie about which backend is serving candidates.

A native condition remains small:

Condition::Ann {
    column_id: 6,
    query: vec![0.10, 0.45, 0.78, 0.23],
    k: 10,
}

The vector must match the column dimension and contain finite values. HNSW remains approximate, so production qualification compares sampled results against brute force rather than treating k = 10 as a promise that the mathematically closest ten rows always appear.

The point is the filters around the graph

A nearest-neighbour list is rarely the product query. The product query is closer to “find semantically similar documents for this tenant, created in the last ninety days, containing this exact identifier, excluding archived rows, then fuse dense and sparse relevance and rerank the best candidates exactly.”

MongrelDB handles that shape by making every index speak RowId. HNSW produces semantic candidates, Bitmap indexes enforce low-cardinality equality, PGM handles numeric or time ranges, FM handles exact substring containment, Sparse accepts SPLADE-style weighted terms, and MinHash supplies set-similarity candidates; hard filters reduce the candidate set, named retrievers fuse through deterministic reciprocal-rank fusion, and an optional final stage scores a bounded window against stored full-precision vectors.

SQL exposes scored table functions such as ann_search_scored, sparse_search_scored, and hybrid_search_scored. Trusted embedded SQL can also use the Boolean ANN predicate, while remote SQL requires the scored bounded path so deadlines, work ceilings, and candidate limits remain enforceable.

What the measurements do and do not say

The published BENCHMARKS.md records release-build measurements from one Linux x86-64 machine, including a 4.4828 microsecond accepted put without fsync, a 4.6721 millisecond durable commit, and 8.0387 milliseconds for Bitmap equality over one million rows. Those values describe the operational engine around the vector index; they are not HNSW latency numbers, and using them as though they measured ANN would be dishonest.

The ANN qualification suite currently asserts recall floors around 0.90 for Dense HNSW and 0.95 for BinarySign on its deterministic oracle data. A recall floor is a regression tripwire, not a universal quality forecast; corpus shape, embedding dimension, m, construction effort, search effort, filters, and the chosen distance representation all change the result.

The benchmark that matters is your own corpus with your own filters while writes are happening, because a graph that looks wonderful on an isolated random-vector test can still be the wrong database once memory, update rate, tenant isolation, and tail latency enter the room.

Where this shape stops working

If one ANN index already exceeds the memory and storage budget of a node, if dozens of remote writers need independent horizontal scaling, or if the vector workload needs managed multi-region failover, use a dedicated service built for that job. An embedded database removes an operational boundary; it does not repeal machine limits.

For desktop software, local RAG, agent memory, edge jobs, test harnesses, and single-node applications whose source rows and retrieval indexes belong together, the embedded boundary is often the simpler one, and the free MongrelDB Viewer gives that boundary a schema browser, SQL workbench, ANN maintenance surface, and MCP bridge without adding another server.

MongrelDB is an independent open-source database written in Rust. It is not MongoDB, not a MongoDB fork, and not a MongoDB-compatible replacement. The name describes the architecture: operational writes, columnar runs, specialized indexes, DataFusion SQL, and optional server deployment share one engine even though those ideas came from database families that are usually operated separately.

What MongrelDB is

The production center is an embedded or single-node operational database. An append-only WAL and group commit feed a Bε-tree memtable keyed by RowId and epoch, mutable state flushes into immutable .sr sorted runs with PAX-style columnar pages, and readers merge those layers under MVCC snapshots.

Six public secondary index families resolve through one RowId space:

• Roaring Bitmap for low-cardinality equality and indexed byte prefixes;
• PGM learned range for ordered numeric and time predicates;
• FM-index for exact substring containment;
• ANN with supported HNSW, DiskANN, IVF, Dense, BinarySign, and product-quantized combinations;
• Sparse inverted vectors for SPLADE-style weighted retrieval;
• MinHash LSH for approximate set similarity and deduplication candidates.

Primary-key lookup is implicit and currently uses an ordered-map implementation behind the public surface. The packed-memory-array tier is internal. Calling those two additional public indexes would make the marketing number larger and the documentation worse, so the accurate public count is six.

DataFusion 54 supplies SQL including joins, recursive CTEs, window functions, CREATE TABLE AS SELECT, materialized views, multi-statement execution, JSON functions, and scored retrieval table functions. Native conditions remain available when a typed application call expresses the query more directly than SQL.

The unusual part is mixed retrieval

A typical retrieval query is not just nearest neighbours. It asks for semantic candidates belonging to one tenant, within a date window, containing an exact identifier, then mixes dense and sparse relevance and reranks a bounded candidate set against full-precision vectors.

MongrelDB’s index families all return RowIds, so equality, range, substring, vector, sparse, and set-similarity paths can combine before the engine decodes the requested columns. Named retrievers fuse through reciprocal-rank fusion, optional exact-vector reranking preserves both approximate and final scores, and remote AI paths enforce candidate, work, deadline, and concurrency ceilings.

That is the engine’s strongest differentiator. An application can keep source rows, operational transactions, analytical projections, and model-derived representations in one local recovery model instead of synchronizing a primary store, vector service, and search index.

What the current measurements say

The published release-build fixture on an Intel Core Ultra 9 386H reports:

These are one-machine engineering results with commands and fixtures in BENCHMARKS.md. They are not cross-engine guarantees, and the durable number is milliseconds, not the single-digit microseconds an earlier version of this article incorrectly claimed.

A production decision runs the application’s own rows, projections, filters, durability policy, encryption mode, and concurrent workload on deployment hardware. Vendor benchmarks are useful for finding a path worth testing, not for avoiding the test.

Where SQLite still wins

SQLite has decades of deployment, an exceptionally stable format, tiny distribution friction, broad language support, mature administration tools, and a community large enough that most failure modes already have a search result. If the application needs reliable relational storage, ordinary indexes, and SQL without native vector, sparse, FM, or mixed analytical requirements, SQLite is the safer default.

MongrelDB also does not open SQLite files or preserve SQLite API compatibility. The SQLite encryption comparison is explicit about that boundary.

Where DuckDB still wins

DuckDB is built for local analytical SQL and has a mature ecosystem around Parquet, data frames, and scan-heavy workloads. If the workload is primarily ingest-then-analyse, with no need for an operational transaction engine and its specialized retrieval indexes, DuckDB is the more established analytical choice.

MongrelDB uses columnar sorted runs and DataFusion because operational applications still need scans and aggregation, not because it should replace an analytics engine at the workload DuckDB was designed to dominate. The embedded HTAP guide explains the mixed-workload boundary and the resource contention that comes with it.

Where a vector service still wins

A dedicated vector database is the right shape when the vector index must exceed one node, scale independently, serve many remote writers, or provide managed multi-region availability. MongrelDB’s embedded and single-node profiles remove a service boundary; they do not remove machine limits.

For desktop, edge, local RAG, agent memory, test harnesses, and services whose vectors belong transactionally with operational rows, an embedded vector database can be the simpler architecture.

Security and operations

Encrypted tables protect sorted-run pages, WAL frames, and persistent result-cache entries with AES-256-GCM. Passphrases derive keys through Argon2id and HKDF, raw high-entropy keys are supported, and HashiCorp Vault Transit can wrap the database root key. Searchable columns can derive equality and order-preserving range tokens, with the frequency and order leakage those techniques imply.

The optional daemon adds authenticated multi-process access, TLS, OIDC or SCRAM sign-in, a MySQL-compatible listener, replication, and cluster machinery. The public website recommends the embedded or single-node server profile first, because that path has the clearest qualification evidence; distributed capability exists, but deployment claims should follow tested operational evidence rather than architecture diagrams.

Who should evaluate it

MongrelDB is worth a fixture when an application needs several of these at once:

• durable operational writes and multi-table transactions;
• local columnar scans and DataFusion SQL;
• dense, sparse, substring, range, and equality retrieval in one row-identity model;
• encryption integrated with WAL and storage pages;
• embedded ownership with an optional daemon for multiple clients;
• native or HTTP access across the current 35-language client matrix.

It is not the conservative default for ordinary relational storage, the first choice for warehouse-only analytics, or the right service for a vector index that already needs a cluster. New database engines earn trust by narrowing claims, publishing the fixture, documenting unfinished surfaces, and being easy to remove after an evaluation; that is a less exciting pitch than “one database for everything,” and it is much closer to how production systems get chosen.