Calling this general full-text search is convenient but imprecise. The FM-index finds containment candidates for a literal pattern. MongrelDB also has a separate FTS document surface and ranking function, while sparse retrieval handles learned weighted terms. Those paths can work together, but an FM-index does not turn a substring into a tokenized relevance model by itself.

Why ordinary indexes miss %needle%

A sorted B-tree can seek a known prefix because values beginning with user: occupy one ordered interval. It cannot seek an arbitrary middle fragment without another representation, because strings containing needle may appear anywhere in lexical order.

A full scan solves the problem by checking every value. That is correct and often good enough for a small table, but cost grows with the bytes examined. The FM-index stores a transformed view of the corpus that supports backward search over the pattern, narrowing the suffix interval one symbol at a time through rank operations.

The search work depends primarily on pattern length plus the cost of locating and materializing matches. Output still matters: a two-byte pattern matching half the corpus cannot return half the corpus for free.

What the Burrows-Wheeler transform contributes

The Burrows-Wheeler transform rearranges text so repeated contexts cluster together. A wavelet tree supplies compact rank queries over that transformed sequence, and the FM-index combines those pieces with sampled location metadata so a matching interval can resolve back to candidate rows.

That structure is valuable when the application needs literal containment over stored text and does not want a separate search service. It is not automatically the best answer for hot write-heavy token search, fuzzy matching, typo tolerance, language stemming, or relevance-ranked document retrieval; those are different query models with different indexes.

How it appears in MongrelDB

The schema declares an FM secondary index on a bytes/text column:

IndexDef {
    name: "body_fm".into(),
    column_id: 4,
    kind: IndexKind::FmIndex,
}

A native condition requests containment:

Condition::FmContains {
    column_id: 4,
    pattern: b"error:42".to_vec(),
}

SQL can push down a recognized containment shape:

SELECT id, body
FROM logs
WHERE body LIKE '%error:42%';

The FM path produces candidate RowIds. DataFusion rechecks SQL pattern semantics where the index result is a superset, which keeps pushdown an optimization rather than a change in query truth. Native FmContains expresses the exact literal containment condition directly.

For anchored prefixes such as LIKE 'user:%', a Bitmap index on a bytes column can enumerate distinct keys and union matching prefixes exactly, which is usually a tighter path than asking the FM-index to solve a query whose anchor already gives the ordered key domain enough information.

Why the shared RowId matters

The substring result can intersect with a tenant Bitmap, a learned time range, an ANN candidate set, or any other hard condition before rows and requested columns are decoded. This is the useful part of keeping text search in the operational engine: the text match does not become a detached document identifier that the application must join back to a row stored somewhere else.

A hybrid query can ask for semantic neighbours whose body contains an exact code and whose timestamp falls inside an incident window. HNSW, FM, and PGM each do the work they were built for, then the engine combines their RowIds under one snapshot.

Index maintenance is derived state

The authoritative data is the committed row version. Secondary index generations can be checkpointed and rebuilt from runs plus mutable state. Online create, replace, and drop operations build or publish generations without rewriting the table, with a short barrier at publication.

That distinction matters for crash reasoning. The WAL protects committed changes to source data; the FM structure is an acceleration path that can be reconstructed, not a second authoritative database whose divergence the application must reconcile.

Maintenance cost still exists. Building a transformed corpus is more work than appending one posting to a small in-memory map, and frequent updates to large indexed text deserve a representative benchmark. MongrelDB exposes REINDEX and rebuild paths because no immutable text structure remains cheap under every update pattern.

When to choose another search path

Use a tokenized inverted index when the product needs word analysis, stemming, fuzzy terms, facets, mature BM25 tuning, or a search-only workload large enough to justify a dedicated engine. Use sparse retrieval when learned lexical expansion and weighted terms are the signal. Use ANN when semantic similarity matters. Use a Bitmap prefix path when the query is anchored and the distinct-key set is manageable.

Use the FM-index when exact substring containment is the requirement, the data belongs transactionally with the rest of the row, and combining that containment with ordinary database filters is worth more than operating a separate text-search service.

The current index behavior is documented in docs/06-indexes.md, and the broader embedded vector database guide shows how the text path combines with dense and sparse retrieval.