Why middleware auth is the wrong layer for a database file

The classic shape, and I wrote this shape myself in the mysql_* era more times than I want to admit, is that the application holds one shared database password, every request funnels through application code that checks the user’s session before it touches the connection, and the database itself trusts anyone who shows up with the one password it knows. That model survives contact with production exactly as long as every path to the data goes through the application, which is true right up until someone opens a REPL against the production file, or a second service gets added with its own connection logic, or a backup script starts reading raw pages, and at that point your permission model is a comment in a wiki page rather than a property of the system.

Middleware auth also has a layering problem that gets worse the more surfaces a database exposes, because MongrelDB is not just a daemon; it is an embedded library, a CLI, a set of SDKs in a dozen languages, and an HTTP server, all over the same on-disk format, and if auth lives in the HTTP layer then the embedded open path has no auth at all unless you build a second auth system for it, and now you have two systems to keep honest instead of one.

Users and roles live in the catalog

MongrelDB stores users, roles, and grants inside the database catalog itself, the same _meta blob that holds table schemas, procedures, and triggers, so there is no separate auth file to back up, no external user database to keep in sync, and no drift between “who the app thinks exists” and “who the engine thinks exists.” Passwords are hashed with Argon2id at the OWASP-recommended parameters (19 MiB memory, time cost 2, parallelism 1), which is the same construction the engine uses to derive the key-encryption key for encrypted databases, and plaintext passwords are never stored or logged anywhere in the system.

The permission model is deliberately small: All, Admin, Ddl, and per-table Select, Insert, Update, and Delete, matched literally with no wildcard syntax, so select:* grants access to a table actually named * and nothing else. Two details are worth calling out because they are the ones people get wrong when they skim: a principal with the is_admin flag short-circuits every check, which is how the bootstrap admin works, and Permission::All explicitly does not imply Admin, so granting someone everything on every table still does not let them create users or hand out grants, which is the separation you want between “can touch all the data” and “can mint new identities.”

The SQL frontend speaks the standard vocabulary, so nothing here requires learning a bespoke control language:

CREATE USER alice WITH PASSWORD 's3cret-pw';

CREATE ROLE analyst;
GRANT SELECT ON orders TO analyst;
GRANT INSERT ON orders TO analyst;
GRANT analyst TO alice;

SHOW USERS;
SHOW ROLES;

What require_auth actually enforces

By default the catalog auth is advisory, meaning the engine stores users and permissions and your application can consult them with check_permission, but reads and writes are not gated, which keeps MongrelDB drop-in compatible with the SQLite-style “open the file and go” workflow. The interesting mode is opt-in per database: set require_auth = true in the catalog and the storage layer itself starts enforcing, which changes the failure semantics in a way middleware never can.

use mongreldb_core::Database;

// Create a credentialed database with a bootstrap admin.
let db = Database::create_with_credentials("./secure_db", "admin", "s3cret-pw")?;

// Later: plain open fails with AuthRequired.
// Database::open("./secure_db")?  -> Err(AuthRequired)

let db = Database::open_with_credentials("./secure_db", "alice", "alice-pw")?;
// Every Table, Transaction, and SQL operation on this handle is now
// checked against alice's permissions. A missing grant returns
// PermissionDenied from the engine, not from a wrapper.

With enforcement on, a Table::put requires Insert on that table, an SQL UPDATE requires both Update and Select, DDL and compaction require Ddl, user and role management requires Admin, and the daemon’s HTTP auth middleware stops being the only line of defense, because the storage layer re-checks the resolved principal on every operation even after the middleware has already accepted the request. That is the defense-in-depth shape you actually want: the HTTP layer asserts who you are, and the engine enforces what that identity may do, so a bug in either layer alone does not silently open the whole file.

The error surface maps cleanly onto HTTP when the daemon is involved, which keeps client code boring: AuthRequired and InvalidCredentials come back as 401, PermissionDenied comes back as 403, and AuthNotRequired is a 400 for the case where you passed credentials to a database that does not want them.

What it costs

The honest cost numbers are small and they sit in the right places. One Argon2id verification is tuned to land around 50 milliseconds on current server hardware, and you pay it once per open or once per authenticated HTTP request, after which the engine caches the resolved principal and every per-operation check is a cheap permission comparison against an Arc-cloned AuthState handle rather than another hash. The cached principal is worth one operational caveat: a revoked grant does not bite on an already-open handle until the next authentication, which is why the engine exposes refresh_principal() for the cases where you cannot wait for a reconnect, and why the daemon re-resolves the principal per request rather than per connection. The deliberate expense of the hash is the feature, not the overhead, because it caps online password guessing at roughly 20 attempts per second per core, though the v1 implementation does not add lockouts or throttling on top of that, so weak passwords remain weak passwords and no storage layer can fix that for you.

The composability story is the part I would have killed for in 2009: credential enforcement and page-level encryption are orthogonal, so create_encrypted_with_credentials gives you a database where the bytes on disk are AES-256-GCM ciphertext and the logical operations still require a catalog identity, and the two systems share nothing beyond the Argon2id primitive and the password that feeds it, which means a mistake in one does not weaken the other.

What it does not stop

This is the section most vendor posts skip, and skipping it is how you end up with a CVE and a sheepish blog post. Credential enforcement is a logical access control, not a cryptographic one, so an attacker with raw disk access who parses the file format directly can read the data without ever calling the API, and an attacker who can write to the _meta directory can flip require_auth off or rewrite the catalog, which is why the docs tell you plainly that filesystem permissions on the database directory are the real boundary and why the recovery path for lost credentials is “edit the catalog offline,” a path that grants no power an attacker with disk access did not already have. If your threat model includes the disk getting pulled, you want the encryption layer too, and if your threat model includes the machine itself, you want full-disk encryption or hardware keys underneath all of it, because a database file, no matter how carefully it hashes its passwords, cannot defend the host it runs on.

The modern-equivalent close here is simple: we spent twenty years putting auth in front of databases because databases were big shared servers with one front door, and the embedded engine breaks that assumption by having as many doors as there are processes that can open the file, so the auth has to move to the one place every door leads through, which is the storage layer itself.

The shape of the problem

SQLite SEE works by presenting the database file as encrypted pages on disk and decrypting them into the page cache on read, which means the WAL, the shared memory region, and every tempfile that hits the filesystem are all covered by the same symmetric key. The encryption boundary is the entire file-level storage surface, and the key is held in memory for the lifetime of the process. That is the right model. Any encryption that does not cover the WAL is not encryption at rest, it is encryption at rest for the parts of the database that are not being actively written, and anyone who has watched a production incident knows that the interesting data is usually in the WAL during the incident.

The hard part is not the encryption itself; AES-256 in an appropriate mode is solved, and you reach for a library rather than rolling your own. The hard part is the key lifecycle, the page-level IO pattern that a database engine uses, and the throughput cost when every read and write hits the crypto layer before anything else.

What we built instead

MongrelDB uses a two-layer encryption surface: the page layer and the WAL layer share a derived key that is derived from a master key via HKDF-SHA256 with a per-table salt, and the master key itself is injected at startup from an environment variable or a plugin hook rather than stored on disk anywhere. The page encryption operates on 4KB page boundaries with AES-256-GCM, which gives us authenticated encryption so that a corrupted or tampered page on disk is detectable rather than silently misread, and page-layer key rotation is handled by re-encrypting pages in-place on a background thread during a controlled maintenance window, with the next-write rewrite path covering any pages that were not caught by the re-encryption pass. The WAL gets its own per-segment key derivation so that a WAL segment written at time T is not decryptable with a key derived after time T if the master key has been rotated in the interim.

The reason for the plugin hook rather than a fixed key storage mechanism is that different deployment shapes have different constraints around secret distribution, and the plugin hook keeps the engine honest about the abstraction boundary rather than baking in assumptions about where keys come from.

What it costs

The measured number is between 8 and 15 percent throughput reduction on write-heavy workloads at the storage layer, measured on the same hardware and same fsync profile as the unencrypted baseline. Read-heavy workloads see less, closer to 3 to 7 percent, because the page cache absorbs a portion of the read path and encrypted pages in cache cost nothing extra to serve. The GCM authentication tag adds 16 bytes per page, which increases WAL segment size by a small percentage on write-heavy workloads, and the per-segment WAL key derivation adds a small constant cost to each WAL segment open.

The number that matters more than the percentage is the shape of the cost curve. Encryption at rest is a constant factor on every IO operation, which means the overhead is proportionally largest on small transactions with many fsync calls and proportionally smallest on large sequential writes where the crypto cost is amortized over many rows per fsync. If your workload is dominated by bulk import, the measured overhead will be lower than if your workload is dominated by single-row commits with synchronous durability requirements.

The comparison that gets asked for

SQLite SEE uses a similar page-level encryption model with AES-128-CBC by default, and the performance profile is close enough that the real differentiator is not the crypto algorithm but the authentication property and the key management surface. CBC mode is unauthenticated, which means a bit-flip attack on an encrypted page produces modified plaintext with no checksum to catch it; GCM is authenticated, which means MongrelDB encrypted pages detect tampering rather than returning silently corrupted data, and that property is the engineering argument worth making explicitly rather than leaving it as an implicit difference. We did not build our own crypto to beat SEE on throughput, we built it because the closed-source distribution model does not fit a database that ships as a single open-source binary with no vendor relationship, and if your threat model includes the database vendor as a potential adversary, you should be using hardware-enforced encryption anyway, but if your threat model is “the disk gets pulled or the machine gets stolen,” a software encryption layer at the page level covers the practical attack surface for most threat models short of nation-state adversaries with physical disk access and a budget for hardware forensics.

The tradeoff is what it always is: you are trading a known-cost constant factor on every IO operation for protection against a class of incidents that are rare, expensive when they happen, and impossible to retrofit after the fact. Whether that tradeoff is worth it for your workload is not a question we can answer for you, but the numbers above are the inputs to that calculation.