Why middleware idempotency is not enough

The middleware version of this pattern is well understood, and it is also where the pattern quietly breaks. A reverse proxy or an application-level interceptor can dedupe retries that pass through it, but it cannot dedupe a retry that arrives after the process crashed, because its dedupe table is usually in-memory or in a cache with weaker durability than the database itself, and a cache entry that vanishes on restart is a promise that expires at exactly the wrong moment. The worst case is the gap between commit and response, where the database has durably applied the write and the middleware never learned that fact; the client retries, the middleware has no record, the database executes again, and you are back to two rows. You can paper over this with a dedupe table inside the same database, and people do, but then you are hand-rolling transactional semantics around a unique constraint and hoping you got the race conditions right, which is precisely the kind of work a storage engine should be doing for you.

What MongrelDB does instead

The core transaction API takes an optional idempotency request at commit time, carrying three things: the key itself, an owner string so two tenants can never collide on the same key, and a fingerprint of the request so the engine can tell a genuine retry apart from a key being reused for different work. When the transaction commits, the engine writes the idempotency record into a durable ledger that is anchored to the commit log, which means the key-to-receipt binding survives a restart under the same authority as the data pages, and an unkeyed commit never creates the ledger file at all, so you pay nothing for the feature when you do not use it.

The behavior on retry is the interesting part. The idempotency check runs inside the commit path before the transaction’s writes are ever proposed to the log, so if a commit arrives with a key and fingerprint that already have a completed record, the engine does not re-apply the staged writes or append anything new; it replays the original commit receipt, the client sees the same transaction identity it would have seen the first time, and the row count stays exactly where the first commit left it. If a commit arrives with a key that exists but a different fingerprint, the engine aborts with a conflict error, because reusing a key for different work is a bug in the caller and the correct response is to refuse, not to guess. If the key has a time-to-live and the record has expired, the record is swept and the key becomes available again, which keeps the ledger bounded for workloads like checkout flows where the retry window is minutes, not days.

In the Rust core that looks like this, lifted from the engine’s own test suite:

let mut first = db.begin();
first.put("t", vec![(1, Value::Int64(1))])?;
let epoch = first.commit_idempotent(IdempotencyRequest {
    key: "checkout-42".into(),
    owner: "store-api".into(),
    fingerprint: request_fingerprint,
    ttl: None,
})?;

// Network drops, client retries the same logical request.
let mut retry = db.begin();
retry.put("t", vec![(1, Value::Int64(2))])?;
let retry_epoch = retry.commit_idempotent(IdempotencyRequest {
    key: "checkout-42".into(),
    owner: "store-api".into(),
    fingerprint: request_fingerprint,
    ttl: None,
})?;

assert_eq!(retry_epoch, epoch);      // same receipt, no re-execution
assert_eq!(row_count(&db, "t"), 1);  // one row, not two

The daemon layer builds the same guarantee into the HTTP surface, because a write that goes through mongreldb-server deserves the same crash safety as a write through the embedded API. Buffered SQL writes accept an Idempotency-Key header or an idempotency_key field in the JSON body, the daemon durably publishes an intent before it executes anything, and after a known terminal outcome it publishes a receipt, with a file fsync and atomic rename on Unix and write-through replacement on Windows. The awkward truth of that design is that the commit and the receipt cannot be one atomic operation across a filesystem and a database, so if the process dies in the narrow gap between them, the daemon refuses to guess; a retry with the same key gets a non-retryable QUERY_OUTCOME_UNKNOWN, the intent stays on disk as a tombstone, and an operator verifies the outcome before recovery, because auto-expiring that tombstone into a fresh execution is how you charge a credit card twice. Receipt and intent files hold HMAC-authenticated hashes, never the raw key or the SQL, and on encrypted databases the HMAC key is derived from the database’s own key-encryption key, which means the idempotency store is authenticated by the same root of trust as the data itself and does not become a side channel.

From the client side

None of this matters if using it is painful, so the PHP client threads the key straight through the transaction surface, and the whole retry-safety story collapses into one extra argument at commit:

$txn = $db->beginTransaction();
$txn->put('orders', ['id' => 1042, 'total' => 9900, 'status' => 'paid']);
$txn->put('inventory', ['sku' => 'WIDGET-1', 'on_hand' => 41]);

// Safe to call again on timeout or connection drop:
$results = $txn->commit(idempotencyKey: 'order-1042-payment');

The Transaction::commit() signature takes the key as an optional parameter and drops it into the idempotency_key field of the batch payload, so the retry path in your application is just “catch the connection exception and call the same code again,” and the dedupe work happens where the fsyncs happen. The batch itself is atomic, so the order row and the inventory decrement commit together or not at all, and the idempotency record commits with them, which closes the gap that middleware cannot close.

What it costs

The honest tradeoff is capacity and bookkeeping. The embedded engine’s ledger is a single checksum-framed file, optionally sealed with the metadata key on encrypted databases, and it is bounded by two hard limits: at most 65,536 records, with the oldest committed records evicted first and in-flight reservations never evicted, and a 64 MiB cap on the file itself, beyond which the engine fails closed rather than risk a corrupted or replayed record. A key with no explicit TTL is kept until that bounded-size eviction takes it, so an embedded application that keys every write will eventually cycle the oldest records out, while a keyed write with a TTL sweeps on schedule. The daemon layer adds its own bounds on top, because one server may speak for many databases and many tenants: MONGRELDB_SQL_IDEMPOTENCY_MAX_ENTRIES (4,096 entries by default) caps the store, completed receipts expire after MONGRELDB_SQL_IDEMPOTENCY_TTL_SECS (24 hours by default, a safety bound rather than the expected retry window, which for a checkout flow is minutes), and each owner is capped at a quarter of the total capacity so one noisy tenant cannot starve the rest; when the store is full the daemon fails closed with a capacity error rather than evicting a receipt someone might still retry against. Crash-left intents consume capacity until an operator resolves them, which is deliberate, because an unresolved intent is an unresolved question about your data and the correct behavior is to keep asking it. There is also a small amount of extra fsync work per keyed write for the ledger mutation, which is real but bounded, and it is the price of making the retry guarantee survive power loss instead of merely surviving a TCP reset.

The modern equivalent of an old habit

In the mysql_* era the standard advice was “make your writes idempotent by design,” which in practice meant a unique key on a natural column and an INSERT ... ON DUPLICATE KEY UPDATE, and it worked, but only for the shapes of write that fit a single-row upsert, and only if you remembered to design the table that way up front. Storage-layer idempotency keys are the same instinct generalized: instead of contorting your schema so duplicates are harmless, you name the logical request once and let the engine recognize the retry, whatever shape the write takes, whether it is one row or a forty-operation batch. The guarantee moved down the stack, which is where guarantees like this have always belonged.