Recursive CTEs, evaluated by MongrelDB instead of DataFusion

The query layer is DataFusion over Arrow record batches, and DataFusion v54’s recursive CTE support is incomplete in a specific, dangerous way: it mis-handles column aliases on the base case, which means the query runs and the answer is wrong, and a wrong answer that looks right is the worst thing a database can hand you. Rather than ship that, the engine intercepts WITH RECURSIVE at the command dispatcher before DataFusion ever sees the statement and evaluates it with the standard semi-naive algorithm, which is: run the base case, register the result as a temp table named after the CTE, then iterate, evaluating the recursive arm against only the rows the previous round produced, the delta, rather than the whole accumulated table, and stop when a round comes back empty:

WITH RECURSIVE subordinates(id, name, depth) AS (
  SELECT id, name, 0 FROM employees WHERE manager_id IS NULL
  UNION ALL
  SELECT e.id, e.name, s.depth + 1
  FROM employees e JOIN subordinates s ON e.manager_id = s.id
)
SELECT id, name, depth FROM subordinates ORDER BY depth, id;

Delta-only evaluation is the detail that makes the feature honest, because evaluating each round against just the new rows instead of the full working set is what keeps the per-round cost proportional to progress rather than to history, and it is what the UNION versus UNION ALL distinction hangs on: with plain UNION the engine dedupes the accumulated set with a SELECT DISTINCT pass each round and stops as soon as a round adds nothing genuinely new, which means cyclic graphs terminate the way the standard says they should, while UNION ALL keeps bag semantics and simply accumulates each delta, so a truly cyclic walk under UNION ALL is bounded by a hard ceiling of ten thousand iterations with a cancellation checkpoint on every round, and then the outer query runs against whatever accumulated. The remaining limits are printed in the parser rather than discovered in production: exactly one statement per call, exactly one CTE in the WITH clause, and anything more exotic gets a clear error instead of a plausible-looking wrong result.

Window functions come free, and that is the point

ROW_NUMBER(), SUM() OVER (...), the ranking family, all of it rides on DataFusion’s window support directly with no interception, because DataFusion’s window implementation is good and there was no reason to write our own, and this part of the “how much real SQL do you get” answer should feel almost disappointing, since the engineering decision was to not engineer. The query that used to be a self-join with a correlated subquery in the MySQL 5.6 years, top three per region with a running total, is one scan now:

SELECT region, product, revenue,
       ROW_NUMBER() OVER (PARTITION BY region ORDER BY revenue DESC) AS rank_in_region,
       SUM(revenue) OVER (PARTITION BY region) AS region_total
FROM sales
ORDER BY region, rank_in_region;

The window frame evaluates over Arrow batches the engine already has in flight, and because a window function is just another expression to the planner it composes with everything else in this post, so you can wrap it in a CTE, feed it into CREATE TABLE AS SELECT, or freeze it into a materialized view, and none of those combinations need special handling anywhere in the SDK.

Materialized views that publish or do not

CREATE MATERIALIZED VIEW stores the defining query in the catalog alongside a last_refresh_epoch, and REFRESH MATERIALIZED VIEW name re-runs it, and the refresh path is where the actual decisions live, because there are two of them and the engine picks between them by inspecting your definition. If the definition is a plain single-table GROUP BY with COUNT and SUM outputs and none of the clauses we deliberately do not maintain incrementally, no joins, no filter, no HAVING, no DISTINCT, no LIMIT, no nested CTE, the engine maintains the view incrementally against a checkpointed position in the commit log, falling back to a rebuild of the incremental state when the checkpoint cannot carry forward. Everything else takes the full rebuild path: run the defining query, stage the results into a shadow building table, and publish the swap atomically only when the whole refresh succeeded, because a failed refresh discards the shadow and leaves the old data exactly where it was, and that behavior is a test in the repo, not an aspiration in a README.

The guardrails are equally unglamorous and equally deliberate: REFRESH is rejected inside an explicit transaction, the name must be a single unqualified identifier, a definition that has drifted to return a different number of columns than the materialized schema fails the refresh instead of silently truncating, and per-batch byte limits keep a pathological rebuild from eating the process. The honest tradeoff is that refresh is on demand rather than continuous, so readers see the last published epoch until the new build finishes, and freshness is a cron line you already know how to write, which is the Postgres operating model transplanted into an engine that does not need a Postgres.

CTAS and multi-statement batches round it out

CREATE TABLE AS SELECT runs the inner query, infers the schema from the result columns, refuses to guess when the inferred primary key comes back NULL, and stages rows under the same byte limits and build-then-publish machinery the materialized view refresh uses, because building a table from a query and rebuilding a table from a query are the same problem and should have one implementation:

await db.sql(`
  CREATE TABLE sales_summary AS
  SELECT region, COUNT(*) AS n, SUM(revenue) AS total
  FROM sales
  GROUP BY region;
`);

Multi-statement execution works the way you would want for fixture loads and one-shot setup scripts, with semicolon-separated statements in a single call, and the only subtlety is that the splitter runs before command dispatch and stays quiet when the first semicolon lives inside a BEGIN ... END block, because CREATE TRIGGER bodies contain semicolons of their own and a splitter that cannot tell a statement boundary from a trigger body is a bug waiting for a migration to find it:

const rows = await db.sqlRows(`
  INSERT INTO config VALUES ('seed_version', '3');
  UPDATE stats SET dirty = 1;
  SELECT COUNT(*) AS n FROM stats;
`);

What this buys and what it costs

The MySQL-era answer to reporting questions was a second database, first a read replica with long queries pointed at it, later a warehouse with a pipeline, and both answers existed because the application database could not evaluate the query you actually needed to ask. The claim here is narrower and more useful than “we support SQL”: recursive CTEs are evaluated by an algorithm whose correctness you can audit in an afternoon, window functions are whatever DataFusion’s sort-based evaluation costs and not a penny of custom code, materialized views refresh atomically or not at all with incremental maintenance for the count-and-sum cases that make up most real dashboards, and the same statement parses identically from TypeScript, Rust, Python, or the PHP client because there is one parser and one planner behind all four, which is the entire thesis of Kit compressed into a single sentence: the SDKs differ in syntax, never in semantics. The costs are printed rather than hidden, incremental maintenance covers COUNT and SUM on a single table today, recursive evaluation is delta-based rather than fully memoized, and refresh is pull-based rather than streaming, and we would rather you read those limits in an error message or a blog post than find them in a dashboard that went quietly stale.