<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Aggregates on MongrelDB</title><link>https://www.mongreldb.com/articles/tags/aggregates/</link><description>Recent content in Aggregates on MongrelDB</description><image><title>MongrelDB</title><url>https://www.mongreldb.com/assets/og-mongreldb.png</url><link>https://www.mongreldb.com/assets/og-mongreldb.png</link></image><generator>Hugo</generator><language>en-US</language><lastBuildDate>Fri, 06 Nov 2026 09:00:00 -0600</lastBuildDate><atom:link href="https://www.mongreldb.com/articles/tags/aggregates/index.xml" rel="self" type="application/rss+xml"/><item><title>Extended SQL Function Helpers: JSON, Date/Time, Math, Aggregate</title><link>https://www.mongreldb.com/articles/2026/11/extended-sql-function-helpers-json-date-time-math-aggregate/</link><pubDate>Fri, 06 Nov 2026 09:00:00 -0600</pubDate><guid>https://www.mongreldb.com/articles/2026/11/extended-sql-function-helpers-json-date-time-math-aggregate/</guid><description>MongrelDB registers a full library of SQLite-shaped extended SQL functions inside the engine with honest volatility markers wired into the result cache, and Kit&amp;#39;s TypeScript helpers emit those calls as quoted SQL fragments.</description><content:encoded><![CDATA[<p>A query language is only as good as the functions you can reach without thinking, because nobody in the middle of an incident wants to stop and write a date parser in application code just to answer &ldquo;what happened last Tuesday.&rdquo; I learned the MySQL function list in the early 2000s the way everyone did, by typing <code>DATE_FORMAT</code> and <code>NOW()</code> and <code>MD5()</code> into a client until they were muscle memory, and when SQLite later grew the <code>json1</code> extension that same kind of catalog became the part of the database people actually memorized, far more than any wire protocol or storage format. So when we built the SQL frontend for MongrelDB on DataFusion, the function catalog was treated as part of the contract rather than an afterthought, because an engine that parses <code>SELECT</code> but shrugs at <code>json_extract</code> is really telling you to go write your own JSON path evaluator in every application that touches it, and that is the exact moment the &ldquo;the database speaks SQL&rdquo; claim stops meaning anything.</p>
<h2 id="the-catalog-lives-in-the-engine-not-in-the-sdk">The catalog lives in the engine, not in the SDK</h2>
<p>The whole library is registered in one pass over the DataFusion <code>SessionContext</code> from <code>crates/mongreldb-query/src/extended_sql_functions.rs</code>, a little over a hundred names, and the shapes are deliberately SQLite-shaped because that is the vocabulary twenty years of developers already carry around. The JSON family covers <code>json_extract</code>, <code>json_type</code>, <code>json_array_length</code>, <code>json_set</code>, <code>json_insert</code>, <code>json_replace</code>, <code>json_remove</code>, <code>json_quote</code>, and <code>json_valid</code>, with <code>json_each</code> and <code>json_tree</code> registered as table-valued functions for exploding arrays and walking whole documents, and every one of them has a <code>jsonb</code> twin that reads the binary JSONB form directly instead of re-parsing text on every row. The idioms transfer unchanged:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> json_extract(payload, <span style="color:#e6db74">&#39;$.city&#39;</span>) <span style="color:#66d9ef">AS</span> city,
</span></span><span style="display:flex;"><span>       json_type(payload, <span style="color:#e6db74">&#39;$.tags&#39;</span>)   <span style="color:#66d9ef">AS</span> tags_type
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> events
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">WHERE</span> json_valid(payload) <span style="color:#f92672">=</span> <span style="color:#ae81ff">1</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> e.id, tag.value
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> events e, json_each(e.payload, <span style="color:#e6db74">&#39;$.tags&#39;</span>) <span style="color:#66d9ef">AS</span> tag;
</span></span></code></pre></div><p>The date and time family, <code>date</code>, <code>time</code>, <code>datetime</code>, <code>julianday</code>, <code>unixepoch</code>, <code>strftime</code>, and <code>timediff</code>, parses <code>'now'</code>, RFC 3339 strings, <code>YYYY-MM-DD HH:MM:SS</code>, bare dates, bare times, and raw unix seconds, and the modifier grammar covers <code>'start of day'</code>, <code>'unixepoch'</code>, and signed offsets like <code>'-7 days'</code> or <code>'+2 hours'</code>, which is a smaller modifier list than SQLite&rsquo;s full set and we are honest about that, but it covers the offsets people actually write in dashboards and cron queries:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> datetime(<span style="color:#e6db74">&#39;now&#39;</span>, <span style="color:#e6db74">&#39;start of day&#39;</span>, <span style="color:#e6db74">&#39;-7 days&#39;</span>) <span style="color:#66d9ef">AS</span> window_start,
</span></span><span style="display:flex;"><span>       timediff(<span style="color:#e6db74">&#39;now&#39;</span>, created_at)                <span style="color:#66d9ef">AS</span> age_seconds
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> sessions;
</span></span></code></pre></div><p>Math gets the full trig and log run from <code>acos</code> through <code>tanh</code> with <code>degrees</code> and <code>radians</code> for the conversions, plus <code>ceil</code>, <code>floor</code>, <code>round</code>, <code>sign</code>, <code>pi</code>, and a <code>pow</code>/<code>power</code> alias pair, and the string family brings <code>instr</code>, <code>substr</code>, <code>hex</code>/<code>unhex</code>, <code>printf</code>/<code>format</code>, the trim variants, <code>unicode</code>, and even <code>soundex</code>, which is one of those functions you mock until the day a customer-support search needs it and then you are very glad it exists.</p>
<h2 id="aggregates-that-carry-state">Aggregates that carry state</h2>
<p>The aggregate side is the part I would point a skeptic at first, because <code>median</code>, <code>percentile</code>, <code>percentile_cont</code>, and <code>percentile_disc</code> are registered in <code>crates/mongreldb-query/src/percentile.rs</code> as real DataFusion aggregate UDFs with accumulator state, which means they participate in <code>GROUP BY</code> the way <code>count</code> and <code>sum</code> do instead of being a two-pass job where you export rows and compute percentiles in a notebook. The semantics follow Postgres, <code>percentile_cont</code> interpolates between neighbors and <code>percentile_disc</code> returns the first value at or past the cutoff, and <code>group_concat</code> and <code>string_agg</code> handle the string-rollup case that reporting queries always end up needing:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> endpoint,
</span></span><span style="display:flex;"><span>       <span style="color:#66d9ef">count</span>(<span style="color:#f92672">*</span>)                          <span style="color:#66d9ef">AS</span> calls,
</span></span><span style="display:flex;"><span>       percentile_cont(latency_ms, <span style="color:#ae81ff">0</span>.<span style="color:#ae81ff">95</span>) <span style="color:#66d9ef">AS</span> p95,
</span></span><span style="display:flex;"><span>       percentile_disc(latency_ms, <span style="color:#ae81ff">0</span>.<span style="color:#ae81ff">5</span>)  <span style="color:#66d9ef">AS</span> p50
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> request_log
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">GROUP</span> <span style="color:#66d9ef">BY</span> endpoint
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ORDER</span> <span style="color:#66d9ef">BY</span> p95 <span style="color:#66d9ef">DESC</span>;
</span></span></code></pre></div><h2 id="kits-helpers-are-quoting-not-magic">Kit&rsquo;s helpers are quoting, not magic</h2>
<p>On the TypeScript side, Kit does not try to build an expression DSL over these functions; each helper in <code>packages/kit/src/sql.ts</code> returns a <code>{ sql }</code> fragment, column arguments pass through identifier quoting, string and scalar arguments pass through <code>sqlLiteral</code> with proper <code>''</code> escaping, and the fragments compose into ordinary template literals, so the value being added is quoting discipline and a name you can autocomplete rather than a new abstraction to learn:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-ts" data-lang="ts"><span style="display:flex;"><span><span style="color:#66d9ef">import</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">groupConcat</span>,
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">jsonExtract</span>,
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">percentileCont</span>,
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">unixEpoch</span>,
</span></span><span style="display:flex;"><span>} <span style="color:#66d9ef">from</span> <span style="color:#e6db74">&#39;@visorcraft/mongreldb-kit&#39;</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">const</span> <span style="color:#a6e22e">rows</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">await</span> <span style="color:#a6e22e">db</span>.<span style="color:#a6e22e">sqlRows</span>(<span style="color:#e6db74">`
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">  SELECT </span><span style="color:#e6db74">${</span><span style="color:#a6e22e">jsonExtract</span>(<span style="color:#a6e22e">events</span>.<span style="color:#a6e22e">payload</span>, <span style="color:#e6db74">&#39;$.city&#39;</span>).<span style="color:#a6e22e">sql</span><span style="color:#e6db74">}</span><span style="color:#e6db74">      AS city,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">         </span><span style="color:#e6db74">${</span><span style="color:#a6e22e">percentileCont</span>(<span style="color:#a6e22e">events</span>.<span style="color:#a6e22e">latency_ms</span>, <span style="color:#ae81ff">0.95</span>).<span style="color:#a6e22e">sql</span><span style="color:#e6db74">}</span><span style="color:#e6db74">    AS p95_ms,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">         </span><span style="color:#e6db74">${</span><span style="color:#a6e22e">groupConcat</span>(<span style="color:#a6e22e">events</span>.<span style="color:#a6e22e">tag</span>, <span style="color:#e6db74">&#39;|&#39;</span>).<span style="color:#a6e22e">sql</span><span style="color:#e6db74">}</span><span style="color:#e6db74">               AS tags,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">         </span><span style="color:#e6db74">${</span><span style="color:#a6e22e">unixEpoch</span>(<span style="color:#e6db74">&#39;now&#39;</span>, <span style="color:#e6db74">&#39;start of day&#39;</span>).<span style="color:#a6e22e">sql</span><span style="color:#e6db74">}</span><span style="color:#e6db74">           AS today_utc
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">  FROM events
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">  GROUP BY city
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">`</span>);
</span></span></code></pre></div><p>The one honest seam is <code>mathFn(name, ...args)</code>, which takes the function name as raw SQL because the alternative is maintaining an enum of every math function forever, so the rule is trusted names only, and if you are building a query from user input you pick from a fixed list instead of interpolating the name.</p>
<h2 id="the-volatility-tradeoff">The volatility tradeoff</h2>
<p>The cost side of shipping clock and randomness functions is that their results change between identical executions, and a result cache that served a cached <code>datetime('now')</code> would be a wrong answer delivered quickly, so every function in the registry carries a volatility marker and the executor scans the query text for calls into the volatile set, <code>date</code>, <code>time</code>, <code>datetime</code>, <code>julianday</code>, <code>unixepoch</code>, <code>strftime</code>, <code>timediff</code>, <code>random</code>, and <code>randomblob</code>, and simply refuses to serve those queries from the result cache. That is the tradeoff made explicit rather than documented in a footnote: you get the convenience of clock functions anywhere in your SQL, you lose the cache hit on exactly the queries that use them, and the engine makes that determination from the SQL text itself instead of trusting every caller to remember. The immutable majority, <code>json_extract</code>, <code>sqrt</code>, <code>percentile_cont</code>, and friends, stays fully cacheable, so the common analytical query pays nothing for the policy.</p>
<p>The old world had this right in a way we almost forgot: the MySQL and SQLite function lists followed you for a decade because they were stable, complete, and documented once, and the modern equivalent for an engine built on DataFusion is not to invent a fresh catalog with fresh names but to register the vocabulary people already have, mark each function with an honest volatility, and let every SDK inherit the whole library for free because there was never anything to reimplement, rename, or re-document on the client side.</p>
]]></content:encoded></item></channel></rss>