JSON Substitution: Do Not Reject Malformed Input

If you have ever run a PHP app that ingests data from users, from legacy imports, or from a SOAP endpoint somebody swore was “basically UTF-8,” you know the failure mode that matters: one byte sequence that is not valid UTF-8 somewhere deep in a ten-thousand-row payload, and the question is whether your database client refuses the entire request over that one byte or deals with it and moves on. The strict-correctness answer, the one json_encode gives you out of the box, is to refuse: the function returns false, json_last_error() reports JSON_ERROR_UTF8 (the error code arrived in 5.5, the throwing variant in 7.3), and whole batches of otherwise-good rows have died on that altar because a Word export pasted a smart quote through a Windows-1252 pipeline. The MongrelDB PHP client takes a position on this, and the position is worth explaining because it is two policies, not one: recover what is recoverable, reject what is not, and be honest about where the line sits.

The one function every request goes through

Every post() and put() in the client funnels its payload through a single private encodeJson(), and the flags are the whole story:

private function encodeJson(mixed $data): string
{
    try {
        return json_encode(
            $data,
            \JSON_THROW_ON_ERROR | \JSON_INVALID_UTF8_SUBSTITUTE,
        );
    } catch (\JsonException $e) {
        throw new QueryException(
            'Request payload cannot be JSON-encoded: ' . $e->getMessage()
            . '. (INF, NAN, and recursive structures have no JSON representation.)',
        );
    }
}

JSON_INVALID_UTF8_SUBSTITUTE is the flag PHP added in 7.2, and what it does is replace each invalid byte with U+FFFD, the Unicode replacement character, instead of failing the encode, so a truncated multi-byte sequence can leave two or three consecutive � characters behind, which is worth knowing if you ever diff a stored value against its input. That is the recoverable case, and the reasoning is proportionality: the surrounding data is still valid and still meaningful, the smart quote becomes a � in one field of one row, and refusing the whole request over it punishes ten thousand good rows for one bad byte. It is a trade, and the honest way to state it is that the substitution is silent at write time; nothing logs which rows changed, and the evidence only appears when you read the data back, but that is still the better failure, because anyone who lived through the mysql_real_escape_string era remembers the alternative, where the bad byte surfaced three layers away as a mangled string or a dropped connection and you spent an afternoon finding it, and a visible replacement character in the stored row beats an exception that blames the batch.

What substitution refuses to cover

The other policy is the one that keeps substitution honest, because there are values with no JSON representation at all, and for those the client throws a typed QueryException at the boundary rather than coercing something plausible-looking into the payload:

use Visorcraft\MongrelDB\Database;
use Visorcraft\MongrelDB\Exceptions\QueryException;

$db = new Database('http://127.0.0.1:8453');

try {
    $db->table('metrics')->insert(['value' => INF]);
} catch (QueryException $e) {
    // Request payload cannot be JSON-encoded: Inf and NaN cannot be JSON encoded.
    // (INF, NAN, and recursive structures have no JSON representation.)
}

INF, NAN, and recursive structures fall in this bucket, and the distinction from the UTF-8 case is that coercion here would corrupt data rather than merely blemish it; a NAN quietly turned into 0 or null is a wrong number that looks right, which is worse than an error, and a recursive array turned into anything at all is a lie about your object graph. The exception message carries the underlying JsonException text plus the reminder of what cannot be encoded, so the failure you catch names the actual cause instead of a generic “encoding failed.”

There is a second, quieter responsibility in the recursion case, and it is timing: PHP’s json_encode detects circular references itself and throws, so the client never has to walk your structure to find the cycle, which matters because a naive pre-check would be the thing that tears on a deeply nested payload in the first place. The error arrives from the encoder, gets wrapped once, and surfaces with your request context still intact.

Why this belongs in the client, not in your controllers

The usual arrangement is that every application reinvents this badly in a helper somewhere, half the call sites remember to use it and half do not, and the behavior drifts between the import job and the web tier until a production incident teaches you which half was which. Putting the policy at the client’s single encoding chokepoint means the rule is uniform across insert, update, sql, procedure calls, and everything else that crosses the wire, because there is exactly one place the JSON gets made. The modern equivalent is how you would not let every controller negotiate its own charset with the response object; the framework owns the boundary, and here the client owns the boundary, and the two policies, substitute the recoverable and reject the unrepresentable, are the entire contract. It is a small function, maybe fifteen lines, but it encodes a decision most PHP codebases never make explicitly, and making it once in the right place beats making it forty times inconsistently.