What broke when I stopped trusting my own validator

I build InvoiceHub — an API that validates, generates, and converts EU e-invoices: UBL 2.1, UN/CEFACT CII, Germany's XRechnung, the Factur-X hybrid PDF format. Every document it hands back has already been checked against the official CEN EN 16931 Schematron. That's the entire pitch, really: you get pre-validated output, not “probably fine” output.

Here's the part of that pitch I hadn't sat with properly: the code that generates a document and the code that validates it live in the same package, written by the same person, on the same afternoons. If I get one assumption wrong about, say, how XRechnung wants its settlement block structured, there's a decent chance I get it wrong the same way in both places. The validator doesn't catch the generator's mistake — it shares it. And it'll keep passing every document I throw at it, because I'm not in the business of generating documents specifically designed to break my own rules.

So over the last couple of days I built a validation harness whose entire job is to stop me from grading my own homework: run real output through tools I didn't write, and see whether they agree with me. A few things came out of that which are more interesting than “everything passed eventually,” so here's the honest version, bugs included.

First, a number I'd had wrong for a while

Before reaching for any external tool, I wanted to answer something simpler: does the Schematron I've got compiled actually contain every rule from the official CEN artifact, or some subset that happens to look complete? I already had a regression test for this — a hardcoded count of distinct rule IDs baked into the compiled ruleset, 955, meant to catch the day someone (me) accidentally ships a truncated build.

Turns out the regex behind that count was itself incomplete. It matched BR-* and UBL-CR-*/UBL-SR-* rule identifiers but missed UBL-DT-* datatype rules entirely — and the official conformance corpus tests those directly (UBL-DT-01.xml, UBL-DT-06.xml, UBL-DT-07.xml are real files in it). Widening the pattern brought the true count to 979. The old baseline wasn't catching a regression for two years; it was quietly undercounting by 24 rules the whole time. Not dramatic, but exactly the kind of thing you only find by going and building the thing that's supposed to find it.

The bug I found by staring at a suspiciously round zero

Building the coverage report itself hit a stranger problem first. Saxon-JS — the engine that actually runs the Schematron transforms — mutates its own compiled stylesheet object the first time you use it, attaching live parentNode references for its internal tree. That's fine at runtime. It is not fine if, later in the same process, you call JSON.stringify() on that same object to count rule IDs in it, because it's now circular and JSON.stringify just throws.

My rule-counting function had always called JSON.stringify on demand and never once broken — because I'd never called it after running an actual transform in the same process, until I put a coverage-report test in the same file as 1,161 corpus tests that exercise the engine first. Fix was boring once I found it: snapshot the JSON text once at module load, before Saxon-JS has touched anything, and count against that frozen string instead of the live object.

Right after that fix, a second bug showed up immediately, and it's the one I'm least proud of. The official CEN test corpus isn't a folder of standalone invoices — each file is a wrapper containing several <test> blocks, each one a deliberately partial invoice fragment plus an assertion about which single rule it's meant to trip. To report which rules the corpus actually exercises, I parse a <scope> tag out of each file. My header-boundary search was xml.indexOf('<test') — which is also, of course, a prefix of <testSet, the very first tag in the file. So it found the wrong occurrence every single time, truncated the search before it ever reached <scope>, and returned nothing. Silently. On all 279 files.

Nothing crashed. A test just printed rules_exercised_by_corpus=0 for a corpus I'd already run 1,128 real assertions against successfully — and the only reason I caught it is that zero looked too clean to be true.

What the “official” corpus does and doesn't actually cover

With both of those fixed, the real numbers: my UBL Schematron has 979 distinct rules, of which CEN's own corpus exercises 201. My CII Schematron has 806 rules, of which the same corpus exercises two. That's not my engine's fault — CEN's corpus is overwhelmingly built out of UBL fixtures, and it's an honest gap in what an “official” test suite actually proves, not something I get to paper over just because it's not mine. It's also the reason the next problem didn't show up in this corpus at all — nothing in it was positioned to find it.

What a real oracle catches that a corpus can't

XRechnung is Germany's public-sector e-invoicing profile, and the receivers on the other end run inbound documents through KoSIT's validator — the actual jar KoSIT publishes, not my guess at what their rules say. So I ran generated XRechnung output, both UBL and CII syntax, through that jar in CI.

UBL came back clean. CII came back rejected outright — not a business-rule complaint, a flat XML Schema violation (cvc-complex-type.2.4.a: Invalid content was found starting with element PostcodeCode). My CII serializer was writing address fields in the wrong order: postcode after the street lines, when the UN/CEFACT schema wants it before. A second pass turned up the same species of bug in the settlement block, with payment-means emitted after payment-terms instead of before.

My own Schematron-based validator was never going to catch either of those, structurally — Schematron runs on top of a document that's already assumed to be schema-valid, so it was faithfully checking business rules against XML that wasn't even well-formed by the spec's own rules, and reporting the result as passing. Two minutes against a real German government validator found what an unbounded number of runs against my own validator never would have.

The part that actually stung: Factur-X embeds a CII document under the hood, produced by calling that exact same serializer. Every Factur-X PDF I'd generated carried the identical defect, silently — because Factur-X's own check only verifies PDF/A-3 conformance, never the embedded XML's schema conformance on its own. So this wasn't an XRechnung-shaped bug, it was three shipped, GA, paid formats — plain CII, XRechnung-CII, and Factur-X — all producing documents a strict receiver would bounce, while my own API told every caller they were valid.

Where that leaves things

Every one of these got caught the same way: by refusing to let my validator be the last word on itself. Counting rules against the artifact's own structure instead of trusting a cached number. Running generated output through something I didn't write and can't accidentally agree with. None of it was expensive to set up, and all of it would have kept shipping quietly otherwise.

I'm not going to pretend this closes the book — CII coverage from CEN's own corpus is still thin next to UBL, and that's a known gap I'm tracking, not one I've fixed. But “I found this because an outside tool disagreed with me” is a much better sentence than “I'm assuming this is fine because my own validator says so,” and now, for these formats at least, it's the one I get to write.