Consider how it works elsewhere. Accounts are audited by a firm that did not prepare them. Drugs are trialled under protocols set outside the company that will sell them. Vehicle crashworthiness is rated by bodies with no commercial interest in the result. Building work is signed off by an inspector who does not work for the builder.
None of these systems is perfect. All of them exist because we learned, expensively, that self-assessment under commercial pressure produces predictable answers.
Medicine has the cleanest natural experiment. When a US health institute’s large trials were examined before and after prospective declaration of outcomes became the norm, the proportion showing a significant benefit fell from 57% of trials published before 2000 to 8% of those published after. Thirty studies became twenty-five; seventeen positive results became two. Nothing about the underlying science changed. What changed was who wrote down the outcome measure, and when.
The authors are careful — they say pre-registration “may have contributed” to the trend, and it is an observational finding across fifty-five trials. We are careful with it too. But a fifty-point swing is not a rounding artefact, and it is the most direct evidence available that defining the test in advance and in public changes the result.
Now consider a company deploying an autonomous system, or a driver-monitoring product, or a collision-prediction model. It selects its test data. It defines what counts as a detection, a miss and a false positive. It chooses which baseline to compare against. It publishes the result. And in most cases nobody outside the company can reproduce any of it.
How the numbers get made
The manipulation, where it happens, is rarely fraudulent. It is definitional, and it is often invisible even to the people doing it.
- Choose the test set. Every dataset has a distribution. Evaluate on data drawn from the same distribution you trained on and the result is excellent and uninformative.
- Define the categories. What counts as an “edge case” is a choice. Draw the boundary generously and your long-tail performance improves without a line of code changing.
- Pick the comparator. Comparing against a weak baseline, or against a system optimised for something else, produces a large multiple that means very little.
- Blend unlike tests. The most common one, and the most damaging: reporting an independent institution’s test of one product alongside an internal benchmark of another, in a single sentence, so the credibility of the first transfers to the second.
And the industry that best demonstrates the failure mode is not software. Real-world nitrogen-oxide emissions from certain diesel vehicles were found to exceed the standard by fifteen to thirty-five times — by vehicles that had passed laboratory certification. The divergence was only discoverable because researchers tested off the known test conditions. The mechanism was literal evaluation-gaming: the software detected that it was being tested and behaved differently.
That last one is worth being specific about, because we have to be careful of it ourselves. Independent controlled testing by the Virginia Tech Transportation Institute measured our alert performance. Our BADAS model publishes results across its own long-tail benchmark with the category definitions attached. These are two different things, measured by two different parties, about two different products. Writing “independently ranked first across every benchmark” would merge them into a claim neither one supports, and it would be the sort of sentence that gets written by accident in a hurry.
The failure mode is not lying. It is a true sentence and a true sentence placed next to each other so that a third, untrue thing is implied.
What independent verification would need
If this industry is going to have the institution it lacks, four things have to be true of it.
- Held by someone with no stake. The evaluating party must not sell a competing system, and must not be paid on the outcome.
- Data the system has not seen. Held-out real-world events, drawn from a distribution the developer did not choose, with the outcome known.
- Definitions fixed in advance and published. What counts as a detection, a miss, a false positive and an edge case, written down before the run and readable afterwards.
- Reproducible. Another party running the same protocol should get the same answer. If they cannot, it was a demonstration rather than a measurement.
Our position, and its obvious problem
We are not a neutral commentator here. We hold a large real-world record, we do not build a competing autonomy stack and we do not write insurance, which puts us in an unusually good position to supply held-out evaluation data. That is a commercial opportunity for us and you should read this essay knowing that.
The honest version of our position is therefore narrower than “we should be the referee.” It is this: the data used to evaluate a system should come from somewhere other than the party being evaluated, and the definitions should be public. We are one possible source of the first. We are not, and should not be, the authority on the second.
A standards body, a public benchmark with governance, or a regulator would all be better than any vendor including us. In the absence of any of those, published definitions and held-out data are the minimum, and we publish ours — including the model, openly, so anybody can test it against their own roads and disagree with us in public.
A correction we owe the reader
It would be easy, and wrong, to say that autonomy developers never submit to peer review. They do. The leading programme has published crash-rate comparisons in peer-reviewed traffic-safety journals, and independent academics have published methodological critiques of them in the same venues. That is a functioning scholarly conversation and it is more than most of this industry manages.
The narrower point survives, and it is the one that matters: those papers are authored by the party being evaluated, using proprietary exposure data no reviewer can independently recompute, against human baselines the authors themselves constructed. Peer review checks reasoning. It does not re-run the numbers. When a manufacturer retroactively revised its own published safety statistics and the downward adjustments fell disproportionately on the comparison category that flattered its own product, the only reason anyone knows is that an outside academic went and checked.
And regulators find what self-reporting does not. One investigation opened into an autonomous programme surfaced nine additional similar incidents after it began. Another operator was fined $1.5 million for filing incomplete reports under a mandatory reporting order — the omitted detail being that its vehicle had dragged a pedestrian.
How other industries actually solved it
Two mechanisms are worth copying and neither requires secrecy.
Anonymous procurement. The vehicle safety testing bodies buy their test cars from dealerships as an ordinary customer would. No vehicle comes from the manufacturer. The protocol is fully published; what the manufacturer cannot control is which car gets tested. That is the physical-world equivalent of a held-out set the developer cannot access, and it defeats the standard objection that independent evaluation means secret evaluation.
Publication contingent on registration. Medical journal editors could not compel anyone to pre-register a trial, so they made publication conditional on it. The AI equivalent — conferences declining benchmark results not pre-registered against a held-out set — is the single most actionable precedent in this essay, and it requires no regulator at all.
What would change our mind
A credible independent institution emerging — a standards body, a consortium, an insurer-funded test house. If one exists, our role is to supply held-out data to it and stop arguing. We are making this case because the institution is absent, not because we want the job.
What we are least sure about
Whether independence survives contact with money. Audit independence is imperfect, mandated trial reporting was widely ignored for years, and one national regulator has still not finalised conflict-of-interest guidance more than a decade after a law required it. Independence regimes decay. The honest claim is that they decay more slowly than self-assessment does, not that they solve the problem.
The cost of not fixing this
Self-graded safety claims work until the first serious public failure of a system that had excellent published numbers. At that point the reasonable inference is not “that company overstated” but “none of these numbers mean anything,” and every honest actor in the category pays for it.
Which is the practical argument for fixing it now, while it is a discipline the industry chooses rather than one imposed on it after an incident.