Suppose we publish that our detector finds 99% of faces. A reasonable person reads that as: almost everyone is protected, and the occasional person is not.
That is not what it means. It means each face has a one-in-a-hundred chance of being missed. And a frame from a city street does not contain one face. It contains five, or ten, or twenty.
The multiplication
For a frame to be clean, every face in it must be found. At 99% per face:
| Faces in frame | Probability the frame is clean |
|---|---|
| 3 | 97.0% |
| 5 | 95.1% |
| 10 | 90.4% |
| 20 | 81.8% |
At twenty faces in frame, nearly one frame in five contains someone identifiable. Push the detector to 99.9% and the same frame is clean 98% of the time — better, and still not what the headline number implied.
The gap between the two numbers is not a rounding artefact. It is the difference between a claim about faces and a claim about scenes, and only one of them is the thing a person cares about when they walk past a camera.
The same problem, in time
Video makes it worse, because a face does not appear once. It appears in every frame it is visible for, and it has to be covered in all of them. Miss it in one and it is exposed.
Treat the frames as roughly independent and a 99% per-frame detector covers a thirty-frame track 74% of the time. Over three hundred frames — ten seconds of footage — it drops to about 5%. To reach 99% coverage of a three-hundred-frame track you would need per-frame recall of 99.9966%, which nobody has and nobody will.
Detection on every frame independently is not an architecture. It is a way of guaranteeing failure and reporting it as success.
The consequence is a design requirement rather than an optimisation. You detect on some frames and propagate the redaction region across the whole track, forward and backward, dilated generously. Coverage becomes a property of the track. The per-frame number stops being the number that matters, which is fortunate, because the per-frame number was never going to be good enough.
Why a single figure is always the wrong figure
Two more corrections, both of which push the honest number down and the required evidence up.
Faces are not independent samples. Several faces in one frame share the same lighting, the same weather, the same motion blur, the same camera state. Statistically they behave as one observation wearing several coats. Measure a thousand faces drawn from two hundred frames and you have nothing like a thousand independent trials — closer to three or four hundred, depending on how strongly conditions correlate within a frame. Any confidence interval that ignores this is narrower than the evidence supports.
Averages conceal the failure that matters. A pooled 99.5% that hides 94% on small distant plates at dusk is not a measurement, it is a blend. And dusk is not an edge case; it is the commute. The number a reader needs is the worst stratum, not the mean — broken out by object size in pixels, illumination, weather, occlusion, angle and camera position.
Clear the bar in every stratum and you have said something. Clear it on average and you have said nothing, at greater length.
What honest evidence costs
Worth stating, because the effort is why nobody does it. Supporting “at least 99% recall, at 95% confidence” with a clean run takes on the order of three hundred annotated instances. Supporting 99.9% takes around three thousand. Allow a handful of misses and both roughly double.
Then apply the two corrections above. Stratify across a dozen meaningful conditions, adjust for clustering within frames, and hold the confidence level across all of them at once, and a credible published figure needs tens of thousands of annotated instances — each one labelled by somebody who was told, in advance and in writing, what counts as a face. A face at eight pixels. A face in a wing mirror. A face on a billboard. A face on a passenger’s shirt.
That annotation standard is the real artefact. The percentage is downstream of it, and a percentage published without it is a number without a denominator.
So we will publish two numbers
The one we lead with is the share of frames in which nothing identifiable survives, because that is the claim a journalist can test by opening the product and looking. Underneath it, per-face and per-plate recall, stratified, with intervals, with the counting rule attached.
The second number will be higher and less meaningful. We are putting it second deliberately.
What would change our mind
A convincing argument that per-frame-clean is the wrong headline — for instance that in practice a single missed face in an otherwise redacted frame carries materially less risk than the arithmetic implies, because context has been removed around it. We do not currently believe that, but it is the strongest counterargument and we have not seen it properly made.
What we are least sure about
Whether frames should be treated as independent at all when computing track coverage. They are not — a detector that misses a face in one frame is more likely to miss it in the next, because the reason for the miss usually persists across the track. That correlation makes track coverage better than the naive multiplication suggests in some conditions and worse in others, and we do not yet have a defensible model of it. It is the largest open question in our own methodology and we would rather name it than smooth over it.