What ten billion miles can't tell you

The autonomous vehicle industry spent a decade counting miles. But mileage alone was never going to tell us whether a machine was ready for the road.

In 2016, RAND researchers Nidhi Kalra and Susan M. Paddock asked how many miles autonomous vehicles would have to drive to demonstrate, with statistical confidence, that they were safer than human drivers. Their answer was hundreds of millions of miles, and in some scenarios hundreds of billions. U.S. roads recorded 1.19 traffic deaths per 100 million vehicle miles traveled in 2024, according to the National Highway Traffic Safety Administration. When the event you’re measuring is that rare, proving a meaningful difference takes an enormous amount of driving.

The math was sound. The researchers themselves warned that developers couldn’t simply drive their way to safety. But somewhere along the way, mileage became shorthand for something it was never designed to measure: how much an autonomous system actually understood about the road.

Miles became the number every program reported: in investor decks, launch announcements, state filings. California required autonomous-vehicle testing companies to report their mileage and disengagements, and miles per disengagement became a public scoreboard. Miles were easy to count, they only went up, and they sounded like progress.

But a mile doesn’t measure what a system has learned.

The thousandth lap of the same downtown loop at two in the afternoon adds another mile to the total. It doesn’t necessarily add another meaningful situation. Repetition can help prove consistency, but it can’t substitute for the unexpected.

What a system needs is range. Different roads and cities, weather, hours, construction, the driver in the next lane doing something nobody planned for. A mileage race doesn’t necessarily reward that. It can reward driving more in the places that are easiest to drive.

There’s a structural reason this happened, and it isn’t carelessness. Many of the industry’s largest driving datasets were built by companies developing and deploying their own systems, so the data went where the business went. A robotaxi program drives the neighborhoods it serves. A trucking program drives its freight lanes. An automaker’s test fleet drives the roads near its engineering center.

Each choice makes sense on its own. But put together, you get data shaped like a stack of business plans, and the world people actually drive in looks very different.

That creates blind spots. Hours of clear, mapped, well-marked roads, but potentially far fewer examples of a school bus stopped on a two-lane highway at dusk in freezing rain. The rare event is often the one a system most needs to recognize, and the one a deployment-driven collection plan is least likely to encounter.

Rare is a per-mile idea. Across all the driving in the United States, strange things happen every day. You find them by observing more roads, in more places, under more conditions, rather than repeatedly covering the same routes. Going around the same block faster won’t get you there.

This matters because the hardest situations aren’t necessarily the ones that produce the most miles. They’re the ones that expose what a system doesn’t yet know.

We see this in our own data. Across 1.1 billion miles of observed driving, Empiric Earth recorded more than 1.8 million near misses and nearly 96,000 collisions. Hard braking peaked at 5 p.m., when roads are crowded with commuters. But the hours when a near miss was most likely to become a collision were between 9 a.m. and noon. The busiest hours weren’t necessarily the most dangerous. If we’d only counted miles or focused on the periods with the most activity, we would have missed that distinction.

So if miles aren’t enough, what should we be measuring instead? I’d start with four questions.

1.

How representative is the data of the world where the system will operate?

Count the distinct roads, regions, conditions and situations, and be clear about what’s missing. A dataset that covers four cities well can be enormously useful for a system operating in those four cities. It shouldn’t be presented as representative of driving everywhere.

2.

How current is it?

Roads change. Lanes get repainted, a school opens, a highway gets torn up for two years. A record nobody refreshes describes a place that no longer exists. We should know not only when data was collected, but how often it is updated.

3.

How many of the hard cases are in it, and how do we know?

Rare and consequential events matter, but so does how they’re identified. Definitions should be established in advance, observations should be validated, and we should be able to distinguish genuinely different situations from repeated examples of the same one. A million examples don’t mean much if nobody can explain what they represent.

4.

Who collected it, and what incentives shaped the record?

Every collection strategy has blind spots. A record built to support a particular deployment will naturally reflect that deployment’s needs. Independent collection can reduce some of those incentives, but independence alone doesn’t guarantee representativeness or accuracy. We should understand how the data was gathered, what was excluded and how its quality was verified.

Scale is necessary. It isn’t sufficient.

None of this makes mileage meaningless. Empiric Earth has observed more than ten billion miles, and that scale matters. But the number alone doesn’t tell you what’s in the record. The value comes from what those miles reveal: the situations captured, the conditions represented, the changes observed and the gaps we can identify.

And this is bigger than cars.

Every company building robots for warehouses, sidewalks, farms and homes will face the same temptation. Hours of operation will become the new miles, and someone will run the same aisle ten thousand times and announce ten thousand hours of experience.

Those hours may demonstrate reliability in that aisle. They won’t necessarily tell you what happens when the layout changes, a worker steps into the path or something the machine has never encountered appears in front of it.

The AV industry has spent a decade confronting the limits of mileage as a measure of progress. The rest of Physical AI doesn’t need to repeat that experience.

We should stop confusing how much a machine has operated with how much it has learned, or how safely it will behave when something unexpected happens.

Count the situations.
The miles will follow.

Zach Greenberger is CEO of Empiric Earth.