Proving self-driving cars are safe could take up to hundreds of years under the current testing regime, a new Rand Corporation study claims. Source: Self-driving cars may not be proven safe for decades: report  The statistical analysis in this paper looks fine, but the problem is even worse for aircraft (since they are far safer per mile than autos.) Yet new aircraft are sold after approx 3 years of testing, and less than 1 million miles flown. How?

From the report:

we will show that fully autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their reliability in terms of fatalities and injuries. Under even aggressive testing assumptions, existing  fleets would take tens and sometimes hundreds of years to drive these miles.

How does the airline industry get around the analogous statistics? By understanding how aircraft fail, and designing/testing for those specific issues, with carefully calculated specification limits. They don’t just fly around, waiting for the autopilot to fail!

Failure-Free Miles Needed to Demonstrate Maximum Failure Rates (Rand report 2016)

But this will probably require a much better understanding of automobile guidance than anyone has today. Of course, most of the same knowledge is needed to design the autonomous system in the first place. My friend Don Norman says that accumulating such knowledge is the main reason Google has so many cars out, driving around the Bay Area. They “want” crashes and near-crashes, to learn from them.

To their credit, the authors are aware of some alternative testing methods, and their paper does a valuable job of framing the “why is this going to be hard” problem.