Autopilot safety statistics

[PacManMX]

Best article I've seen so far re Autopilot safety stats. A lot of people have said what this article states, but this goes further into the numbers. Apologies if it is a duplicate post but I did multiple searches to make sure it isn't.

How safe is Tesla Autopilot? A look at the statistics.

 

[Ohji]

This article was well written and extremely informative -- thanks for the link!

 

[Lauz]

It is a terrible article. One of the (many) mistakes:

Quote;
But Bubslug, apparently not a statistician, didn’t include a confidence range. Due to the low number of vehicle-years for the Tesla, its confidence range would have been huge—something on the order of 0 to 75.
/Quote

If Tesla's numbers were based on a statistical sample - then yes, you would estimate a confidence range. If you interview 5000 people on who they will vote for? Hillary or Trump. Then yes, you also estimate a confidence interval since you are extrapolating your result from 5000 people to the entire electorate.

However, when you count all the votes and declare a winner, there will be no confidence range. Why? Because you counted the entire population. No extrapolation - no confidence range.

Tesla's numbers are population numbers, not extrapolated/statistically derived from a sample or a subset of cars. That is the power of having a fleet of cars online 24x7 always reporting back.

 

[ecarfan]

That article makes a number of valid criticisms of Musk's claim. Here are some of them:

Quote: "The NHTSA number that Musk presumably used to derive his one-fatality-every-94 million-mile benchmark is the Fatality Rate per 100 Million VMT (Vehicle Miles Traveled). For the last few years, that number has hovered a bit above 1.00, which translates to a miles-per-fatality number a bit under 100 million. This traffic fatality number from the agency, however, happens to include bicycles, motorcycles, pedestrians, 18-wheelers and buses. In fact, only 36 percent of the “traffic fatalities” listed by NHTSA in 2015 were occupants of passenger cars. (Another 28 percent were classified as light trucks, most of them presumably SUVs and pick-ups.) Tesla’s statistical comparison essentially equates the Florida Autopilot crash fatality with a pedestrian being run over by a bus. This is apples-vs-aardvarks. Because of these glaring representative-sample flaws, Tesla’s comparison “has no meaning,” according to Alain Kornhauser a Princeton transportation professor, quoted inMIT Technology Review."

Quote: "Even if we limit the statistical comparison of Autopilot Teslas to other passenger vehicles, many factors can skew the numbers in Autopilot’s favor. Tesla Autopilot driving occurs primarily on limited-access four-lane highways, which are typically safer than other types of roads—particularly on a per-mile basis. Most Autopilot driving is done in daytime, good weather, on dry roads and in good visibility. Autopilot Teslas are typically driven by wealthy middle-aged males, a demographic with a generally good driving record. Not many are likely piloted by teen-agers, or drunks, two groups with far worse crash rates. Among passenger vehicles, the Tesla is a very heavy vehicle with a low center of gravity and excellent crashworthiness. All of those factors would be expected to give an Autopilot-equipped Tesla a lower fatality rate than other passenger vehicles-even if the Autopilot is turned off."

I think those criticisms are reasonable.

 

[Lauz]

Quote: "Even if we limit the statistical comparison of Autopilot Teslas to other passenger vehicles, many factors can skew the numbers in Autopilot’s favor. Tesla Autopilot driving occurs primarily on limited-access four-lane highways, which are typically safer than other types of roads—particularly on a per-mile basis. Most Autopilot driving is done in daytime, good weather, on dry roads and in good visibility. Autopilot Teslas are typically driven by wealthy middle-aged males, a demographic with a generally good driving record. Not many are likely piloted by teen-agers, or drunks, two groups with far worse crash rates. Among passenger vehicles, the Tesla is a very heavy vehicle with a low center of gravity and excellent crashworthiness. All of those factors would be expected to give an Autopilot-equipped Tesla a lower fatality rate than other passenger vehicles-even if the Autopilot is turned off."

I think those criticisms are reasonable.

Even if you limit the comparison to autopilot Tesla's driven on autopilot versus autopilot Tesla's manually driven on highways you would end up with the same problem.

Autopilot is most like disabled due to bad weather conditions. Or autopilot users might decide longer trips due to the reduced level of fatigue, thus ending up comparing short and medium trip-length with long trip-lengths. Good weather conditions versus bad.

You can ALWAYS raise objections towards some third party effect not accounted for.

The article in question does not raise these concerns. Instead he tries to change the metric miles driven to vehicle years. A metric ridden with its own problems. Which, he failed to address in any way.

 

[Ohji]

It is a terrible article. One of the (many) mistakes:

Quote;
But Bubslug, apparently not a statistician, didn’t include a confidence range. Due to the low number of vehicle-years for the Tesla, its confidence range would have been huge—something on the order of 0 to 75.
/Quote

If Tesla's numbers were based on a statistical sample - then yes, you would estimate a confidence range. If you interview 5000 people on who they will vote for? Hillary or Trump. Then yes, you also estimate a confidence interval since you are extrapolating your result from 5000 people to the entire electorate.

However, when you count all the votes and declare a winner, there will be no confidence range. Why? Because you counted the entire population. No extrapolation - no confidence range.

Tesla's numbers are population numbers, not extrapolated/statistically derived from a sample or a subset of cars. That is the power of having a fleet of cars online 24x7 always reporting back.

Interesting point! I actually know very little about the actual math of statistics, so I am curious what the correct approach would have been. Is there a statistically valid way of representing a range of possibilities with the data available from Tesla? To me, the data Tesla has provided isn't nearly enough to make any claim about autopilot being safe or not -- death while driving on Autopilot is so infrequent that far more miles driven will be needed for a statistically significant result. There must, however, be a way of representing a range of possible results that would gradually get narrower as more miles are driven.

I think the most accurate part of the article is that there just isn't enough data to justify any claim that using Autopilot is safer than driving without it. The system is active in only a very small set of driving circumstances and the demographics of the folks using it are not representative of the entire driving population. It would be too hard to control for all the variables, no?

 

[PacManMX]

You can ALWAYS raise objections towards some third party effect not accounted for.

The article in question does not raise these concerns. Instead he tries to change the metric miles driven to vehicle years. A metric ridden with its own problems. Which, he failed to address in any way.

I don't think the purpose of the article is to state the problems with all other metrics. It's to say the comparison by Musk is literally nonsense. He (smartly) stays away from making claims about the safety of autopilot by pointing out that it can not be discerned by using even industry accepted metrics (which of course have flaws, but which are at least not apples vs aardvarks).

If Tesla wanted to get serious about safety stats they could release detailed data and allow third parties to analyze. This won't happen for obvious reasons (which isn't unique to Tesla). So for now we are stuck with subjective opinions and stats that are released as newsbytes. This article does a good job of helping readers discern for themselves the validity of said stats.

Don't get me wrong, I LOVE autopilot; but I don't know if it is safer or not than my own manual driving (in my case it probably is!). Safety certainly wasn't a primary reason for me buying the car as all the cars I was considering are "safe enough" in my opinion.

 

[Ohji]

Interesting point! I actually know very little about the actual math of statistics, so I am curious what the correct approach would have been. Is there a statistically valid way of representing a range of possibilities with the data available from Tesla? To me, the data Tesla has provided isn't nearly enough to make any claim about autopilot being safe or not -- death while driving on Autopilot is so infrequent that far more miles driven will be needed for a statistically significant result. There must, however, be a way of representing a range of possible results that would gradually get narrower as more miles are driven.

I think the most accurate part of the article is that there just isn't enough data to justify any claim that using Autopilot is safer than driving without it. The system is active in only a very small set of driving circumstances and the demographics of the folks using it are not representative of the entire driving population. It would be too hard to control for all the variables, no?

Actually, I just realized that maybe comparing the death rate in Teslas driven with autopilot on vs the death rate of Teslas driven with Autopilot off might work at a very basic level (still a lot of issues with this approach). I think the article mentioned that. Tesla has so much data that they could look at a defined sets of parameters (i.e. highway driving of 5+ miles in conditions where autopilot would be available) and look at the rate of accidents/fatalities with autopilot on vs off. Would control for a lot of the unknowns...

 

[thimel]

Interesting point! I actually know very little about the actual math of statistics, so I am curious what the correct approach would have been. Is there a statistically valid way of representing a range of possibilities with the data available from Tesla? To me, the data Tesla has provided isn't nearly enough to make any claim about autopilot being safe or not -- death while driving on Autopilot is so infrequent that far more miles driven will be needed for a statistically significant result. There must, however, be a way of representing a range of possible results that would gradually get narrower as more miles are driven.

I think the most accurate part of the article is that there just isn't enough data to justify any claim that using Autopilot is safer than driving without it. The system is active in only a very small set of driving circumstances and the demographics of the folks using it are not representative of the entire driving population. It would be too hard to control for all the variables, no?

I think a good measure would be the number of airbag deployments in Teslas with AP on compared to the number in Teslas without AP available driven on the same types of roads and conditions as those for which AP was used. This would greatly improve the statistics compared to deaths. It also tries to reduce the bias due to AP primarily being used in easy driving conditions.

 

[Lauz]

Interesting point! I actually know very little about the actual math of statistics, so I am curious what the correct approach would have been. Is there a statistically valid way of representing a range of possibilities with the data available from Tesla? To me, the data Tesla has provided isn't nearly enough to make any claim about autopilot being safe or not -- death while driving on Autopilot is so infrequent that far more miles driven will be needed for a statistically significant result. There must, however, be a way of representing a range of possible results that would gradually get narrower as more miles are driven.

The current official number of 1 fatality per. X million miles driven is a fact. It is not like many other fatality statistics estimated. Meaning it can be accepted as the absolute truth today. No confidence intervals, no 'buts'.

However, if you want to predict the future fatality rate? It is almost going to be impossible. We simply don't have enough data to make predictions any better than assuming the current track record is our best guesstimate. And even with more and more data being collected as time goes, we will always have the challenge that autopilot improvements are changing the numbers on a daily basis. Newer versions of software and hardware, as well as fleet learning constantly tweaks the parameters of our study.

This entire debate is IMHO blown out of proportions. We know that driving a 10 year old car compared to a brand new Tesla is much more dangerous. But we still allow it. We know that young males are much more likely to end up in an accident and get killed. But we still allow them on the road. And don't get me started with phones.... Or why allow cars to drive above the speed limit at all when we have the technology to prevent it?

All we have is a ground truth consisting of the fatality track record so far. It is looking good compared to other well known factors that make our lives on the road more dangerous.

Elon simply pointed out the ground truth. Fatality numbers so far - which is our only valid guideline right now.

 

[S4WRXTTCS]

We can't measure Autopilot safety because it's not meant to be used without supervision of the human. We can't measure the safety of AP+Human because the human can't be trusted.

The whole point of AP is to make it safer on each iteration of the SW/HW. That's something you can't do with humans, or at least humans would get really upset by that kind of attempt at evolving human driving.

Even if we were going to try to measure the safety of Autopilot we'd have to reset the statistics at each interval of firmware if a significant change happen.

7.1 is significantly different than 7.0 because the addition of nags.
8.0 is significantly different than 7.1 because we now have the "no AP for you" red hands of shame.
8.1 will be significant different than 8.0 because AEB will use additional radar data plus white listing to enable the car to stop for stopped objects (so far the number one cause of driver fatality in a Tesla under AP).