Why Autopilot Safety Claims are BS

[ProtonSF]

 

[neurocutie]

1. What do you expect? (smart drivers should disengage AP in difficult situations. AP is not meant for those situations).
2. What do you expect? (ditto to the above)
3. Autopilot isn't meant for city driving anyways, so its not part of the "claim".

 

[ProtonSF]

1. What do you expect? (smart drivers should disengage AP in difficult situations. AP is not meant for those situations).
2. What do you expect? (ditto to the above)
3. Autopilot isn't meant for city driving anyways, so its not part of the "claim".

I guess it's just not clicking for you, so let me simplify it: Tesla is cherrypicking data. Full stop. If an accident is about to happen almost every driver will immediately take control of the vehicle, thus EXCLUDING that collision from Tesla's count. Likewise, if Autopilot refuses to engage in difficult driving situations, that automatically taints the data. And given that your per-mile chance of a collision in a city is far greater than highway driving, and AP is almost always used on highways, that is also a major apples-to-oranges comparison. Finally, Tesla drivers are generally safer drivers than the general public. So all of Tesla's claims about how safe Autopilot is might be technically correct, but when put in the proper context those claims goes out the window. If you don't understand this then I'm not sure what to tell you - clearly you're not statistician.

 

[Tam]

...clearly you're not statistician...

No, I am not but how do you explain:

"For those driving without Autopilot and without our active safety features, we registered one accident for every 978 thousand miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles."

Tesla also posts the numbers for classic Tesla that has no Active Safety Features and no Autopilot as well. Those cars don't choose the least risky roads to run on.

 

[ProtonSF]

No, I am not but how do you explain:

"For those driving without Autopilot and without our active safety features, we registered one accident for every 978 thousand miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles."

Tesla also posts the numbers for classic Tesla that has no Active Safety Features and no Autopilot as well. Those cars don't choose the least risky roads to run on.

Did you miss the part about how Tesla drivers are on average older, and therefore, generally safer drivers? Again, until Tesla provides full data transparency, no real conclusions can be made.

 

[Microterf]

Did you miss the part about how Tesla drivers are on average older, and therefore, generally safer drivers? Again, until Tesla provides full data transparency, no real conclusions can be made.

Those 2 guys in Texas were 59 and 69. Not very young and apparently not very safe.

 

[Tam]

Those 2 guys in Texas were 59 and 69. Not very young and apparently not very safe.

Let's give these 2 elderly men a break. They were reportedly both engineers and friends.

I am not sure but if I got to their age and see all those young people partying in a Tesla running at 65MPH with no one in the driver seat, shouldn't I want to test that out since others said it's impossible because of the weight sensor, seatbelt sensor, torque sensor, countdown timer..?

https://youtu.be/lu2hPViqHy4


And there's even a demonstration of a young man crawling out of the driver's seat to take a nap in the back seats while the car was running at 64MPH.

If these young people could do it, I don't see why these very well-trained elderly engineers can't test that out either.

https://youtu.be/PZK8MApWdaU

https://youtu.be/PZK8MApWdaU

 

[Barklikeadog]

No, I am not but how do you explain:

"For those driving without Autopilot and without our active safety features, we registered one accident for every 978 thousand miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles."

Tesla also posts the numbers for classic Tesla that has no Active Safety Features and no Autopilot as well. Those cars don't choose the least risky roads to run on.

All that does is prove that it is the driver. If your entire fleet is made up primarily of upper income middle aged people.... Then that is what your safety numbers should be compared to. That is far and away the safest driving demographic. If you had elderly, poor and teenagers driving your vehicle... Then the result would suddenly look like the 'average car' on the road.

How does the Model X compare to the Volvo xc90?

 

[ProtonSF]

I'm honestly extremely dissapointed at how few of my fellow Tesla drivers seem to understand the basics on stats and data integrity.

If you really believe the "AP is 10X safer" figure, PM me because I have a bridge to sell you!

 

[GZDongles]

I guess it's just not clicking for you, so let me simplify it: Tesla is cherrypicking data. Full stop. If an accident is about to happen almost every driver will immediately take control of the vehicle, thus EXCLUDING that collision from Tesla's count. Likewise, if Autopilot refuses to engage in difficult driving situations, that automatically taints the data. And given that your per-mile chance of a collision in a city is far greater than highway driving, and AP is almost always used on highways, that is also a major apples-to-oranges comparison. Finally, Tesla drivers are generally safer drivers than the general public. So all of Tesla's claims about how safe Autopilot is might be technically correct, but when put in the proper context those claims goes out the window. If you don't understand this then I'm not sure what to tell you - clearly you're not statistician.

I more or less agree with cherry picking data, but Tesla does count crashes that occur just after autopilot was disengaged in their stats. So if someone sees an imminent accident and disengages autopilot and then crashes, that will still count as an accident with autopilot engaged.

 

[ProtonSF]

I more or less agree with cherry picking data, but Tesla does count crashes that occur just after autopilot was disengaged in their stats. So if someone sees an imminent accident and disengages autopilot and then crashes, that will still count as an accident with autopilot engaged.

Per their disclosure, it's 5 seconds. If an accident happens 5 seconds after AP disengages, it's not counted. So if you suddenly come onto a stretch of highway that is iced over, and disengage AP, you're still at a massively higher risk of a collision because the ice doesn't just dissapear after 5 seconds. Same with other weather conditions and driving conditions, including being surrounded by trucks or shitty drivers.

 

[GZDongles]

Per their disclosure, it's 5 seconds. If an accident happens 5 seconds after AP disengages, it's not counted. So if you suddenly come onto a stretch of highway that is iced over, and disengage AP, you're still at a massively higher risk of a collision because the ice doesn't just dissapear after 5 seconds. Same with other weather conditions and driving conditions, including being surrounded by trucks or shitty drivers.

But at that point, the crash isn't the "fault" of autopilot, so I think that's fair to exclude. Of course it skews the statistics when you are only supposed to use autopilot in already-safe driving conditions, but at least they aren't excluding accidents where the car is about to slam into a brick wall and the driver disengages in the last second.

 

[ProtonSF]

But at that point, the crash isn't the "fault" of autopilot, so I think that's fair to exclude. Of course it skews the statistics when you are only supposed to use autopilot in already-safe driving conditions, but at least they aren't excluding accidents where the car is about to slam into a brick wall and the driver disengages in the last second.

Yeah, I mean, we're not talking about a little tiny tainting of the data here. The bias is massive. The claims about AP safety are fundamentally unsound when compared to any other data set.

 

[ZeApelido]

Yeah, I mean, we're not talking about a little tiny tainting of the data here. The bias is massive. The claims about AP safety are fundamentally unsound when compared to any other data set.

Well I certainly agree that Tesla is reporting conditional probabilities that are not adjusted for the fact that conditions are not exactly the same in with Autopilot and without Autopilot comparisons. I've complained about that before.

But I think you are way over the top with your reaction.

If a fully controlled comparison was desired, you would have to control for comparing performance of Autopilot on the same highways as without, for the same number of miles as without, etc...

But if you really wanted it controlled you would have to control of individual users, evening out their contribution to both conditions (even for the same road, users that use Autopilot might differ in driving competence to users than don't use it). This obviously becomes a big pain, because assumptions have to be made all over.

The end result is likely a diminished effect of superiority of Autopilot.

Tesla, like basically every other company in the world, will use statistics as marketing. That their results may not be accepted in a peer-reviewed journal doesn't really differentiate them from any other company.

Certainly, if looking for regulatory approval, stronger data would be needed.

I think your worry about crashes 5 seconds after disengagement are a minor concern.

 

[helvio]

I can't say I don't understand where you're coming from. Yes, every company will try to make their data look nicer than it is. But I also don't buy that the data make-up is massive. No data is 100% accurate, as it's mostly collected by sample, we don't have an everything-sensor on every single device.

Tesla almost kind of does, but still, there are exceptional cases where it will not register, like the antenna being the first damaged component - who knows. I'm being hypothetical.

But I suggest this quick read from their website (Tesla Vehicle Safety Report) which I believe you've read:

Methodology:
We collect the exact amount of miles traveled by each vehicle with Autopilot active or in manual driving, and do so without identifying specific vehicles to protect privacy. We also receive a crash alert anytime there is a crash that is correlated to the exact vehicle state at the time. This is not from a sampled data set, but rather this is exact summations. To ensure our statistics are conservative, we count any crash in which Autopilot was deactivated within 5 seconds before a crash, and we count all crashes in which the crash alert indicated an airbag or other active restraint deployed. In practice, this correlates to nearly any crash at about 12 mph (20 kph) or above, depending on the crash forces generated. On the other hand, police-reported crashes from government databases are notoriously under-reported, by some estimates as much as 50%, in large part because most fender benders are not investigated. We also do not differentiate based on the type of crash or fault, and in fact, more than 35% of all Autopilot crashes occur when the Tesla vehicle is rear-ended by another vehicle. In this way, we are confident that the statistics we share unquestionably show the benefits of Autopilot.

Don't take me wrongly here. I am always open to defying the data. But you have to present BETTER data. Which I wouldn't classify a YouTube video from what seems to be an export of a PowerPoint presentation with a poorly ended piano solo as.

Also I flat out call BS on the age argument. Where I live, you see all sorts of people driving Teslas.
Don't forget they have a $30K car all the way up to $250K now. Maybe more. It's not beyond reach, just the wait time sucks.

EDIT: Quick addendum to the claim that AP is mostly engaged on highways.
Yes it is. Which for a non-AP car, might be boring, tiresome and lead to human error.

 

[Barklikeadog]

Also I flat out call BS on the age argument. Where I live, you see all sorts of people driving Teslas.
Don't forget they have a $30K car all the way up to $250K now. Maybe more. It's not beyond reach, just the wait time sucks.

Cmon man, don't play the 'what I see' card.
From master vehicle owner database:
55% model 3 owners live in wealthiest 10% of zip codes.
Avg 3 buyer income is 128,000 (s and x much higher)
84% owners model 3 are male (x is 71%, SS is 77)
Median age 46 ( model x 52, a 54)
Overall tesla owners have an 88% home ownership rate. ( general pop 64%)

I've been beating this drum about BS safety stats for a while now.
Tesla owners are far and away the safest driving demo there is. Higher income and middle aged. Other vehicles with the same drivers (like Volvo) have similar numbers. They cannot and should not be compared to the 'average car' as elon loves to do. Better drivers are also more likely to pay for extra safety features. It's the driver, not the car.

 

[hgmichna]

I think a most valuable statistical result is comparing older Teslas that do not have Autopilot with newer ones, because this isolates the role of Autopilot. Are there any figures anywhere?

 

[qdeathstar]

All that does is prove that it is the driver. If your entire fleet is made up primarily of upper income middle aged people.... Then that is what your safety numbers should be compared to. That is far and away the safest driving demographic. If you had elderly, poor and teenagers driving your vehicle... Then the result would suddenly look like the 'average car' on the road.

How does the Model X compare to the Volvo xc90?

what this points too is that poor uneducated people shouldn’t be driving. We should invest in real mass transit instead

 

[helvio]

Cmon man, don't play the 'what I see' card.
From master vehicle owner database:
55% model 3 owners live in wealthiest 10% of zip codes.
Avg 3 buyer income is 128,000 (s and x much higher)
84% owners model 3 are male (x is 71%, SS is 77)
Median age 46 ( model x 52, a 54)
Overall tesla owners have an 88% home ownership rate. ( general pop 64%)

I've been beating this drum about BS safety stats for a while now.
Tesla owners are far and away the safest driving demo there is. Higher income and middle aged. Other vehicles with the same drivers (like Volvo) have similar numbers. They cannot and should not be compared to the 'average car' as elon loves to do. Better drivers are also more likely to pay for extra safety features. It's the driver, not the car.

Good information. Sources?

Also, you say they shouldn’t compare to the average car. Who should they compare to, and why?

It seems to me that you’re complaining of the sake of complaining. This has potential to be a very productive debate with serious insights.

I would suggest you go to NHTSA Data here

Crash Data Systems | NHTSA

www.nhtsa.gov

And run your own metrics. Post the results and a brief explanation of why your comparison is more “apples to apples” than not.

 

[qdeathstar]

Good information. Sources?

Also, you say they shouldn’t compare to the average car. Who should they compare to, and why?

It seems to me that you’re complaining of the sake of complaining. This has potential to be a very productive debate with serious insights.

I would suggest you go to NHTSA Data here

Crash Data Systems | NHTSA

www.nhtsa.gov

And run your own metrics. Post the results and a brief explanation of why your comparison is more “apples to apples” than not.

100 bucks says he won’t do it.