Model X Crash on US-101 (Mountain View, CA)

[Economite]

Or perhaps, Musk has conditioned himself to know when to trust the system and when to take over, he remains attentive to the vehicles path at all times during the interview. I believe we all do the same, and rely on our own understanding and driving experiences to determine what the car can and cannot do.

Right... But if (i) Tesla's CEO, primary owner, and most visible spokesperson demonstrates the product on national tv with his hands off , and (ii) Tesla's CEO, primary owner, and spokesperson, essentially says "this is not a problem" when asked by a television personality whether he should be driving that way and (iii) as you say, "we all do the same" (drive hands-free but attentive); then you really have to wonder how serious Tesla is about its instructions that "you must keep your hands on the steering wheel at all times."

The instructions start to look kind of like the warnings on Q-Tip packages that the end of a Q-Tip should never enter the ear canal. The instructions seem less geared towards guiding proper use of the product, and more geared towards protecting Tesla from the legal consequences of the actual most-common use.

 

[Economite]

f you had owned the car during this time, or you read the report carefully, you would be able to see that they distinguish between AS and other AP functions. I can't help you any further.

The report does do an excellent of describing the various features and how they work. No question about that.

But it the drop from 1.3 crashes per million miles to 0.8 crashes per million miles is just comparing two buckets of miles: (i) all miles (whether AP features are being used or not) driven in AP1 hardware vehicles running a pre-AS version of the software and (ii) all miles driven in AP1 hardware vehicles (whether AP features are being used or not) while using the AS version of the software. The miles in the second bucket differ from those in the first bucket not only because the driver had the option to use AS during those miles, but also because the car was running later versions of AEB, BSM and all the other software features.

Thus, the statistic does not prove (or attempt to prove) that the very impressive drop in crash rates was a result of the presence of AS rather than improvements to those other features.

Besides, given that (i) most drivers are very judicious in their use of AS (and many were downright afraid of using it), (ii) the appropriate use case (eve broadly defined) for AS excludes many of the most dangerous kinds of driving conditions (heavy rains, poorly marked roads, anything with intersections/traffic signals), (iii) the crash statistic measured all crashes that activated airbags, not just the very serious crashes that might occur most often on highways, and (iv) the miles measured in both buckets are all miles, not just those with AS actually being used; do you really think it is remotely plausible that the ability to activate AS could have eliminated about 40% of all crashes fleet wide?

 

[mongo]

The report does do an excellent of describing the various features and how they work. No question about that.

But it the drop from 1.3 crashes per million miles to 0.8 crashes per million miles is just comparing two buckets of miles: (i) all miles (whether AP features are being used or not) driven in AP1 hardware vehicles running a pre-AS version of the software and (ii) all miles driven in AP1 hardware vehicles (whether AP features are being used or not) while using the AS version of the software. The miles in the second bucket differ from those in the first bucket not only because the driver had the option to use AS during those miles, but also because the car was running later versions of AEB, BSM and all the other software features.

Thus, the statistic does not prove (or attempt to prove) that the very impressive drop in crash rates was a result of the presence of AS rather than improvements to those other features.

Besides, given that (i) most drivers are very judicious in their use of AS (and many were downright afraid of using it), (ii) the appropriate use case (eve broadly defined) for AS excludes many of the most dangerous kinds of driving conditions (heavy rains, poorly marked roads, anything with intersections/traffic signals), (iii) the crash statistic measured all crashes that activated airbags, not just the very serious crashes that might occur most often on highways, and (iv) the miles measured in both buckets are all miles, not just those with AS actually being used; do you really think it is remotely plausible that the ability to activate AS could have eliminated about 40% of all crashes fleet wide?

Lane departure warning and AEB/FCW are shown to reduce crash rates in general, so AP vs non-AP should follow that trend regardless (not that AP is better than other OEM's versions, but that it is better than not having the assist features).

Lane departure warning cuts crashes

 

[Barklikeadog]

The report does do an excellent of describing the various features and how they work. No question about that.

But it the drop from 1.3 crashes per million miles to 0.8 crashes per million miles is just comparing two buckets of miles: (i) all miles (whether AP features are being used or not) driven in AP1 hardware vehicles running a pre-AS version of the software and (ii) all miles driven in AP1 hardware vehicles (whether AP features are being used or not) while using the AS version of the software. The miles in the second bucket differ from those in the first bucket not only because the driver had the option to use AS during those miles, but also because the car was running later versions of AEB, BSM and all the other software features.

Thus, the statistic does not prove (or attempt to prove) that the very impressive drop in crash rates was a result of the presence of AS rather than improvements to those other features.

Besides, given that (i) most drivers are very judicious in their use of AS (and many were downright afraid of using it), (ii) the appropriate use case (eve broadly defined) for AS excludes many of the most dangerous kinds of driving conditions (heavy rains, poorly marked roads, anything with intersections/traffic signals), (iii) the crash statistic measured all crashes that activated airbags, not just the very serious crashes that might occur most often on highways, and (iv) the miles measured in both buckets are all miles, not just those with AS actually being used; do you really think it is remotely plausible that the ability to activate AS could have eliminated about 40% of all crashes fleet wide?

you have explained it perfectly. You can't connect those two dots and say that X caused Y. Lane departure, blind spot warnings and AeB have reduced accident rates
in other vehicles... and they don't have autopilot.

 

[bhzmark]

remotely plausible that the ability to activate AS could have eliminated about 40% of all crashes fleet wide?

Do you think it is remotely plausible that that the ability to activate and use AS which is the primary change in the software during that time does not account for at least a substantial portion of that substantial decrease?

 

[Economite]

Do you think it is remotely plausible that that the ability to activate and use AS which is the primary change in the software during that time does not account for at least a substantial portion of that substantial decrease?

Absolutely... In fact, my hypothesis is that the drop in crashes would have been even bigger if AS were not added to the suite of features but all of the other software improvements still occurred.

The NHTSA crash statistic in no way disproves this hypothesis. It also does not prove it.

 

[bhzmark]

Absolutely... In fact, my hypothesis is that the drop in crashes would have been even bigger if AS were not added to the suite of features but all of the other software improvements still occurred.

The NHTSA crash statistic in no way disproves this hypothesis. It also does not prove it.

On what basis? during most of that period, for most of those cars, they already had AEB before and after AS.

There are three periods of car deliveries:

1) 4 months of initial ramp up : December 2014 to March 2015: cars delivered with AP hardware, but limited software.
2) 8 months of strong delivery : March 2015 to late October 2015: AEB functionality enabled

3) ~ 12+ months of strong delivery October 2015 through unspecified time in 2016: cars delivered: AS functionality enabled

Only cars in period #1 didn't have AEB. -- it was only the first third of the time period and January and Feb of that period were very slow. Most of the miles traveled in the first bucket (#1 and #2) with the 40% higher rate were for after March 2015 when AEB was enabled.

Also they mention that the comparison was between miles for cars without AS "installation" and after installation. A small but significant number of drivers don't buy AS at first (but do get AEB and FCW) . So they are also contributing to the pool of miles that have AEB, and all its improvements, but not AS, and they are in the 40% higher airbag deployments.

With lots of confounding factors nothing will be proven. But some hypotheses will be more or less supported by the evidence. There is no support for the hypothesis that AEB improvements alone, available to those who didn't buy AS, explain why miles traveled with AS available result in 40% fewer deployments.

 

[Economite]

My point is that all of the software features (AEB, TACC, BSM, etc) received periodic updates/improvements throughout the data period. Because AS was not available in any form during roughly the first half of the period, none of the miles where AS was available to the driver were driven on cars with the earlier versions of the other software features. All of those miles were driven with updated versions.

Thus for bucket one miles, drivers had no ability to choose to use AS, and most miles were driven with relatively earlier versions of other features.

Bucket 2 miles were ones where drivers not only had the choice of using AS, but also were exclusively using relatively later (and improved) versions of the other features.

 

[Ugliest1]

In other news, researchers are very close to determining exactly how many angels can dance on the head of a pin.

 

[bhzmark]

Yes that is what you said before and it is no more persuasive the second time around -- especially ignoring the points I made about the actual timing of the changes and the composition of the data.
For instance including people who never bought AP yet had the AEB updates. About 25% of cars don't even include AP when sold yet have AEB. That population is in the first bucket with their improved AEB.

My point is that all of the software features (AEB, TACC, BSM, etc) received periodic updates/improvements throughout the data period. Because AS was not available in any form during roughly the first half of the period, none of the miles where AS was available to the driver were driven on cars with the earlier versions of the other software features. All of those miles were driven with updated versions.

Thus for bucket one miles, drivers had no ability to choose to use AS, and most miles were driven with relatively earlier versions of other features.

Bucket 2 miles were ones where drivers not only had the choice of using AS, but also were exclusively using relatively later (and improved) versions of the other features.

 

[S4WRXTTCS]

The issue I have with the 40% crash reduction is that we're not given access to the data that Tesla used to determine that number.

We have this excellent write up by the NHTSA, but it includes this statistic that came from Tesla.

If we're going to be concerned about confirmation bias then shouldn't we check Tesla's confirmation bias?

Why didn't they start with V6.2 of the firmware that included all the safety components? Why start the data collection earlier than the point where all the safety features were enabled? The pre-AS had more miles logged than post-AS so it wasn't a question of miles.

Did Tesla actively research the causes of the accidents? Or did they just simply use logged data to determine if there was an airbag deployment over X number of miles. If they only did that then why aren't they routinely updating it? Where we can see how AP1, and AP2 compare? As someone else pointed out the AP1 data is of very little value to someone who is trying to gauge the overall safety of AP2.

My hypothesis is AP2 will surpass the safety of AP1 when AP2 has the neural net activated for the rear facing side camera's. That way it will improve the side/blind spot monitoring to a point where the crash avoidance can work better.

 

[S4WRXTTCS]

On what basis? during most of that period, for most of those cars, they already had AEB before and after AS.

There are three periods of car deliveries:

1) 4 months of initial ramp up : December 2014 to March 2015: cars delivered with AP hardware, but limited software.
2) 8 months of strong delivery : March 2015 to late October 2015: AEB functionality enabled

3) ~ 12+ months of strong delivery October 2015 through unspecified time in 2016: cars delivered: AS functionality enabled

Only cars in period #1 didn't have AEB. -- it was only the first third of the time period and January and Feb of that period were very slow. Most of the miles traveled in the first bucket (#1 and #2) with the 40% higher rate were for after March 2015 when AEB was enabled.

Also they mention that the comparison was between miles for cars without AS "installation" and after installation. A small but significant number of drivers don't buy AS at first (but do get AEB and FCW) . So they are also contributing to the pool of miles that have AEB, and all its improvements, but not AS, and they are in the 40% higher airbag deployments.

With lots of confounding factors nothing will be proven. But some hypotheses will be more or less supported by the evidence. There is no support for the hypothesis that AEB improvements alone, available to those who didn't buy AS, explain why miles traveled with AS available result in 40% fewer deployments.

Tesla shouldn't have included people without AP enabled in the study at all.

The reason is TACC is not just a convenience feature, but it's also a safety feature. It allows people to set a reasonable following speed, and with in on there are two sets of eyes on the road. Plus it relaxes drivers so they drive less aggressively (something reported over, and over when ever ACC is brought up).

I don't challenge the idea that lane-steering/lane-keeping technologies do add a layer of safety. That's been shown on numerous studies, but I've never seen anything that suggests it's 40% especially over FCW/Lane-departure warning/AEB/TACC/blindspot monitoring/etc.

 

[Economite]

For instance including people who never bought AP yet had the AEB updates. About 25% of cars don't even include AP when sold yet have AEB. That population is in the first bucket with their improved AEB.

Right... Which is why I said that in Bucket 1 "most miles were driven with relatively earlier versions of other features." Some miles in Bucket 1 were driven with relatively later versions. By contrast, all miles in Bucket 2 were driven with the relatively later versions of the non-AS features.

Under my hypothesis, if the miles in Bucket 1 with later versions of software were excluded from that bucket, then the crashes per million miles figure for Bucket 1 would be higher. The crash statistic gap between Bucket 1 and Bucket 2 if this exclusion occurred would be even wider, but the gap would still be because of improvements to the other systems, not because of the availability of AS.

 

[MrAustraliaTax]

you really have to wonder how serious Tesla is about its instructions that "you must keep your hands on the steering wheel at all times."

Tesla have always marketed Autopilot as being far more capable than it has ever been, from the earliest days of AP1 and throughout the painful farce of AP2 development.

After sufficient accidents and near miss videos, they now say
"Hands on wheel" (wink wink - not really).

It's deeply unfortunate that the most comprehensive resource for finding the actual limitations of AP (1 and 2) is embedded dashcam videos here on TMC.

 

[Saghost]

It's deeply unfortunate that the most comprehensive resource for finding the actual limitations of AP (1 and 2) is embedded dashcam videos here on TMC.

It's a moving target, too. Autopilot is more capable this week then it was last week. (Literally, at least in my case - 2018.14 almost completely eliminated the exit lane swerve for freeway driving in the right lane.)

Tesla's continuing to improve AP is overall a great thing and important for the future, but combined with the lack of documentation it does mean it's hard to know which things to worry about.

 

[mongo]

It's a moving target, too. Autopilot is more capable this week then it was last week. (Literally, at least in my case - 2018.14 almost completely eliminated the exit lane swerve for freeway driving in the right lane.)

Tesla's continuing to improve AP is overall a great thing and important for the future, but combined with the lack of documentation it does mean it's hard to know which things to worry about.

The driver should worry about everything. The car is there to compensate for the driver, not to replace them (yet). If a situation occurs that makes the driver feel nervous, that is sign enough to act (or crank down the aggressiveness setting).

Example: you can have a flat at anytime, so you should always have a firm grip on the wheel. (not that flats are common).

 

[bhzmark]

But I've seen no data or fact that supports the idea you need for your hypothesis that the tweaks to AEB after October 2015 were more material and affected air bag deployments more than the massive change in the availability of auto steering.

@s4wrxcctx reminds me of the distinction btw AS and TACC. And if they came out at the same time, I think it is a fair point that it is the combination with TACC that might explain the decrease. But since AS cannot be used without TACC I think of them together.

My own anecdotal data point is that using AS has made my driving safer. I'm more inclined to just pick a lane and put AS on instead of finding the fastest lane and changing lanes more. Many people are doing that that could partially explain why as driving is safer.

Additionally if it's a given that people are going to text to some extent, then texting with AS on is bound to be a lot safer than texting without it.

Those consequences of using AS are much more likely to explain the decrease in air bag deployments than the mere later tweaks to AEB.

Under my hypothesis, if the miles in Bucket 1 with later versions of software were excluded from that bucket, then the crashes per million miles figure for Bucket 1 would be higher. The crash statistic gap between Bucket 1 and Bucket 2 if this exclusion occurred would be even wider, but the gap would still be because of improvements to the other systems, not because of the availability of AS.

 

[greghanssen]

Tesla have always marketed Autopilot as being far more capable than it has ever been, from the earliest days of AP1 and throughout the painful farce of AP2 development.

After sufficient accidents and near miss videos, they now say
"Hands on wheel" (wink wink - not really).

Just curious.. did AP1 have the same "must have hands on wheel at all times" byline? When did the message change? Just since the Florida accident?

 

[bhzmark]

Just curious.. did AP1 have the same "must have hands on wheel at all times" byline? When did the message change? Just since the Florida accident?

Yes it did. And the owner's manual from the very beginning when AP was available in Oct 2015 says it very prominently.

The main changes after the Florida crash are described in the nhtsa report --- I believe primarily more frequent hands on nags and eventually training the vision to recognize semi trailers across the road , which caused low overpasses to generate false positives.

 

[bhzmark]

miles were driven with relatively earlier versions of other features.

TACC failed to brake at stop, nearly accident.

And aeb was working quite well in it's early versions as reported with many anecdotes in this forum. Despite the thread title aeb stopped the car from a collision that the driver was steering towards. He is just complaining that tacc didn't anticipate that he would not follow the Prius so tacc didn't apply a more gentle braking. Instead aeb stopped the car abruptly -- Back in April 2015, 7 months prior to AS. Good thing aeb was so good, long before AS.