Airbag Deployment Increased 59% after AP deployment (Fact Checking NHTSA)

[Bladerskb]

Wow. Its crazy how the NHTSA tried to withhold info from the public all this time.

http://www.safetyresearch.net/Library/NHTSA_Autosteer_Safety_Claim.pdf

New Analysis Challenges Bold Tesla Claims | Safety Research & Strategies, Inc.


"Remarkably, NHTSA’s announcement was not accompanied by any of the data underlying this astonishing claim. NHTSA failed even to cite the numerators and denominators of the crash rates to back up its analysis."

"To replicate and better understand NHTSA’s study, we filed a Freedom of Information Act request on February 24, 2017 for “all of the mileage and airbag deployment data supplied by Tesla analyzed by ODI to calculate the crash rates shown in Figure 11.. In addition, we request[ed] all records related to any statistical summaries, formulas, models, adjustments, sample weights, and/or any other data or methods relied upon to calculate the crash rates shown in Figure 11.”

NHTSA responded by letter dated March 31, 2017, stating that “[t]he agency expects to provide a response by April 14, 2017.”10 In fact, we never heard from the Agency again until we sued the Department of Transportation on June 28, 2017 to obtain the requested data."

 

[OPRCE]

Wow. Its crazy how the NHTSA tried to withhold info from the public all this time.

http://www.safetyresearch.net/Library/NHTSA_Autosteer_Safety_Claim.pdf

New Analysis Challenges Bold Tesla Claims | Safety Research & Strategies, Inc.


"Remarkably, NHTSA’s announcement was not accompanied by any of the data underlying this astonishing claim. NHTSA failed even to cite the numerators and denominators of the crash rates to back up its analysis."

"To replicate and better understand NHTSA’s study, we filed a Freedom of Information Act request on February 24, 2017 for “all of the mileage and airbag deployment data supplied by Tesla analyzed by ODI to calculate the crash rates shown in Figure 11.. In addition, we request[ed] all records related to any statistical summaries, formulas, models, adjustments, sample weights, and/or any other data or methods relied upon to calculate the crash rates shown in Figure 11.”

NHTSA responded by letter dated March 31, 2017, stating that “[t]he agency expects to provide a response by April 14, 2017.”10 In fact, we never heard from the Agency again until we sued the Department of Transportation on June 28, 2017 to obtain the requested data."

Excellent work there by SRS in exploding their little game. Sadly it is no surprise to see confirmation that Tesla is still doing its best to evade an accurate analysis of its very jugged statistics for AP, which I have argued produces more severe accidents per mile than the average driver on the same roads: When can we read a book?

 

[S4WRXTTCS]

I agree with what the safety research article says about the NHTSA. They have become an agency that surrenders it's oversight in favor of industry guidance.

But, the statistics they calculated are no more accurate than what the NHTSA provided.

They just traded one bad methodology for another. The flaw in their methodology is their sample data is so small that noise is way more pronounced. Plus Auto-steer was released in Oct, and so the driving conditions are different in the before and after auto-steer. People who have ADAS features tend to use them to help out during bad conditions. This can lead to people doing things that they wouldn't normally do. I know I drove a lot more during the winter because I felt safer with my AWD Model S.

The other problem with the study is the technology was changing throughout the entire duration of the sample length. Autopilot wasn't released as a fully functioning piece of SW that just had auto-steer added to it. No, it was a series of releases that added/changed various ADAS features (AEB, FCW, LDW, TACC, AutoSteeer, Collision Avoidance, etc).

You also had changing driving behavior. where you had people getting more comfortable with TACC as time went on. TACC is both a something that can add a lot of safety (second set of eyes, increased following distances, less road rage, etc), and a something that can be easily misunderstood. At the time I don't think most drivers knew it couldn't always see stopped objects, and it was really only because of media coverage and experiences of others drivers that educated them.

I do think it's important to point out that the data collected for the time isn't relevant anymore so it can't be used as anything more than of historical curiosity. Back then AP allowed a kind of freedom that simply isn't allowed anymore, and technology used in HW2+ is mostly different.

 

[mongo]

Wow. Its crazy how the NHTSA tried to withhold info from the public all this time.

Yeah, area 51, am I right?

There's been some discussion on reddit already: New Analysis Challenges Bold Tesla Claims - for the vehicles in which all of the relevant data are known, Whitfield found that the airbag deployment crash rate increased by 59 percent after Autosteer technology was added : teslamotors

Here's a partial debunking by u/tracksyde:

So basically, this guy tossed out all the data for the cars where the starting mileage was not reported (miles on the car when Autosteer was installed). The claim is that the NHTSA assumed "miles-at-time-of-installation" for the cars without complete mileage data was zero.

The sample data was based on model year 2014 through 2016 S and 2016 X vehicles.

Autopilot was first offered in October 2014.

It appears to me that it is quite possible that it is true that all the cars without complete mileage data should be treated as 0 miles since most EAP purchases are made at the time of purchase, not afterwards via OTA.

Even if its not zero, its definitely somewhere in-between. But for this study's author to toss out the other 39,000 or so vehicle samples and then base his claims on just that subset shows he's simply doing it to create an inflammatory headline.

 

[mongo]

I believe they are guns for hire who pretend to be a safety watch dog. For example, they do not reveal their sources of funding. There is no indication of fund solicitation -- yet they have to have funds in order to operate. Which means they have private clients, which means they have private/hidden motivations. Their paper talks plainly about how they filed an FOI and then sued NHTSA for the underlying data, but is not plain about how they reinterpreted it to reverse the NHTSA results.

For example, "In practice, we do not understand the nature of the mechanism or process that generates the mileage exposure gap but it is not random."

Or, they imply that Tesla provided NHTSA with incomplete data. It's really just a baseless insinuation, and the worst kind because they offer no factual basis for their claim (the insinuation that Tesla gave them bad data is not supported by their statement). Here's the quote:
The damning part is the last. It is equally true to rephrase it as "...suggests that Tesla's comprehensive data is even more detailed than it used to be." It is a non-sequitur and offers no insight into the data they used for analysis. They offer not a single reason as to why the data would be incomplete in reporting airbag deployment (which, overall, is the aim of their insinuation).

I don't have the time to read it carefully and critically enough to grasp the implications of what they leave unsaid, what they overlook, and what they reframe. But if it were as straightforward as they claim it would not be necessary to have such a convoluted description. In other words, it smells of baffling through bullshit, carefully spinning facts to reach the "right" results while using constant twists and turns to confound the reader so that what is missed is the trickery employed.

As a different (and much simpler) example of reaching the desired results, some years ago I read a piece that perpetuated one of the many lies about Mayan calendars and it did so by turning on a very particular point: it presented some math and gave a wrong answer. The presumption was clearly that the reader would follow along the presented logic but not verify that the math calculated to what was claimed. An easy way to work from facts to bullshit.

 

[mongo]

Agreed. These guys are likely being funded by some law firm suing Tesla or some special interest group trying to drag or discredit Tesla.

Toyota Critic Safety Research & Strategies Founder Admits Report Funding Came From Firms Suing Toyota » AutoGuide.com News

 

[Bladerskb]

Yeah, area 51, am I right?

That's completely wrong and not surprising seeing it comes from /r/teslamotors

I'm going to try to summarize the report to make it more understandable. Please feel free to offer corrections.

The data is available for examination here. The columns of interest are:

U - Total mileage on the car
AA - Odometer reading before enabling Autosteer
AB - Odometer reading after enabling Autosteer
AR - Airbag events before Autosteer
AS - Airbag events after Autosteer

Theoretically, columns AA and AB should be equal to each other. For some reason only 5,714 of 43,781 vehicles have the two values the same. The remaining vehicles have missing/different values between columns AA and AB. The NHTSA made assumptions on how to interpret those missing/different values and those assumptions appear to be sketchy.

The report identifies four classes of cars within the dataset:

1. Cars where columns AA and AB are equal to each other (5,714 out of 43,781 cars). For these cars, we have 100% complete information. If you then calculate the rate of airbag deployments for this class of cars, it appears that airbag deployments go up 59% after enabling Autosteer. Feel free to interpret that result however you wish.
   
   
2. Cars where column AA and AB are both missing (14,791 out of 43,781 cars). The NHTSA assumed that this meant that Autosteer was enabled straight from the factory, so they assigned the total mileage for each car to after Autosteer was enabled. However, the dataset actually shows that for these cars there were 3 airbag deployments before Autosteer was enabled, so the dataset itself shows that the assumption that Autosteer was enabled straight from the factory was incorrect. For this class of cars, an unknown amount of the mileage that was assigned to after Autosteer actually belongs to before Autosteer.
   
   
3. Cars where column AA is smaller than AB (8,881 out of 43,781 cars). For these cars, that means that there are unexplained miles between before Autosteer and after Autosteer. Those unexplained miles should theoretically be split between before Autosteer and after Autosteer, but the exact amount to split between the two is unknown. The NHTSA appears to assign those unexplained miles to neither before or after Autosteer. For this class of cars, the report notes the number of unexplained miles is more than 2.6 times the number of miles assigned to before Autosteer. This signal-to-noise ratio calls into question the usefulness of the data for these cars. Additionally, the report notes that the number of unexplained miles is skewed dramatically towards the cars that have a small number of miles before Autosteer, which indicates bias in any conclusion drawn from the data for this class of cars.
   
   
4. Cars where column AA is missing and column AB is non-zero (14,260 out of 43,781 cars). In this case, the odometer reading before Autosteer is unknown, but the odometer reading after Autosteer is known. In this case, the NHTSA assumes that the number of miles after Autosteer is "total mileage" minus the "odometer reading after enabling Autosteer". However, the NHTSA then assumes that the number of miles before Autosteer is 0. Each car in this class therefore gives mileage credit towards after Autosteer and no mileage credit towards before Autosteer, skewing the results in favor of after Autosteer.
   

Even if you ignore the conclusion drawn for the first class of cars, the later three classes of cars introduce biases that make the conclusion drawn by the NHTSA unsustainable.

 

[bhzmark]

Tesla's response is:

QCS’ analysis dismissed the data from all but 5,714 vehicles of the total 43,781 vehicles in the data set we provided to NHTSA back in 2016. And given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5% of the total mileage that Tesla vehicles have traveled to date, and about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.

NHTSA’s original report did not only indicate that “Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation”, but the agency also concluded that it “did not identify any defects in the design or performance of the AEB or Autopilot systems” nor did it find “any incidents in which the systems did not perform as designed” (Page 10, last paragraph and Page 11, last paragraph, respectively). They also found that “the potential for driver misuse was evaluated as part of Tesla’s design process and solutions were tested, validated, and incorporated into the wide release of the product” (Page 10, first paragraph).

Our own vehicle safety data for Q3 and Q4, which includes data from roughly two billion miles driven in Tesla vehicles, shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.​

q4 data is:

In the 4th quarter, we registered one accident for every 2.91 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot, we registered one accident for every 1.58 million miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 436,000 miles.*​

The full fake research firm report (but it appears not the data itself) is available on scribd link here: Claim that Tesla’s crash rate dropped with Autopilot contested, Tesla responds

Can anyone point to a specific section that has the flaw in the report from the fake research firm funded by tort lawyers?

 

[Bladerskb]

The full fake research firm report (but it appears not the data itself) is available on scribd link here: Claim that Tesla’s crash rate dropped with Autopilot contested, Tesla responds

Can anyone point to a specific section that has the flaw in the report from the fake research firm funded by tort lawyers?

Fake? If you didn't get your news from the bubble maybe you will know that facts don't lie. 1 + 1 is 2 no matter how much you try to dispute it last year. You should try getting your info from a real news agency rather than a Tesla fan site. NHTSA themselves knew the entire thing was BS and tried to distance themselves from it.

In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus
Bloomberg - Are you a robot?

But now NHTSA's full data set is available, and, if anything, it appears to contradict Musk's claims. The majority of the vehicles in the Tesla data set suffered from missing data or other problems that made it impossible to say whether the activation of Autosteer increased or decreased the crash rate. But when QCS focused on 5,714 vehicles whose data didn't suffer from these problems, it found that the activation of Autosteer actually increased crash rates by 59 percent.

In some ways this is old news. NHTSA distanced itself from its own findings last May, describing them as a "cursory comparison" that "did not assess the effectiveness" of Autosteer technology. Moreover, the NHTSA report focused on version 1 of the Autopilot hardware, which Tesla hasn't sold since 2016.

The data that the NHTSA used to reach that conclusion was incomplete and sometimes ignored by the agency in its analysis, according to Quality Control Systems. It attempted to re-create the NHTSA’s analysis using the same data used by the agency, which the agency provided following a public-records lawsuit filed by Quality Control Systems. The company has criticized the NHTSA over auto safety issues in the past.

“Our replication of NHTSA’s analysis of the underlying data shows that the agency’s conclusion is not well-founded,” Randy Whitfield, president of Quality Control Systems, wrote in the report.

In a statement, the NHTSA said it is “reviewing the report released by Quality Control Systems Corp. with interest and will provide comment as appropriate.”

The NHTSA has come under under pressure to better explain how it made its finding. Last May, the agency clarified its initial observation, saying the “cursory” comparison of crash rates did not assess the effectiveness of Autosteer or whether crashes occurred while the system was engaged.

 

[banned-66611]

Tesla's response is:

QCS’ analysis dismissed the data from all but 5,714 vehicles of the total 43,781 vehicles in the data set we provided to NHTSA back in 2016. And given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5% of the total mileage that Tesla vehicles have traveled to date, and about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.

NHTSA’s original report did not only indicate that “Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation”, but the agency also concluded that it “did not identify any defects in the design or performance of the AEB or Autopilot systems” nor did it find “any incidents in which the systems did not perform as designed” (Page 10, last paragraph and Page 11, last paragraph, respectively). They also found that “the potential for driver misuse was evaluated as part of Tesla’s design process and solutions were tested, validated, and incorporated into the wide release of the product” (Page 10, first paragraph).

Our own vehicle safety data for Q3 and Q4, which includes data from roughly two billion miles driven in Tesla vehicles, shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.​

q4 data is:

In the 4th quarter, we registered one accident for every 2.91 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot, we registered one accident for every 1.58 million miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 436,000 miles.*​

The full fake research firm report (but it appears not the data itself) is available on scribd link here: Claim that Tesla’s crash rate dropped with Autopilot contested, Tesla responds

Can anyone point to a specific section that has the flaw in the report from the fake research firm funded by tort lawyers?

Their response has been debunked too. For example, they claim that people using AP were less likely to have an accident than all other drivers. That ignores the fact that AP is for highway use only, and highways have lower accident rates anyway. It ignores that Teslas are newer cars, and more expensive cars with richer owners, and old car/cheap car/low income owner all correlate with higher accident rates.

The stuff about the sample size is pure misdirection. The sample size is plenty big enough for statistically valid results. The fact that there are more Teslas out there now is irrelevant. The fact that there have been software updates is irrelevant, especially since those "updates" initially made AP worse and didn't change it's fundamental behaviour.

 

[Inside]

Having AEB reduces accidents V.S. not having it. That's all there is to it. AEB rolled out at the safe time as AutoPilot and is the only reason accidents were reduced (if they even were). Don't let Tesla confuse the issue to you.

AutoPilot (the self-steering feature) is a huge convenience, but using it makes you less-safe, not more-safe. If AutoPilot was safer than the typical human then they wouldn't insist that you keep your hands on the wheel, if fact they'd insist on the opposite. Having your hands on the wheel would be less safe than having the computer do the steering (assuming AutoPilot was better than typical human driving). I'm super impressed with the current build of AutoPilot and by what it can do and what it can handle, but it is a *long* way off from being a better driver than I am. Using it does not make me feel more safe.

 

[Kant.Ing]

Not to mention when sensing a collision or an airbag deployment is unavoidable, as responsible Tesla owners and drivers, we would spontaneously disengage AP and never blame on it. See who are cooking the numbers big time.

 

[SupersonicP3D]

All of these statistics are bunk and no one is doing analysis properly. Tesla is mixing highway/city. NHTSA looks like is just making up a bunch of numbers. And this research report is not much better. Airbag events often total a car (and/or cause the owner to sell it). The second owner of these cars has a different driving style, less money to purchase autopilot, or the car may be scrapped altogether. It's plausible to argue that the cars that had autopilot switched on later in their lives must have "survived" the period before autopilot. Plus we know that autopilot miles are only 10% of total fleet miles (though the vast majority of owners purchase the feature), meaning that the vast majority of the miles traveled "after autopilot" were hand-driven.

Until we get a real apples to apple comparison (highway speeds, miles driven engaged vs. disengaged, same times of day, same weather conditions, same seasons, same stretch of roads), it'll be really hard to draw any real conclusions.

 

[bhzmark]

Fake? If you didn't get your news from the bubble maybe you will know that facts don't lie. 1 + 1 is 2 no matter how much you try to dispute it last year. You should try getting your info from a real news agency rather than a Tesla fan site. NHTSA themselves knew the entire thing was BS and tried to distance themselves from it.

In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus
Bloomberg - Are you a robot?

Quality Control Systems (QCS) Corp. is a fake firm. It is one guy: Randy Whitfield a guy out of his house, whose main occupation is serving as an expert witness for plaintiff law firms trying to extort money from car mfrs for some real (SUV turn overs, but he even lost some of those cases), but mostly fake (unintended acceleration), defects in cars.

There is nothing specific or persuasive in his report. He gives no reason why he culled the data so dramatically. It is much less persuasive than the NHTSA report.

I challenge you to quote a specific detail from his fake report and defend it showing why his data conclusion is more relevant the NHTSA report or the current data that Tesla reports quarterly.

 

[Bladerskb]

Quality Control Systems (QCS) Corp. is a fake firm. It is one guy: Randy Whitfield a guy out of his house, whose main occupation is serving as an expert witness for plaintiff law firms trying to extort money from car mfrs for some real (SUV turn overs, but he even lost some of those cases), but mostly fake (unintended acceleration), defects in cars.

There is nothing specific or persuasive in his report. He gives no reason why he culled the data so dramatically. It is much less persuasive than the NHTSA report.

I challenge you to quote a specific detail from his fake report and defend it showing why his data conclusion is more relevant the NHTSA report or the current data that Tesla reports quarterly.

I dont have to, it's already been done.
In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus

Dont let your love for Tesla cloud your judgment.

 

[S4WRXTTCS]

Quality Control Systems (QCS) Corp. is a fake firm. It is one guy: Randy Whitfield a guy out of his house, whose main occupation is serving as an expert witness for plaintiff law firms trying to extort money from car mfrs for some real (SUV turn overs, but he even lost some of those cases), but mostly fake (unintended acceleration), defects in cars.

There is nothing specific or persuasive in his report. He gives no reason why he culled the data so dramatically. It is much less persuasive than the NHTSA report.

I challenge you to quote a specific detail from his fake report and defend it showing why his data conclusion is more relevant the NHTSA report or the current data that Tesla reports quarterly.

The way I see it is we have this data, and this data is available to anyone to analyze however they see fit.

The problem is only the NHTSA had a direct line to Tesla to asks questions about the data provided.

I honestly can't figure out why the NHTSA did some of what they did. As an example they counted 18 crashes under the pre-autosteer column for vehicles that had ZERO pre-autosteer miles.

So I think there is some confusion about why things were counted the way they did. I can't make heads or tails of what the NHTSA came up with, and they're supposed to know what they're doing.

I got the figure of the 18 vehicles from this article (same article that Blader linked to). In the comments section someone posted a link to the data.

In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus

Can anyone make sense of the what the NHTSA did, and why they counted things in the columns they did?

 

[ZeApelido]

Where is the raw data? I'll parse it.

 

[S4WRXTTCS]

Where is the raw data? I'll parse it.

http://www.safetyresearch.net/Library/2018-11-26%20Redacted%20PE16_007_PRODUCTION%20DATA_jlq_working_file_10Jan2017%20.xlsx.

It's in this comment section.

In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus

Too much redacted data to be of any substantial value though.

 

[bhzmark]

Any update on this?

 

[banned-66611]

What kind of update? Tesla won't release any useful data so there is no way to know how dangerous or otherwise Autopilot is.