TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC

Airbag Deployment Increased 59% after AP deployment (Fact Checking NHTSA)

Discussion in 'Autonomous Vehicles' started by Bladerskb, Feb 9, 2019.

  1. Bladerskb

    Bladerskb Senior Software Engineer

    Joined:
    Oct 24, 2016
    Messages:
    1,338
    Location:
    Michigan
    #1 Bladerskb, Feb 9, 2019
    Last edited: Feb 9, 2019
    Wow. Its crazy how the NHTSA tried to withhold info from the public all this time.

    http://www.safetyresearch.net/Library/NHTSA_Autosteer_Safety_Claim.pdf

    New Analysis Challenges Bold Tesla Claims | Safety Research & Strategies, Inc.


    "Remarkably, NHTSA’s announcement was not accompanied by any of the data underlying this astonishing claim. NHTSA failed even to cite the numerators and denominators of the crash rates to back up its analysis."

    "To replicate and better understand NHTSA’s study, we filed a Freedom of Information Act request on February 24, 2017 for “all of the mileage and airbag deployment data supplied by Tesla analyzed by ODI to calculate the crash rates shown in Figure 11.. In addition, we request[ed] all records related to any statistical summaries, formulas, models, adjustments, sample weights, and/or any other data or methods relied upon to calculate the crash rates shown in Figure 11.”

    NHTSA responded by letter dated March 31, 2017, stating that “[t]he agency expects to provide a response by April 14, 2017.”10 In fact, we never heard from the Agency again until we sued the Department of Transportation on June 28, 2017 to obtain the requested data."
     
    • Informative x 2
    • Like x 1
    • Funny x 1
  2. OPRCE

    OPRCE Member

    Joined:
    Jun 16, 2018
    Messages:
    324
    Location:
    Bern, Switzerland
    Excellent work there by SRS in exploding their little game. Sadly it is no surprise to see confirmation that Tesla is still doing its best to evade an accurate analysis of its very jugged statistics for AP, which I have argued produces more severe accidents per mile than the average driver on the same roads: When can we read a book?
     
    • Like x 3
    • Funny x 1
  3. S4WRXTTCS

    S4WRXTTCS Active Member

    Joined:
    May 3, 2015
    Messages:
    3,437
    Location:
    Snohomish, WA
    I agree with what the safety research article says about the NHTSA. They have become an agency that surrenders it's oversight in favor of industry guidance.

    But, the statistics they calculated are no more accurate than what the NHTSA provided.

    They just traded one bad methodology for another. The flaw in their methodology is their sample data is so small that noise is way more pronounced. Plus Auto-steer was released in Oct, and so the driving conditions are different in the before and after auto-steer. People who have ADAS features tend to use them to help out during bad conditions. This can lead to people doing things that they wouldn't normally do. I know I drove a lot more during the winter because I felt safer with my AWD Model S.

    The other problem with the study is the technology was changing throughout the entire duration of the sample length. Autopilot wasn't released as a fully functioning piece of SW that just had auto-steer added to it. No, it was a series of releases that added/changed various ADAS features (AEB, FCW, LDW, TACC, AutoSteeer, Collision Avoidance, etc).

    You also had changing driving behavior. where you had people getting more comfortable with TACC as time went on. TACC is both a something that can add a lot of safety (second set of eyes, increased following distances, less road rage, etc), and a something that can be easily misunderstood. At the time I don't think most drivers knew it couldn't always see stopped objects, and it was really only because of media coverage and experiences of others drivers that educated them.

    I do think it's important to point out that the data collected for the time isn't relevant anymore so it can't be used as anything more than of historical curiosity. Back then AP allowed a kind of freedom that simply isn't allowed anymore, and technology used in HW2+ is mostly different.
     
    • Helpful x 1
  4. mongo

    mongo Well-Known Member

    Joined:
    May 3, 2017
    Messages:
    7,400
    Location:
    Michigan
    Yeah, area 51, am I right?

     
  5. mongo

    mongo Well-Known Member

    Joined:
    May 3, 2017
    Messages:
    7,400
    Location:
    Michigan
     
  6. mongo

    mongo Well-Known Member

    Joined:
    May 3, 2017
    Messages:
    7,400
    Location:
    Michigan
  7. Bladerskb

    Bladerskb Senior Software Engineer

    Joined:
    Oct 24, 2016
    Messages:
    1,338
    Location:
    Michigan
    That's completely wrong and not surprising seeing it comes from /r/teslamotors

     
  8. bhzmark

    bhzmark Supporting Member

    Joined:
    Jul 21, 2013
    Messages:
    2,602
    Tesla's response is:

    QCS’ analysis dismissed the data from all but 5,714 vehicles of the total 43,781 vehicles in the data set we provided to NHTSA back in 2016. And given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5% of the total mileage that Tesla vehicles have traveled to date, and about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.

    NHTSA’s original report did not only indicate that “Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation”, but the agency also concluded that it “did not identify any defects in the design or performance of the AEB or Autopilot systems” nor did it find “any incidents in which the systems did not perform as designed” (Page 10, last paragraph and Page 11, last paragraph, respectively). They also found that “the potential for driver misuse was evaluated as part of Tesla’s design process and solutions were tested, validated, and incorporated into the wide release of the product” (Page 10, first paragraph).

    Our own vehicle safety data for Q3 and Q4, which includes data from roughly two billion miles driven in Tesla vehicles, shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.​

    q4 data is:

    In the 4th quarter, we registered one accident for every 2.91 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot, we registered one accident for every 1.58 million miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 436,000 miles.*​

    The full fake research firm report (but it appears not the data itself) is available on scribd link here: Claim that Tesla’s crash rate dropped with Autopilot contested, Tesla responds

    Can anyone point to a specific section that has the flaw in the report from the fake research firm funded by tort lawyers?
     
    • Like x 2
  9. Bladerskb

    Bladerskb Senior Software Engineer

    Joined:
    Oct 24, 2016
    Messages:
    1,338
    Location:
    Michigan

    Fake? If you didn't get your news from the bubble maybe you will know that facts don't lie. 1 + 1 is 2 no matter how much you try to dispute it last year. You should try getting your info from a real news agency rather than a Tesla fan site. NHTSA themselves knew the entire thing was BS and tried to distance themselves from it.

    In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus
    Bloomberg - Are you a robot?

     
    • Funny x 1
  10. Kuro68k

    Kuro68k Member

    Joined:
    Oct 10, 2017
    Messages:
    554
    Location:
    Roaming
    Their response has been debunked too. For example, they claim that people using AP were less likely to have an accident than all other drivers. That ignores the fact that AP is for highway use only, and highways have lower accident rates anyway. It ignores that Teslas are newer cars, and more expensive cars with richer owners, and old car/cheap car/low income owner all correlate with higher accident rates.

    The stuff about the sample size is pure misdirection. The sample size is plenty big enough for statistically valid results. The fact that there are more Teslas out there now is irrelevant. The fact that there have been software updates is irrelevant, especially since those "updates" initially made AP worse and didn't change it's fundamental behaviour.
     
    • Like x 2
  11. Inside

    Inside Member

    Joined:
    May 10, 2017
    Messages:
    33
    Location:
    United States
    Having AEB reduces accidents V.S. not having it. That's all there is to it. AEB rolled out at the safe time as AutoPilot and is the only reason accidents were reduced (if they even were). Don't let Tesla confuse the issue to you.

    AutoPilot (the self-steering feature) is a huge convenience, but using it makes you less-safe, not more-safe. If AutoPilot was safer than the typical human then they wouldn't insist that you keep your hands on the wheel, if fact they'd insist on the opposite. Having your hands on the wheel would be less safe than having the computer do the steering (assuming AutoPilot was better than typical human driving). I'm super impressed with the current build of AutoPilot and by what it can do and what it can handle, but it is a *long* way off from being a better driver than I am. Using it does not make me feel more safe.
     
  12. Kanting

    Kanting Member

    Joined:
    Apr 21, 2016
    Messages:
    504
    Location:
    California, CA
    Not to mention when sensing a collision or an airbag deployment is unavoidable, as responsible Tesla owners and drivers, we would spontaneously disengage AP and never blame on it. See who are cooking the numbers big time.
     
  13. SupersonicP3D

    SupersonicP3D Member

    Joined:
    Aug 6, 2018
    Messages:
    17
    Location:
    San Francisco, CA
    All of these statistics are bunk and no one is doing analysis properly. Tesla is mixing highway/city. NHTSA looks like is just making up a bunch of numbers. And this research report is not much better. Airbag events often total a car (and/or cause the owner to sell it). The second owner of these cars has a different driving style, less money to purchase autopilot, or the car may be scrapped altogether. It's plausible to argue that the cars that had autopilot switched on later in their lives must have "survived" the period before autopilot. Plus we know that autopilot miles are only 10% of total fleet miles (though the vast majority of owners purchase the feature), meaning that the vast majority of the miles traveled "after autopilot" were hand-driven.

    Until we get a real apples to apple comparison (highway speeds, miles driven engaged vs. disengaged, same times of day, same weather conditions, same seasons, same stretch of roads), it'll be really hard to draw any real conclusions.
     
  14. bhzmark

    bhzmark Supporting Member

    Joined:
    Jul 21, 2013
    Messages:
    2,602

    Quality Control Systems (QCS) Corp. is a fake firm. It is one guy: Randy Whitfield a guy out of his house, whose main occupation is serving as an expert witness for plaintiff law firms trying to extort money from car mfrs for some real (SUV turn overs, but he even lost some of those cases), but mostly fake (unintended acceleration), defects in cars.

    There is nothing specific or persuasive in his report. He gives no reason why he culled the data so dramatically. It is much less persuasive than the NHTSA report.

    I challenge you to quote a specific detail from his fake report and defend it showing why his data conclusion is more relevant the NHTSA report or the current data that Tesla reports quarterly.
     
  15. Bladerskb

    Bladerskb Senior Software Engineer

    Joined:
    Oct 24, 2016
    Messages:
    1,338
    Location:
    Michigan
    I dont have to, it's already been done.
    In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus

    Dont let your love for Tesla cloud your judgment.
     
    • Like x 1
  16. S4WRXTTCS

    S4WRXTTCS Active Member

    Joined:
    May 3, 2015
    Messages:
    3,437
    Location:
    Snohomish, WA
    The way I see it is we have this data, and this data is available to anyone to analyze however they see fit.

    The problem is only the NHTSA had a direct line to Tesla to asks questions about the data provided.

    I honestly can't figure out why the NHTSA did some of what they did. As an example they counted 18 crashes under the pre-autosteer column for vehicles that had ZERO pre-autosteer miles.

    So I think there is some confusion about why things were counted the way they did. I can't make heads or tails of what the NHTSA came up with, and they're supposed to know what they're doing.

    I got the figure of the 18 vehicles from this article (same article that Blader linked to). In the comments section someone posted a link to the data.

    In 2017, the feds said Tesla Autopilot cut crashes 40%—that was bogus

    Can anyone make sense of the what the NHTSA did, and why they counted things in the columns they did?
     
  17. ZeApelido

    ZeApelido Member

    Joined:
    Jun 1, 2016
    Messages:
    495
    Location:
    The Peninsula, CA
    Where is the raw data? I'll parse it.
     
  18. S4WRXTTCS

    S4WRXTTCS Active Member

    Joined:
    May 3, 2015
    Messages:
    3,437
    Location:
    Snohomish, WA
    #18 S4WRXTTCS, Feb 15, 2019 at 6:58 PM
    Last edited: Feb 15, 2019 at 7:03 PM
    • Informative x 1

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC