TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker or making a Paypal contribution here: paypal.me/SupportTMC

Autopilot data growing 1.5M miles/day. Tesla knows by now if it's safer than a human

Discussion in 'Model S' started by calisnow, Jan 29, 2016.

  1. calisnow

    calisnow Active Member

    Joined:
    Oct 11, 2014
    Messages:
    1,128
    Location:
    Los Angeles
    #1 calisnow, Jan 29, 2016
    Last edited: Jan 30, 2016
    Screen Shot 2016-01-29 at 9.45.31 PM.png


    Despite widespread reports of undesirable lane biasing behavior in autopilot cars running 7.1, the question most pressing to me is a simple one: Does Autopilot in its current primitive beta stage result in fewer or more accidents - or neither?

    I did some napkin math and realized that the autopilot data set is growing at over 1.5 million miles per day at this point (extrapolated from fleet size, average miles driven per day, and guesses on percentage of miles driven using Autopilot and percentage of the fleet). The numbers make me think that the various threads and posts here (including mine) by wannabe safety experts claiming to be providing Musk & Co. with valuable insight seem a bit silly.

    As for the data above - you can argue with any one of the numbers but the end result is compelling regardless of the accuracy of any one number - the data set is huge and growing fast. Tesla must already know whether AP makes us safer or not. And if they don't know now - the data is growing so fast they will know very soon.


    • I roughly guesstimate that the customer fleet has put 40 million miles of hands off wheel (give or take 10 million or so) real world data back to Tesla HQ by now - and this data set is growing at a rate of over 1 to 2 million miles per day.
    • If Tesla wants real-time accident numbers they could easily have set up the cars to instantly tell them if the car has collided with an object using the on-board accelerometers.
    • They know when we are using Autopilot and when we aren't - and they also have all the accident data of the pre-autopilot years to compare.
    • There are plenty of available statistics on how frequently the average human driver collides with something - to cite one public source I read the typical driver gets in an accident 1 out of every 165,000 miles driven.
    • They could even be comparing collision rates of drivers who do and don't keep their hands on the wheel with a simple extrapolation - how much time it takes to sense torque on the wheel after a nag. Below some defined threshold the computer can assume this driver is driving with her hands on the wheel.

    With this large and growing fast data set they surely must have a statistician (excuse me "data scientist") at HQ slicing and dicing the collision rate data set in a dozen different ways looking for correlations of AP collision numbers with many different variables - and they probably have reached some fascinating conclusions.

    To name a few obvious collision rate comparison tests off the top of my head:

    • AP cars with AP turned on vs off
    • Collision rate using AP vs age of the driver (obtained via purchaser data)
    • Severity of impact with AP on vs AP OFF
    • They might even have data comparing accident frequency to nag frequency to see if nagging actually reduces accidents

    What will be fascinating is if they ever release this data to the public or use it to influence legislators in position to write regulations about autonomous driving.
     
  2. tmoz

    tmoz Member

    Joined:
    Aug 16, 2015
    Messages:
    78
    Location:
    Gilbert
    The data would be biased. People only engage AP when they think it would be safe to do so.
    So the 1.5 M miles per day tells us how safe AP is when it is allowed to operate in situations that the drive deems safe.
    Extrapolating past that point would not be justified.
     
  3. calisnow

    calisnow Active Member

    Joined:
    Oct 11, 2014
    Messages:
    1,128
    Location:
    Los Angeles
    #3 calisnow, Jan 29, 2016
    Last edited: Jan 29, 2016
    Very good observation, but all data is biased - you do the best you can. You got problems with induction? So do David Hume and Karl Popper. Get in line! :p :smile:

    Even if there is bias in the data because drivers use AP only in conditions they feel are safe - you would still be able to observe whether or not the collision rate of the Tesla fleet as a whole increased or decreased after October 25, 2015 and also compare the overall collision rates of AP-equipped cars vs cars which do not have the hardware.

    If the collision rate of AP equipped cars is lower to a statistically significant degree compared to the collision rate of Teslas with no autopilot then that is a huge win for autopilot.

    And at this point, with this many miles in the fleet - Tesla must know.
     
  4. CHG-ON

    CHG-ON Still in love after all these miles

    Joined:
    Jun 24, 2014
    Messages:
    2,754
    Location:
    Santa Cruz Mountains, USA
    You've got some pretty wild swings in your data and very generous WAGs too. But I like your thinking and believe that you are essentially on the right track. Of course, that may well be because I am biased. Thanks for doing the work on this. We will never know until an independent, non-interested study is done. Are there still any of those?

    I think that autonomous driving has a very long way to go until anybody knows if it is truly viable. I believe it it is. But that is just an uninformed opinion on my part, honestly. I could do all the study in the world and I would still not be able to come up with anything conclusive because it is simply too early to know. I want this this tech to move forward. But I hope that it moves forward carefully.

    BTW, I use AP and TACC every day on backcountry roads and it works surprisingly well, where we are technically not supposed to use it. 7.1 has brought a huge improvement on my roads. But not enough for me to take my eyes off the road. I probably use AP 50% and definately use TACC at least 75% of the time. TACC is, by far, the most useful feature. For me, it works 100% of the time and flawlessly. I had one issue shortly after it came out. None since.
     
  5. gordo

    gordo Member

    Joined:
    Jan 16, 2015
    Messages:
    213
    Location:
    CA
    #5 gordo, Jan 29, 2016
    Last edited: Jan 29, 2016
    OP, do you actually drive an AP car? I don't know what straw-man you think you're battling, but I drive an AP model S every day. And i can tell you that it biases right, dangerously at times. This needs to be fixed before autosteer gets out of beta.
     
  6. calisnow

    calisnow Active Member

    Joined:
    Oct 11, 2014
    Messages:
    1,128
    Location:
    Los Angeles
    I'm sure you're right that it does bias too much - there are too many people claiming it. I've only driven AP 7.0 for about 1,000 miles on a rental before I placed my order, which isn't here yet. And like the rest of you I hope it gets fixed before my car shows up!
     
  7. omarsultan

    omarsultan Active Member

    Joined:
    Jun 22, 2013
    Messages:
    1,496
    Location:
    Northern California
    Its seems you make the assumption that AP is passive when not engage and I am not sure this is true. If I were designing the system, I would have it (a) active and collecting data at all times and (b) when not engaged, still comparing its projected actions against actual driver actions, so its constantly learning. If that is the case then the argument could me made that the 157M mile number is the actual data set. Per part of your post, if I were Tesla, I might also load different parameters in different cars (like nag frequency) to see how it effected driver behavior and AP behavior.

    Regardless, I agree with your premise that the data set and huge and growing, but maybe not the need to chide other owners regarding their concerns, as AP, in its current state is hardly a perfect product.
     
  8. emchen

    emchen Member

    Joined:
    Aug 15, 2015
    Messages:
    91
    Location:
    Valencia, CA
    Bias to right side of the lane seems to have been solved by 7.1 software update in my car. In fact, I worry a little that it gets too close to the divider wall on the left when I'm in the HOV lane along I-5.
     
  9. flathillll

    flathillll Member

    Joined:
    Jul 21, 2015
    Messages:
    181
    Location:
    California
    autopilot is ALWAYS on

    it is in ghost mode when it is off

    that is why you had it a yr before it was turned "on"

    that is how the ner net learns

    once they found it was safer than a human driver

    then it was turned on

    nit before
     
  10. calisnow

    calisnow Active Member

    Joined:
    Oct 11, 2014
    Messages:
    1,128
    Location:
    Los Angeles
    Good points all around - I am assuming that the system is in fact collecting data even when it's not engaged. I guess we don't know that but I agree with you - that's how I would design it. And of course it is far from perfect.

    - - - Updated - - -

    Fair enough - snarkiness edited out now. :)
     
  11. cantdecide

    cantdecide Member

    Joined:
    Dec 21, 2012
    Messages:
    271
    Location:
    Portola Valley, ca
    They know if human+AP is safer under certain driving conditions than humans alone yes.
    They don't know what would have happened with AP alone under conditions where the human took control because the human takes at least slightly different action to AP.
     
  12. cytranic

    cytranic Member

    Joined:
    Aug 14, 2015
    Messages:
    74
    Location:
    FL
    I"m calling BS on the whole fleet learning thing. I dont think they have any idea how autopilot is working across cars.
     
  13. mkjayakumar

    mkjayakumar Active Member

    Joined:
    Aug 18, 2012
    Messages:
    1,726
    Location:
    Plano, TX
  14. JeffS

    JeffS Member

    Joined:
    Oct 7, 2015
    Messages:
    239
    Location:
    Wisconsin
    We know data is being collected. We know Tesla is playing with it. We know they push updates back to us. My guess...that's it for fleet learning.

    We also know AP is always collecting data, so any mile drive. Is a helpful mile.

    I think we are dramatically overestimating the miles per day though. There are maybe 50,000 VINs driving around with AP hardware in place. And not all of those VINs have the option enabled. 1.5MM miles across, say, 40,000 active cars would be shy of 40miles per day on AP. I don't do that. I would say I average maybe 40-50 miles on AP a week.

    Just my thought.
     
  15. omarsultan

    omarsultan Active Member

    Joined:
    Jun 22, 2013
    Messages:
    1,496
    Location:
    Northern California
    Based on what?
     
  16. ohmman

    ohmman Maximum Plaid Member

    Joined:
    Feb 13, 2014
    Messages:
    4,785
    Location:
    North Bay, CA
    omarsultan got it right. Elon himself has said it, and it's a classic supervised learning problem. AP learns while it's off and the human is driving. Elon called the drivers "expert trainers," which is precisely what they are when they're driving with AP off. I've posted in too many threads about this already, so I won't go into it too much further. If indeed AP learns while it's on, that's a reinforcement learning algorithm, which would be piled on top of the supervised learning - but that's unlikely because it's not nearly as powerful.

    I still think people shouldn't be so quick to assume that the data are being used yet. I believe it's being collected because that's what Tesla said. However, nobody has been able to capture a model being pushed OTA to the vehicle when driving or parked. My belief is that those are bundled with firmware updates.
     
  17. AMPUP

    AMPUP Member

    Joined:
    Mar 27, 2014
    Messages:
    467
    Location:
    Dunn Loring, VA
    Have to admit I somewhat agree with you, the vehicle can't even tell you if a supercharger stall is occupied. This talk of AI and the vehicle learning based on how you drive, is I fear, not reality right now.
     
  18. calisnow

    calisnow Active Member

    Joined:
    Oct 11, 2014
    Messages:
    1,128
    Location:
    Los Angeles
    You're probably right, and I don't think you were addressing my post but to be clear I wasn't making any claims about fleet learning - just making the observation that if Tesla wants to they should be able to observe accident rates.
     
  19. Saghost

    Saghost Active Member

    Joined:
    Oct 9, 2013
    Messages:
    2,949
    Location:
    Delaware
    That 6.1 accident per million miles is an overall average, and therefore only valid for a typical driving profile. Since AP miles are mostly freeway or country road miles, I don't think it's reasonable to expect 6.1 per million - those are generally much less likely places to have an accident than city streets or neighborhoods.

    But I would be interested to see the data, especially if someone was able to dig up a better human reference to compare it to. :)
    Walter
     
  20. tmoz

    tmoz Member

    Joined:
    Aug 16, 2015
    Messages:
    78
    Location:
    Gilbert
    Mine still biases right. The other thing is how many times have people had to yank control away from AP to avoid a crash? I feel like I've had to do it several times.
    Sometimes I forget that I have turned AP by tweaking the steering wheel, but then I get rudely reminded when the car gets too close to the other lane and I have to steer. I hope that doesn't lead to a crash some day! At the same time, I also feel that the car follows the road surprisingly well with AP off, making me think there might be a little something going on even while it is off.

    I still think it will be very hard to get safety data from measuring before AP 7.1 was released and now with 7.1 released. You sure can't compare crash rates with and with AP engaged as the types of roads will be different. One could compare same stretches of road when one set of drivers engages AP and the other group doesn't. AP may be safer on roads where the max speed has been limited to +5 mph over the posted limit. But then that data would have to be normalized by those involved in crashes while going +5 over the limit without AP engaged.

    Plenty of statistics to play with, just have to be careful how you play.
     

Share This Page