Open Letter to Elon Musk

[imispgh]

My thought: is the hardware failure rate of a single computer higher or lower than the failure rate of a human brain? At what rate do humans fall asleep at the wheel, get dangerously distracted, crash, etc. versus the rate that computers die?

More redundancy is always better. Two redundant computers is better than one, ten computers is better than two, fifty is better than ten, a hundred is better than fifty, a thousand is better than a hundred... But how do we decide how much redundancy is enough? I think human driving has to be the benchmark.

Human driving is not a safe default option: it kills as many people as HIV/AIDS. Autonomous driving only needs to be, say, 20% safer than that to be safe enough to deploy, in my opinion. Tesla is aiming for at least 2x safer (ideally 10x safer) and will be collecting billions of miles of shadow mode data to confirm that statistically before it deploys fully autonomous driving.

Related: Tesla mentioned that in Hardware 2.5 there is some computing redundancy added, but I am not aware of the details beyond what I just said.

My issue is not with the need for this tech. I support it and Elon's desire to attain it. The issue is the method used to achieve it needs to be effective and not risk lives needlessly. Tesla's approach does not meet that bar. Waz said this and so has John Krafcik recently.

I am now part of the SAE V&V Task Force. They would not have extended that invitation for no reason. Also please read the articles on Waymo's recent paradigm shift on this.

 

[strangecosmos]

Michael, you raised multiple issues, one of which was the need for Tesla to use aerospace-level simulation. How do you feel about Tesla’s simulation efforts, both pure computer simulation and “shadow mode” simulation?

The other issue is Tesla rolling out advanced driver assistance features as part of Autopilot (HW1) and Enhanced Autopilot (HW2). Are you aware that NHTSA conducted a statistical study and found that Autopilot reduced the rate of crashes by 40%? Autopilot has been found to improve safety.

Tesla has recently said that before they roll out a new Enhanced Autopilot feature, it is simulated in “shadow mode” for millions of miles in order to statistically validate its safety. Are you satisfied with this approach?

 

[imispgh]

Michael, you raised multiple issues, one of which was the need for Tesla to use aerospace-level simulation. How do you feel about Tesla’s simulation efforts, both pure computer simulation and “shadow mode” simulation?

The other issue is Tesla rolling out advanced driver assistance features as part of Autopilot (HW1) and Enhanced Autopilot (HW2). Are you aware that NHTSA conducted a statistical study and found that Autopilot reduced the rate of crashes by 40%? Autopilot has been found to improve safety.

Tesla has recently said that before they roll out a new Enhanced Autopilot feature, it is simulated in “shadow mode” for millions of miles in order to statistically validate its safety. Are you satisfied with this approach?

Trent
- The 40% number is unsubstantiated. Jalopnik submitted a FOIA for the data. NHTSA and Tesla refused to supply it.And AAA raised their rates due to increased accidents and cost. I agree that 40% will be true someday. I don't think Tesla is actually achieving that now.
- Elon said they have 2B miles of data. Assuming he is not exaggerating that like he does far too many other things, that indicates simulation was used. However 1 mile of sim use is = to many street miles. I think I saw MCity said 10M. If they had even 1.5B sim miles their capabilities should be much better by now. Also Elon said 6B miles are needed. Unless he knew and didn't say that those are sim miles then he is way off the one trillion public miles needed. And he is using handover/public shadow driving which should not be used for safety reasons and because it can never lead to L4. Why use an unsafe practice if you are using simulation correctly? Makes no sense. This is why Waymo just made the paradigm shift to no handover/L2/L3/public shadow driving and uses simulation instead.

I should note that public shadow driving is not necessarily Tesla's "shadow mode". I am good with hands of driving for AI. It's a necessity and is not dangerous. As soon as you let go of the wheel the game is up.

Tesla would be well served by making the shift Waymo has. They will never make L4 otherwise.

 

[strangecosmos]

Trent
- The 40% number is unsubstantiated. Jalopnik submitted a FOIA for the data. NHTSA and Tesla refused to supply it.

If I recall correctly, NHTSA didn't want to make the data public because it could not be anonymized. Do you not think NHTSA is trustworthy?

And AAA raised their rates due to increased accidents and cost.

If I recall correctly, this had to do with the cost of repairing the aluminum bodies of the Model S and Model X, not the actual rate of accidents.

- Elon said they have 2B miles of data. Assuming he is not exaggerating that like he does far too many other things, that indicates simulation was used.

Hardware 2 cars are currently driving about 4 million real world miles in "shadow mode" per day. Assuming Model 3 production ramps up as planned, these cars will accumulate 11 billions real world miles by 2019.

If they had even 1.5B sim miles their capabilities should be much better by now.

How do you know? Seems like this is just speculation.

Also Elon said 6B miles are needed. Unless he knew and didn't say that those are sim miles then he is way off the one trillion public miles needed.

As I said, assuming the Model 3 production ramp goes as planned, Tesla will rack up 11 billion "shadow mode" miles by 2019. 11 billion is a pertinent figure because it is what the RAND Corporation found could be necessarily to statistically validate the safety of self-driving cars. Where are you getting the 1 trillion miles figure?

At the rate Waymo is going, it will take its small fleet of test cars over 6,000 years to drive 11 billion real world miles. Unless Waymo greatly expands its test fleet, it will never be able to statistically show, using real world data, that its self-driving cars are actually safer than human drivers.

Tesla, on the other hand, will have 11 billion miles of real world data by 2019. In early 2019, its Hardware 2 cars will be driving approximately 1 billion miles per month.

Waymo does roughly 1,000 miles of simulation for every 1 real world mile. If Tesla maintained the same ratio, it would do 11 trillion miles of simulated driving by 2019

Real world miles are a constraint on simulated miles. You need data about real world events and environments to generate an accurate simulation.

As soon as you let go of the wheel the game is up.

Tesla's Level 2 autonomy software, Autopilot and Enhanced Autopilot, requires drivers to keep their hands on the wheel. There is a driver-facing camera installed in new Teslas that may be used in the future to ensure the driver is looking at the road. Tesla has not announced any plans for a Level 3 system. It seems like Tesla plans to skip from Level 2 to Level 4.

 

[imispgh]

If I recall correctly, NHTSA didn't want to make the data public because it could not be anonymized. Do you not think NHTSA is trustworthy?



If I recall correctly, this had to do with the cost of repairing the aluminum bodies of the Model S and Model X, not the actual rate of accidents.



Hardware 2 cars are currently driving about 4 million real world miles in "shadow mode" per day. Assuming Model 3 production ramps up as planned, these cars will accumulate 11 billions real world miles by 2019.



How do you know? Seems like this is just speculation.



As I said, assuming the Model 3 production ramp goes as planned, Tesla will rack up 11 billion "shadow mode" miles by 2019. 11 billion is a pertinent figure because it is what the RAND Corporation found could be necessarily to statistically validate the safety of self-driving cars. Where are you getting the 1 trillion miles figure?

At the rate Waymo is going, it will take its small fleet of test cars over 6,000 years to drive 11 billion real world miles. Unless Waymo greatly expands its test fleet, it will never be able to statistically show, using real world data, that its self-driving cars are actually safer than human drivers.

Tesla, on the other hand, will have 11 billion miles of real world data by 2019. In early 2019, its Hardware 2 cars will be driving approximately 1 billion miles per month.

Waymo does roughly 1,000 miles of simulation for every 1 real world mile. If Tesla maintained the same ratio, it would do 11 trillion miles of simulated driving by 2019

Real world miles are a constraint on simulated miles. You need data about real world events and environments to generate an accurate simulation.



Tesla's Level 2 autonomy software, Autopilot and Enhanced Autopilot, requires drivers to keep their hands on the wheel. There is a driver-facing camera installed in new Teslas that may be used in the future to ensure the driver is looking at the road. Tesla has not announced any plans for a Level 3 system. It seems like Tesla plans to skip from Level 2 to Level 4.

If Tesla really thought L3 should be skipped they would off no ability or modes to allow stearing control to be ceded. They do neither. That latest highway utilizes it as a feature.

One trillion is from Toyota. If you extrapolate Rands data to a 10x better than a human and account for it not being linear you get to 500B and up. I don't care if it's 250B etc. Not possible. And so dangerous it will never be permitted to continue when the dangerous scenarios start hurting people. (Rand stated in their paper this process was not doable). If you apply common sense you can see this makes ZERO sense. The reason I believe folks started off driving on roads is that they assumed simulation couldn't get the job done. They made that assumption looking at their own industry not aerospace. BTW Chris Urmson has some to the same realization. He adds it will take 30 years to get to full L4.

No I do not trust Elon anymore. I used to. Then he let his ego get the best of him. Just listen to how defensive he got on the earnings call. And that is just what he says publicly. As i said in my letter to him He is on the edge between being famous or infamous.

 

[fadedhope]

If Tesla really thought L3 should be skipped they would off no ability or modes to allow stearing control to be ceded. They do neither. That latest highway utilizes it as a feature.

One trillion is from Toyota. If you extrapolate Rands data to a 10x better than a human and account for it not being linear you get to 500B and up. I don't care if it's 250B etc. Not possible. And so dangerous it will never be permitted to continue when the dangerous scenarios start hurting people. (Rand stated in their paper this process was not doable). If you apply common sense you can see this makes ZERO sense. The reason I believe folks started off driving on roads is that they assumed simulation couldn't get the job done. They made that assumption looking at their own industry not aerospace. BTW Chris Urmson has some to the same realization. He adds it will take 30 years to get to full L4.

No I do not trust Elon anymore. I used to. Then he let his ego get the best of him. Just listen to how defensive he got on the earnings call. And that is just what he says publicly. As i said in my letter to him He is on the edge between being famous or infamous.

Exactly. It is scary what ego can do to a person. I totally anticipate a massive class action law suit when the buyers realize they will never get full autonomous driving in the next decade.

 

[WillK]

They made that assumption looking at their own industry not aerospace. BTW Chris Urmson has some to the same realization. He adds it will take 30 years to get to full L4.

They just announced Fully Autonomous Waymo’s self-driving cars are now fully autonomous, public ride service launching soon

30 years early.

 

[strangecosmos]

If Tesla really thought L3 should be skipped they would off no ability or modes to allow stearing control to be ceded. They do neither. That latest highway utilizes it as a feature.

Autopilot and Enhanced Autopilot requires the driver to keep their hands on the wheel. If they don't, after a few warnings the system shuts off and requires the driver to take full manual control.

We don't yet know for sure what the interior camera in new Teslas is for, but it may eventually be used to ensure drivers are paying attention.

One trillion is from Toyota.

I looked into this. If I'm not mistaken, this is simply the number of miles that all Toyota cars in the world drive each year. It wasn't the result of any kind of study, analysis, or calculation.

If you extrapolate Rands data to a 10x better than a human and account for it not being linear you get to 500B and up. I don't care if it's 250B etc. Not possible. And so dangerous it will never be permitted to continue when the dangerous scenarios start hurting people. (Rand stated in their paper this process was not doable).

RAND assumed that there would only be a fleet of 100 test cars. Based on Tesla's current production targets, we can estimate it will have 2 million Hardware 2 cars on the road by 2021.

If the Tesla Network launches at the beginning 2021 (based on the tens of billions of miles driven in shadow mode demonstrating a 20%+ safety improvement over human driving), and if those 2 million cars drive 5x as much as normal since they are shared among 5 people each, then Tesla's fleet will be driving 165 billion fully autonomous real world miles per year at the beginning of 2021.

Each subsequent month, the annual rate of miles driven will increase as more cars are produced. If 2 million cars are produced in 2021, the rate will be 330 billion miles per year at the beginning of 2022.

If self-driving cars drive 10x as many miles as human drivers rather than 5x, then these numbers will be 330 billion miles per year in 2021, and 660 billion miles per year in 2022.

The reason I believe folks started off driving on roads is that they assumed simulation couldn't get the job done.

Tesla, Waymo, and other companies in the field are making extensive use of simulation.

Waymo has done over 2.5 billion simulated miles so far. It's doing 8 million miles per day (a rate of almost 3 billion per year).

Since we know Tesla has a simulation operation used for the same purpose, and since Tesla has more real world data than Waymo to feed its simulation, it seems likely to me that Tesla does at least many simulated miles as Waymo. Possibly many more.

 

[imispgh]

Here is a study using Tesla to show human's cannot utilize L2 properly.
https://www.researchgate.net/public...a_bad_idea_Observations_from_an_on-road_study

"With this in mind the authors of this paper argue that a shift in attitude is required to ensure that the role of the driver within automated driving systems is protected. Tesla, along with other vehicle manufacturers, have designed vehicles that can essentially drive themselves most of the time but still require a human driver to monitor its performance and intervene when necessary. This design ethos has led to a situation in which humans are bound to fail and so “driver error” becomes an inevitable outcome (Stanton & Baber, 2002). Systems designers have created an impossible task (Stanton, 2015) – one that requires the driver to remain vigilant for extended periods. The literature openly reports that humans are poor at doing this (e.g. Molloy & Parasuraman, 1996). With this in mind, whilst strategies to help improve Level 2 systems could be explored, it seems more appropriate at this time to accept that the DD and DND roles are the only two viable options that can fully protect the role of the human within automated driving systems. This in turn means that either the human driver should remain in control of longitudinal and/or lateral aspects of control (i.e. one or the other) or they are removed entirely from the control-feedback loop (essentially moving straight to SAE 4)."

This report adds to others that purposefully distract drivers for a long period. They found it take 5-45 seconds to gain enough situational awareness time to make the right move.. And that monitoring and notification systems cannot make that reliably safe.

This report however is a bit scarier. They didn't ask or direct people to be distracted and over trust. They did it any way.And they found they lost too much situational awareness.

Clearly L2 (steering) and L3 should be skipped.

 

[imispgh]

Autopilot and Enhanced Autopilot requires the driver to keep their hands on the wheel. If they don't, after a few warnings the system shuts off and requires the driver to take full manual control.

We don't yet know for sure what the interior camera in new Teslas is for, but it may eventually be used to ensure drivers are paying attention.



I looked into this. If I'm not mistaken, this is simply the number of miles that all Toyota cars in the world drive each year. It wasn't the result of any kind of study, analysis, or calculation.



RAND assumed that there would only be a fleet of 100 test cars. Based on Tesla's current production targets, we can estimate it will have 2 million Hardware 2 cars on the road by 2021.

If the Tesla Network launches at the beginning 2021 (based on the tens of billions of miles driven in shadow mode demonstrating a 20%+ safety improvement over human driving), and if those 2 million cars drive 5x as much as normal since they are shared among 5 people each, then Tesla's fleet will be driving 165 billion fully autonomous real world miles per year at the beginning of 2021.

Each subsequent month, the annual rate of miles driven will increase as more cars are produced. If 2 million cars are produced in 2021, the rate will be 330 billion miles per year at the beginning of 2022.

If self-driving cars drive 10x as many miles as human drivers rather than 5x, then these numbers will be 330 billion miles per year in 2021, and 660 billion miles per year in 2022.



Tesla, Waymo, and other companies in the field are making extensive use of simulation.

Waymo has done over 2.5 billion simulated miles so far. It's doing 8 million miles per day (a rate of almost 3 billion per year).

Since we know Tesla has a simulation operation used for the same purpose, and since Tesla has more real world data than Waymo to feed its simulation, it seems likely to me that Tesla does at least many simulated miles as Waymo. Possibly many more.

While I understand the math I doubt your assumption is correct for two reasons. One is that if Tesla had more sim miles and did that properly their vehicles would perform far better by now. Second is Musk would realize allowing folks to EVER take their hands off the wheel until L4 is extremely counter-productive. As Waymo found and Toyota has always known.

 

[AnxietyRanger]

Since we know Tesla has a simulation operation used for the same purpose,

All we know for sure is that they'v been hiring for that. We don't know what they have at this time.

and since Tesla has more real world data than Waymo to feed its simulation,

Waymo may have more relevant real world data, though, for autonomous operation. Tesla may not be collecting comparable data yet, from what we've seen the fleet data operation so far is rather simple.

it seems likely to me that Tesla does at least many simulated miles as Waymo. Possibly many more.

That's IMO a massively optimistic leap.

Really @Trent Eady, I think if you tone down your optimism on Tesla, you will get more accurate analysis. There is nothing wrong with your thesis overall, it is one amongst many, but some of this stuff is just so over the top...

 

[EinSV]

Really @AnxietyRanger I think if you tone down your relentless negativity on Tesla, you will get more accurate analysis

FTFY

 

[AnxietyRanger]

Really @AnxietyRanger I think if you tone down your relentless negativity on Tesla, you will get more accurate analysis

FTFY

You guys (@EinSV @mkjayakumar) really think Tesla has as many or more simulated miles than Waymo?

 

[bambam4171]

You guys (@EinSV @mkjayakumar) really think Tesla has as many or more simulated miles than Waymo?

So how many cars does Waymo have on the street (even adding all the cars of the last few years leading up to this vs Tesla? Yea, because every AP2 car counts, ya know?

 

[AnxietyRanger]

So how many cars does Waymo have on the street (even adding all the cars of the last few years leading up to this vs Tesla? Yea, because every AP2 car counts, ya know?

Had @Trent Eady's claim been, Tesla has a bigger fleet of cars than Waymo, I would have found that totally true, because obviously it is, as far as number of cars go.

Though then the separate issue is how much those fleets actually do. Waymo probably does more with every car in their fleet to help autonomous efforts than Tesla does with current AP2 cars (which at most advanced do some labelled visual identification validation so far it seems), given that Waymo has been driving autonomously in public for a decade, and all Tesla has done are a couple of FSD video runs when it comes to actual autonomous stuff. We know from @verygreen that no shadow mode FSD runs in AP2 cars at this time, basically just a Level 2 drivers aid with some vision and logging validation going on... all the while every Waymo on the road is collecting full full Level 4/5 autonomous validation data - and has been doing so for nearly a decade.

But be that as it may, @Trent Eady's claim was that Tesla likely has as many or more simulated miles as Waymo. Now, in simulation Tesla has no fleet advantage. And they certainly don't have the timeline advantage. Waymo has been single-mindedly at this for nearly a decade (some people behind it since mid-00s), Tesla's AP2 effort is much newer.

I simply see that claim as outrageous. It reminds me of the hopeful (but in retrospect naive) AP2 talk in November 2016, much more than the IMO more realistic AP2 talk (on all sides of the divide) we usually have here on TMC in November 2017.

 

[bambam4171]

You likely got a point with the number of simulated miles. However I think as the approach is not really the same it is rather difficult to see who sits exactly where with FSD. Yes, Waymo does have full FSD in a very limited area and they also have gained a few million miles on "open road". However I'm not sure this translates into the same lead this seams to imply and I think that once NN has been sorted Tesla will make much faster progress than Waymo exaclty because of the huge fleet, disengagement reports, requesting photos of corner cases and so on.
Note: I have no real insight into this so this is but a simple opinion

 

[AnxietyRanger]

However I think as the approach is not really the same it is rather difficult to see who sits exactly where with FSD.

This is completely fair.

I think that once NN has been sorted Tesla will make much faster progress than Waymo exaclty because of the huge fleet, disengagement reports, requesting photos of corner cases and so on.

I agree that so goes the theory, at least. We shall see.

 

[strangecosmos]

Simulation scenarios are generated using real world data as input. Real world miles are therefore a limiting factor for useful simulation miles. Brett Winton — a professional analyst with an engineering degree who researchers self-driving cars among other topics — has some insightful thoughts on this on Twitter: Brett Winton on Twitter Brett points out that Tesla has more 3x more real cars (now 4x more) than Waymo has simulated cars.

I actually give a lot of thought and research time to these topics. I pay attention to what experts think. I'm not just making stuff up off the top of my head.

It is plausible that Tesla has a small internal fleet of test cars as well. We know that Tesla tests Autopilot features on closed tracks. There is a private test facility for self-driving cars in northern California. Nevada is also a popular state for testing. While this is just a guess, I think it is a plausible guess.

 

[AnxietyRanger]

Simulation scenarios are generated using real world data as input. Real world miles are therefore a limiting factor for useful simulation miles.

Be that as it may, your claim was Tesla likely has as many or more simulated miles than Waymo. There is nothing to support that, nor is that a plausible rationalization given the larger picture.

Had you said Tesla has more fleet miles than Waymo, that would have been at least plausible. Although even then it can be argued Waymo probably has more autonomous-relevant fleet mileage data, given how they've been running their autonomous fleet for a decade and Tesla basically has some very limited driver's aid shadow mode data from AP at this stage (@verygreen has been very insightful).

I actually give a lot of thought and research time to these topics. I pay attention to what experts think. I'm not just making stuff up off the top of my head.

I don't doubt you do.

However, I think your confirmation bias tends to get the best of you at times. You seem to have a very strong internal thesis guiding you (basically Elon's Law on autonomous), and that IMO at times is misleading you.

It is plausible that Tesla has a small internal fleet of test cars as well. We know that Tesla tests Autopilot features on closed tracks.

Of course they do, we've seen enough evidence of it over the years.

Again, the claim from you was Tesla has as many or more simulated miles as Waymo, an autonomous industry pioneer who has been at this for a decade. There is nothing to support such a wild claim, nor does Tesla's consumer fleet advantage somehow translate into simulated miles advantage.