Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

"Elon’s tweet does not match engineering reality per CJ." - CJ Moore, Tesla's Director of Autopilot Software

This site may earn commission on affiliate links.

Bladerskb

Senior Software Engineer
Oct 24, 2016
3,207
5,550
Michigan
Document
On March 9, 2021, DMV representatives, Miguel Acosta, Nathan Gargiulo, Jennifer Berry and Emily Bisnett participated in a teleconference call via Microsoft Teams with Tesla representatives Eric Williams, CJ Moore, Alex Cobern, & Beth Mykytiuk. The call started at 1:30pm.
Update on City Street Pilot: Currently there are 824 vehicles in the pilot program-753 employees and 71 nonemployees. Pilot participants are across 37 states with majority of participants in California. As of March 2021, pilot participants have driven over 153,000 miles using the City Streets feature. By the end of the Week of March 9, 2021 Tesla will expand this pool of participants to approximately 1600. DMV asked CJ to address, from an engineering perspective, Elon’s messaging about L5 capability by the end of the year. Elon’s tweet does not match engineering reality per CJ. Tesla is at Level 2 currently. The ratio of driver interaction would need to be in the magnitude of 1 or 2 million miles per driver interaction to move into higher levels of automation. Tesla indicated that Elon is extrapolating on the rates of improvement when speaking about L5 capabilities. Tesla couldn’t say if the rate of improvement would make it to L5 by end of calendar year.

Keyword: "Elon’s tweet does not match engineering reality per CJ."
 
Last edited:
Interesting, can anyone explain why that portion, "does not match engineering reality" is whited-out / blanked-out / redacted in the actual document?

Screen Shot 2021-05-06 at 8.07.21 PM.png
 
Ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssshhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh...ehhSsssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssshhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhocking.

Wudyamean l5 ain’t happening next week?
 
The ratio of driver interaction would need to be in the magnitude of 1 or 2 million miles per driver interaction to move into higher levels of automation.

"Driver interaction" seems like a much broader term than disengagement or even intervention. I don't think it would include applying torque to the wheel as that is a driver monitoring requirement. IMO, "driver interaction" probably includes stalk confirmations, corrective braking, accelerating and steering and of course disengagements.

The 1M or 2M miles per driver interaction appears to be a rough ballpark number that Tesla is aiming for in order to reach L5. I don't think it is possible to actually measure or classify levels by their disengagement rate. I know the SAE document says that you cannot quantitatively measure the SAE levels.

In any case, the current FSD Beta is nowhere near 1M or 2M miles per driver interaction. So Tesla is very far from their own stated L5 goal. But it is sneaky that Tesla won't release any data on this. That way, there is no data to quantify how far Tesla is from their L5 goal and they can just ride on the hype instead.

I wish Tesla would release the current miles per interaction so that we could get a sense of how far away Tesla is from their L5 goal. Tesla could release the data of FSD Beta miles and number of driver interventions every quarter and we could see how much progress Tesla is making.
 
Last edited:
"Driver interaction" seems like a much broader term than disengagement or even intervention. I don't think it would include applying torque to the wheel as that is a driver monitoring requirement. IMO, "driver interaction" probably includes stalk confirmations, corrective braking, accelerating and steering and of course disengagements.

The 1M or 2M miles per driver interaction appears to be a rough ballpark number that Tesla is aiming for in order to reach L5. I don't think it is possible to actually measure or classify levels by their disengagement rate. I know the SAE document says that you cannot quantitatively measure the SAE levels.

In any case, the current FSD Beta is nowhere near 1M or 2M miles per driver interaction. So Tesla is very far from their own stated L5 goal. But it is sneaky that Tesla won't release any data on this. That way, there is no data to quantify how far Tesla is from their L5 goal and they can just ride on the hype instead.

I wish Tesla would release the current miles per interaction so that we could get a sense of how far away Tesla is from their L5 goal. Tesla could release the data of FSD Beta miles and number of driver interventions every quarter and we could see how much progress Tesla is making.
What do you mean when you say “their L5” goal? Does Tesla have a different set of goalpost for L5 than the SAE? If so shouldn’t Tesla not say L5 but rather call it something else to avoid confusion?
 
Keyword: "Elon’s tweet does not match engineering reality per CJ."
"Senior Software Engineer" is surprised engineers think comments by Execs don't match engineering reality. I guess you have never worked in a tech company. I can tell you in my 30 years as a "senior software engineer" - comments by execs has *never* matched engineering reality.
 
What do you mean when you say “their L5” goal? Does Tesla have a different set of goalpost for L5 than the SAE? If so shouldn’t Tesla not say L5 but rather call it something else to avoid confusion?

No. I think we can assume that Tesla is using the same definition of L5 as the SAE is.

Here is what the SAE says about L5:

The sustained and unconditional (i.e., not ODD-specific) performance by an ADS of the entire DDT and DDT fallback without any expectation that a user will respond to a request to intervene.

NOTE 1: “Unconditional/not ODD-specific” means that the ADS can operate the vehicle under all driver-manageable road conditions within its region of the world. This means, for example, that there are no design-based weather, time-of-day, or geographical restrictions on where and when the ADS can operate the vehicle. However, there may be conditions not manageable by a driver in which the ADS would also be unable to complete a given trip (e.g., white-out snow storm, flooded roads, glare ice, etc.) until or unless the adverse conditions clear. At the onset of such unmanageable conditions the ADS would perform the DDT fallback to achieve a minimal risk condition (e.g., by pulling over to the side of the road and waiting for the conditions to change).

NOTE 2: In the event of a DDT performance-relevant system failure (of an ADS or the vehicle), a level 5 ADS automatically performs the DDT fallback and achieves a minimal risk condition NOTE 3: The user does not need to supervise a level 5 ADS, nor be receptive to a request to intervene while it is engaged.

EXAMPLE: A vehicle with an ADS that, once programmed with a destination, is capable of operating the vehicle throughout complete trips on public roadways, regardless of the starting and end points or intervening road, traffic, and weather conditions.

The issue is that the SAE defines L5 but does not provide specific metrics for actually measuring when you have achieved L5. So how do you determine when you system meets that definition above?

So by "their goal", I simply mean Tesla's way for measuring when they think they will achieve the SAE definition of L5 above. The "1M - 2M miles per driver interaction" appears to be Tesla's metric that they made up to help them determine when they think they will achieve the SAE definition of L5.
 
Possibly dumb question, but is 'City Street Pilot' the current 'FSD Beta'? I thought there were supposed to thousands in the FSD Beta, this states only 824 vehicles - much smaller...

Yes, "City Street Pilot" is FSD Beta. The email is dated March 9. So the 824 vehicles in FSD Beta is from March 9. The email mentions that by the end of the week of March 9, FSD Beta will expand to 1,600. It's now May, so FSD Beta may have expanded to more than that.
 
  • Informative
Reactions: pkitch
"Senior Software Engineer" is surprised engineers think comments by Execs don't match engineering reality. I guess you have never worked in a tech company. I can tell you in my 30 years as a "senior software engineer" - comments by execs has *never* matched engineering reality.
To be fair, Musk is the Technoking of Tesla, though I don't believe he's the Chief Engineer, like at SpaceX.
 
There's a big difference between releasing FSD at L2 and FSD at L5.

I think there's a decent chance that Tesla will release FSD City Streets at L2 (requires constant monitoring) to the general public this year. But I doubt it will be good enough to meet the L5 definition.
Absolutely. If it's released to the general public in the form of a download button or otherwise it'll clearly be L2 for the foreseeable future and possibly forever.
 
....I wish Tesla would release the current miles per interaction so that we could get a sense of how far away Tesla is from their L5 goal. Tesla could release the data of FSD Beta miles and number of driver interventions every quarter and we could see how much progress Tesla is making.

Agreed.

I suspect the reason for keeping the driver intervention numbers secrete is because it's nowhere near L3 and above any time soon. That means its rate is just like the current wide release of Autopilot or even worse.

The current revelation of FSD beta from youtube videos shows that FSD beta drivers might have to be more vigilant than plain Autopilot drivers.
 
I think there's a decent chance that Tesla will release FSD City Streets at L2 (requires constant monitoring) to the general public this year. But I doubt it will be good enough to meet the L5 definition.
But even that makes Elon a liar- he says they are waiting for the march of 9's to release the FSD beta:


"This is a “march of 9’s” trying to get probability of no injury above 99.999999% of miles for city driving. Production Autopilot is already above that for highway driving." CJ's email makes it clear that they are nowhere near that. Elon's number is 1 in 100 million miles, and CJ says they aren't at 1:1M. So there is zero way Tesla can expect to release the beta button "next month" if they need 3 orders of magnitude improvement in reliability before release. It also really calls into question if base highway AP is that high as well.
 
Last edited:
Yes, but also:
"Tesla said that as of March 9th there were 824 vehicles in the pilot program, including 753 employees and 71 non-employees."

71 non-employees. .007% of all cars. 90% of all the cars running it are employees. It's nothing.

Agreed.

It would be more reasonable for Elon's enthusiasm if hundreds of thousands or millions of Tesla owners have been running the FSD beta a long time ago and we should not be just still in a planning stage of when the download button would still appear someday!

It just doesn't add up that 1 million robotaxis would run around without drivers by 2020 (last year).
 
I suspect the reason for keeping the driver intervention numbers secrete is because it's nowhere near L3 and above any time soon.
Dirty Tesla (Youtube person with FSD beta) is at 3 miles per disengagement.
Only a 1 million time increase in reliability is needed to hit CJ's 1-2 million target. Only 30 million needed to hit Elons 1:100M.
Let's say we manage to increase reliability by 50% every week.
3 to 1 million is only 31 weeks away!
See, we're almost there. Just need to improve exponentially for the next 8 months on something that has taken 5 years to get from 0-3.
Then Elon's number is only 12 weeks behind that!

Ok, Ok, 50% is pretty unrealistic. Let's say we can do 10% per week...
Well, then 1:1M is only 134 weeks away. 2.6 years. Sounds perfect for Elon. It's always 2 years away.