March of 9s: The Beginning of the End

I have had enough experience now with FSD 12 to say that in my opinion, I believe Ashok is correct:

https://twitter.com/x/status/1774673326583210476

I would also stress we are at the beginning of the end. That is, we will hopefully now start to see significant improvement with the "known FSD issues" and "corner case disengagements" with every major FSD release. "The end" (level 3 /4 autonomy within a wide Operational Design Domain (ODD)) is almost certainly year+ away. I'd like to document, just from one consistent anecdotal use case that can be repeated over time, where we are "starting from."

I will be driving a 90ish mile (2-3 hrs) "loop" under FSD to cover a range of driving scenarios. (Mileage is approximate):

For privacy reasons, there are about 15 miles in my loop that are not included in the link above that takes me to/from my actual home.

Here is a Link to inspect the route in detail

This route takes me from NJ Suburbs into and out of Manhattan (NYC) and includes approximately:

• 10 Miles Suburban driving
• 65 Miles "limited access highway" driving (This should be using the FSD 'highway stack' which is not the same as the FSD 12 stack)
  • includes interchanges
  • Includes Tolls
  • Includes Tunnel
• 8 miles of other "highway type driving", (will probably fall under the FSD 12 stack)
• 6 miles of dense city driving...including areas around Times Square, Rockefeller center, etc which will have dense vehicle and pedestrian traffic.

I will not be recording with a phone or anything like that. However I will try and save dashcam footage of anything notable.

I will report on:

1. Interventions (Accelerator presses, particularly if safety related)
2. Disengagements (comfort or safety related)
3. Overall impressions

As we know Version 12.3.x does not support...(but will need to in the future):

1. Smart Summon or "Banish"..so what I call the "first 100 yards and final 100 yards" is not available to test. (Drop offs / pick-ups).
2. "Reversing" while on FSD is not yet supported

Finally, there are what I would say 2 well documented "comfort / safety" issues with FSD 12.3.x that I have also experienced regularly first hand:

1. "Lane Selection Wobble"...for example, approaching an intersection where the single driving lane splits into multiple lanes (turning vs. straight)...FSD may act "indecisively"
2. Unprotected turn (stop sign) behavior. Notably: stops early....then creeps. If no cars detected it may creeps into intersection instead of "just going". Further, if it has creeped into the intersection, THEN detects a car approaching, it may still hesitate and require intervention (accelerator press) to get it going.

In addition to those two consistent issues, I expect to encounter some issues related to routing, and any number of other 'corner case' issues. All things that will ultimately need to be handled, but we expect to see dealt with as we progress though the "March of 9s"...toward the "end of the end".

Although I have driven FSD regularly over the past 3 weeks...I have yet to take it into NYC.

Vehicle: Refresh Model S (2023), Vision Only, HW4. First test will be using:
Firmware version: 11.1 (2024.3.15)
FSD Version: 12.3.4

So...there's the set-up. I expect later today to drive the first loop.

 

[EvilNuff]

Its amazing to me that people actually think full level 5 ie robotaxi is even possible with the current hardware. Just goes to show that PT Barnum is alive and well in this thread.

 

[Joey D]

Elon is so incredibly out of touch with reality. The lane change head fake is very much alive and well in 12.4.

You just can't take this guy seriously on anything. I'm surprised people quote him.

12.4 is not yet released to the public.

He is commenting on an internal build. So before you state that you can't take anything he says seriously, understand what he's saying first.

 

[Joey D]

Its amazing to me that people actually think full level 5 ie robotaxi is even possible with the current hardware. Just goes to show that PT Barnum is alive and well in this thread.

I'm not sure if Level 5 will be possible. But level 4 absolutely. The big question is: what will be the ODD (Operational Design Domain) restrictions applied.

Robotaxi does not need to be level 5 to be a successful or viable product.

 

[legendsk]

To me, v11.x represented a valiant effort on Tesla's part to write software to take data in and output controls to the car well enough to allow the car to drive itself. I find it ironic that SpaceX has a recovery ship named "A Shortfall Of Gravitas", because of how accurately that describes v.11. It is so good it's marvelous, but it is also well short of adequate. It does a very few things reliably, most things unreliably and quite a few things not at all.

And as far as I know v11.x is still being used for highway driving? v12.3.4 is a giant improvement across the board and is being used on city streets? So I don't think it is obvious to most drivers when the software moves back and forth between v11 and v12 as the environment changes? If this is accurately the current status, then drivers are going to have quite different experiences depending on the environment in which they are driving, accounting somewhat for the range of reviews.

If this is really the current status, then I think when highway driving is sufficiently trained and v.12.x is working everywhere, we will quite likely be at the point where FSD is safer than humans. And that seems pretty close.

 

[spacecoin]

I'm not sure if Level 5 will be possible. But level 4 absolutely. The big question is: what will be the ODD restrictions applied.

Robotaxi does not need to be level 5 to be a successful or viable product.

Level 5 won't happen in our lifetimes or perhaps ever. It's an aspirational level.

With current hardware I'm thinking Tesla could be Level 4 in speeds up to 30-40 mph if there is no precipitation. Something along the lines of what Drive Pilot offers if they spend time on getting there. Doubt they will though.

 

[spacecoin]

If this is really the current status, then I think when highway driving is sufficiently trained and v.12.x is working everywhere, we will quite likely be at the point where FSD is safer than humans. And that seems pretty close.

According to Tesla FSD has been safer than a human for years. FSD is years away to be safer than a human by itself.

 

[Joey D]

According to Tesla FSD has been safer than a human for years.

To be clear...According to Tesla, a human driving on Supervised FSD has been safer than a human driving without FSD for years. Tesla has never claimed that unsupervised FSD is safer than humans....yet.

 

[uscbucsfan]

To be clear...According to Tesla, a human driving on Supervised FSD has been safer than a human driving without FSD for years. Tesla has never claimed that unsupervised FSD is safer than humans....yet.

Which this is accurate and measurable, but misleading when some see that and say FSD is safer than a human already look at what Tesla says. You basically have a teen driver and drivers ed driver with 2 sets of pedals with Supervised FSD. Of course that will be safer than anyone alone if they are being hyper vigilant.

 

[Joey D]

All necessary

Which this is accurate and measurable, but misleading when some see that and say FSD is safer than a human already look at what Tesla says. You basically have a teen driver and drivers ed driver with 2 sets of pedals with Supervised FSD. Of course that will be safer than anyone alone if they are being hyper vigilant.

I (nor Tesla) can help if others don't read the stats and statements and misinterpret what was actually said. That's not being misleading. Demonstrating safety is something that is done in steps (and in different ways) by any company pursuing autonomy.

 

[uscbucsfan]

All necessary

I (nor Tesla) can help if others don't read the stats and statements and misinterpret what was actually said. That's not being misleading. Demonstrating safety is something that is done in steps (and in different ways) by any company pursuing autonomy.

Sorry, I meant misleading from the people incorrectly interpreting it. Tesla has never said FSD unsupervised is even real, yet...much less safer than a human.

 

[Dewg]

Level 5 won't happen in our lifetimes or perhaps ever. It's an aspirational level.

With current hardware I'm thinking Tesla could be Level 4 in speeds up to 30-40 mph if there is no precipitation. Something along the lines of what Drive Pilot offers if they spend time on getting there. Doubt they will though.

Level 5 isn't even possible by humans, so extremely doubtful a computer will ever achieve it.

 

[uscbucsfan]

Level 5 isn't even possible by humans, so extremely doubtful a computer will ever achieve it.

I said this somewhere else and people treated me like I was an idiot. "People can drive everywhere in every condition and computers will be better".

 

[JB47394]

Level 5 isn't even possible by humans, so extremely doubtful a computer will ever achieve it.

Do we care about "Level 5" or do we care if Tesla can create a working robotaxi service? The whole SAE level thing is pedantry. Perhaps a useful starting point for a discussion, but I don't see any value in trying to use it beyond that point.

Let's say that Tesla needs a billion clips to properly train their neural networks for the problem space (viable solutions to all driving scenarios). If they can curate the "best" billion clips from the available ten billion clips, then surely the system will be able to drive far better than the average driver. That should make for a serviceable robotaxi. Assuming that the hardware is equal to the task.

Refine from there.

 

[Joey D]

Level 5 isn't even possible by humans, so extremely doubtful a computer will ever achieve it.

If I understand "level 5" correctly...by definition humans are level 5. It's the ability to drive "anywhere that humans typically drive, in all conditions that humans typically drive in." I do not believe level 5 means "be able to drive places where humans don't, in conditions where humans don't successfully drive."

 

[uscbucsfan]

If I understand "level 5" correctly...by definition humans are level 5. It's the ability to drive "anywhere that humans typically drive, in all conditions that humans typically drive in." I do not believe level 5 means "be able to drive places where humans don't, in conditions where humans don't successfully drive."

I don't believe the humans part is mentioned in the SAE document. I can't recall, but the Waymo CEO said Level 5 is impossible because it's "all conditions" and he said people/cars cannot function in all conditions.

 

[Joey D]

Do we care about "Level 5" or do we care if Tesla can create a working robotaxi service?

I think all people really care about is: being autonomously driven from point A to point B safely, efficiently, and low cost.

Level 5 is not required.

The larger the domain for operation, the more valuable the robotaxi service is.

 

[Dewg]

If I understand "level 5" correctly...by definition humans are level 5. It's the ability to drive "anywhere that humans typically drive, in all conditions that humans typically drive in." I do not believe level 5 means "be able to drive places where humans don't, in conditions where humans don't successfully drive."

Everywhere means everywhere in the world, not just your little corner of the country you live in. I bet I can drop you in some Asian country and you'd be hard pressed to drive without causing a significant safety issue.

 

[Joey D]

Everywhere means everywhere in the world, not just your little corner of the country you live in. I bet I can drop you in some Asian country and you'd be hard pressed to drive without causing a significant safety issue.

Driving in NYC is stressful enough! And I made my share of mistakes on a recent trip to Scotland.

 

[GOVA]

Its amazing to me that people actually think full level 5 ie robotaxi is even possible with the current hardware. Just goes to show that PT Barnum is alive and well in this thread.

What is really amazing is imagination myopia in 99.99% of folks who have no idea what are they are talking about. Regardless of Elon's re-iterating the major points every time he opens his mouth.

 

[Dewg]

Driving in NYC is stressful enough! And I made my share of mistakes on a recent trip to Scotland.

One of my places to visit!
Unfortunately, L5 cars aren't at liberty to make their share of mistakes.