FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Dewg]

After the stop, does the car immediately and assertively take a turn? Or does it keep creeping for another 10 seconds when no one in sight? That's the 2nd part of the stop sign problem.

It just goes. I'll see if I can have a passenger take a video next time I think about it. I tried once but got yelled at for picking up my phone.

 

[EVNow]

After the stop, does the car immediately and assertively take a turn? Or does it keep creeping for another 10 seconds when no one in sight? That's the 2nd part of the stop sign problem.

When the intersection (or roundabout) is empty - looks like FSD has more uncertainty about what the lane structure is. But if the intersection / roundabout has traffic, the uncertainty decreases and it acts quickly. Same with having a lead vehicle.

It is also possible some of the 4 way stops are mapped as 2 way stops contributing to the indecisiveness.

 

[kabin]

Lol ... glass half full vs half empty.

View attachment 916424

Yea, maybe the driver of the following vehicle just broke water but the poor performance remains.

I'll go out on a limb and say HW3's last great SW hope (V11) will be unable to solve roundabouts, sluggish intersections, and general indecisiveness.

 

[kabin]

When the intersection (or roundabout) is empty - looks like FSD has more uncertainty about what the lane structure is. But if the intersection / roundabout has traffic, the uncertainty decreases and it acts quickly.

That's not my experience.

 

[EVNow]

Yea, maybe the driver of the following vehicle just broke water

I see ... you want to use the nuclear option for your argument

ps : In the early days of EVs one of the main things people always "pointed out" was "what will you do if your kids are sick / some other health emergency comes up but your car battery is empty". I used to always reply, call 911.

 

[kabin]

After the stop, does the car immediately and assertively take a turn? Or does it keep creeping for another 10 seconds when no one in sight? That's the 2nd part of the stop sign problem.

Yep. It seems they moved the creep into the intersection after the stop sign. I think the issue ties with the b pillar camera's inability to accurately estimate kinematics for moving objects and/or when the ego is moving. Their crawl crutch provides more dwell time at or near 0mph.

 

[JulienW]

Chuck's Memorial Park drive. As usual kinda boring. Wish he would go downtown JAX and do a route there.

 

[Supcom]

Chuck's Memorial Park drive. As usual kinda boring. Wish he would go downtown JAX and do a route there.

Plenty of what we see in other videos. Bad stop location at stop signs, incorrect speed limits, complete fails at roundabouts. Would not pass a slow moving quadracycle. Late move to give VRU extra clearance.

But, unlike other videos, it didn't run a red light.

 

[jeremymc7]

Anyone know if this latest beta is still using USS for cars that have it or has it all progressed to vision.

Unrelated but wonder if the pending new high resolution radar will be retrofitted to older cars that have the now unused radar still installed.

 

[JulienW]

Anyone know if this latest beta is still using USS for cars that have it or has it all progressed to vision.

Unrelated but wonder if the pending new high resolution radar will be retrofitted to older cars that have the now unused radar still installed.

They show up on the screen just like they do now. We don't know IF or to what extent they are used now (69.x.x) but they do show the same in V11. It is likely they are just displayed but FSD Beta doesn't actually use them when driving.

New radar is HW4 and all but certain it CAN'T or won't be offered as an upgrade.

 

[jeremymc7]

Correct me but the handful of HW4 cars shipped so far don’t have the new radar and are still using the older cameras?

HD radar and upgraded (not upcoming additional) cameras are not installed, yet? Only the HW4 computer at this point?

 

[AlanSubie4Life]

I just go by what Chuck says - he is in the driver's seat !

Just resummarizing the details.

1. Success! Ok. ("That was not rush hour", noted by Chuck - got lucky with light traffic)
2. Fail. Arguable 'Success.' Waited too long for opposite side traffic, at 2:03, when clear lanes #1, #2 existed. Took 9 or 10 seconds to proceed. 8mph crossing speed noted (this is slow and the language needs improvement). There was time, so that was ok, but maybe first faster, then slower would be better.
3. Fail. At 3:10 => 9-second visualization of traffic at 50mph, ~200-meter range, impressive, and required.
   Disengagement; did not cross the road; slowed to a low speed in middle of traffic lanes. There was a clear opportunity to just YOLO it at 3:14 - it had 5-6 seconds! Chuck says it needed to wait, but that's only because he's used to it going so darn slow across the road! Most people would go on that gap on this road. It finally continued at 4:04, losing 50 seconds, and exposing the driver to danger.
4. Fail. Disengagement - did not stop entering the road.
5. Fail. Went in front of truck when entering traffic flow and forced it to slow down significantly (at least 5mph)
6. Fail. Arguable 'Success" - missed a couple 6-second gaps and chose a gap with left turning traffic, which it could have just waited for (there was tons of gap to allow the left-turning traffic to complete their maneuver first, before the Tesla turned). Crossing traffic did not arrive until 8:04, so would have had about 6 seconds after allowing completion, though it may have had to use the median if it had waited.
7. Fail. Missed 9:14-9:21 gap. Forced traffic to pass on right after entering traffic flow. The earlier gap did not have traffic until 9:21 on the opposite side of the road, so it was the correct time to go and YOLO it, without any median logic.
8. Fail. Disengagement; steering too erratic, due to bad path planning to go to far lane for a fraction of a second - planner remains the same it seems.

So you could call it 5/8 if you wanted to only count disengagements (not a good threshold of course!).
I call it 1/8 using long-established scoring methods, or 2/8 if I want to be generous.

Argument could be made for 3/8, if I use very low standards.

Problems continue to be the same as 10.69:
1) Not nearly enough crossing speed (looks like dialed down to 10mph when crossing to median for now).
2) Not enough speed when entering traffic flow. Get up to 45mph instantly (somehow it knows the right speed…it is not 50mph here, yet).
3) Weird slow crossing behavior below 10mph. No commit.
4) Not going on the very safe gaps.

Looks to me like basically exactly the same code as 10.69, with slight slider adjustments.

I think the median logic should be a last resort, in the case of no gap where it looked like one might exist, and the car should try to thread into traffic in one maneuver - it is safer because it avoids the left-turning traffic problem - which is a huge potential safety issue.

Rather than persistently being apologists for these failures, I think we as a group need to get serious about demanding that these shortcomings be fixed by Tesla. I think we can all agree that this sort of behavior & performance is not ok. All we need to do is go out and drive ourselves and observe the speeds obtained when crossing, etc. And then we can agree on quantifiable metrics to push for (I've detailed them before in the 10.69 thread - they're quite easy to obtain and get high comfort, high confidence level speeds). It's just really sad to me that the ULT has somehow been allowed to fall off the radar, when it still needs to be at least 100x better, and there are clear avenues to get to at least 10x better. We're just not asking Tesla for it! As long as we pretend this is ok, we're going to continue to get unacceptable results from Tesla. These are certainly not in any way "personal preference" items from me - these are fundamental shortcomings that anyone in the car would immediately notice (just as Chuck does in the videos).

It makes me sad that we have such a long way to go and the obvious improvements and tweaks are not being made, specifically:
1) Much faster crossing speed. (15mph peak or so, as previously discussed and tested and documented, would be appropriate)
2) Use smaller gaps, assertively. 5-6 seconds is the safe, comfortable limit, if sufficient crossing speed is also used. Even shorter is possible if you want to YOLO the turn.
3) Avoid using the median unless required due to persistent heavy traffic. If there is a safer gap coming immediately, use that instead.
4) Get up to 45mph immediately (and then 50mph when the speed limit adjusts), whether median is used or not.
5) Improve the car body language when approaching the creep limit. Stop at the stop line, then immediately creep, with the right language.

It looks like it has all the perception (200 meters or more) - it just needs to start making safe, assertive decisions and take the safe gaps, rather than the risky ones. All the pieces are there - they just have to fix the body language so that the driver knows when the car is going to go, and then make the car assertive & quick.

 

[Knightshade]

Correct me but the handful of HW4 cars shipped so far don’t have the new radar and are still using the older cameras?

HD radar and upgraded (not upcoming additional) cameras are not installed, yet? Only the HW4 computer at this point?

I do not believe that is correct-- folks have already posted the much better looking dashcam footage from the new cameras in HW4 cars

 

[Mardak]

Has anybody noticed what FSD Beta 11.3.1 does in really bad weather? This video seems to keep "Full Self-Driving may be degraded / Poor weather detected" message up basically the whole drive:

Potentially with single stack, there's no basic Autopilot to fall back to while driving, but maybe the weather wasn't poor enough to become "unavailable?" Although even with FSD Beta 11.x, the code and neural networks are still there as one has the choice to set the slider to TACC / Autosteer / FSD Beta in the Autopilot settings. I hadn't noticed before with 10.x, but apparently the FSD Beta creep wall shows up when it falls back to pre-FSD-Beta Autopilot (with Traffic Light and Stop Sign Control toggled):

 

[DarkForest]

That's not my experience.

I’ve noticed that roundabout experience depends heavily on whether or not an intersection is properly mapped as a roundabout.

Map data seems to supplement vision, and vision-only isn’t as good.

 

[jabloomf1230]

I’ve noticed that roundabout experience depends heavily on whether or not an intersection is properly mapped as a roundabout.

Map data seems to supplement vision, and vision-only isn’t as good.

Every one of the 30+ roundabouts near my house () are mapped in Google Maps, TomTom and Open Street Maps. I've never noticed FSDb able to traverse any of them properly if there was even the slightest amount of traffic. Roundabouts are probably one of the reasons why there is no FSDb availability in Europe.

 

[gsmith123]

I hadn't noticed before with 10.x, but apparently the FSD Beta creep wall shows up when it falls back to pre-FSD-Beta Autopilot (with Traffic Light and Stop Sign Control toggled)

I wonder if it's actually using the creep limit or if this is just a visual artifact and the visualization is accidentally overlaying where the creep limit is even when TACC is not using it. What a strange combination of the new and old!

On V11.3.1 does engaging TACC switch back to legacy AP code for speed control?

On pre-V11 releases I haven't loved how steering wheel disengagements make the car fall back to legacy AP based TACC.

 

[sleepydoc]

Has anybody noticed what FSD Beta 11.3.1 does in really bad weather? This video seems to keep "Full Self-Driving may be degraded / Poor weather detected" message up basically the whole drive:

Potentially with single stack, there's no basic Autopilot to fall back to while driving, but maybe the weather wasn't poor enough to become "unavailable?" Although even with FSD Beta 11.x, the code and neural networks are still there as one has the choice to set the slider to TACC / Autosteer / FSD Beta in the Autopilot settings. I hadn't noticed before with 10.x, but apparently the FSD Beta creep wall shows up when it falls back to pre-FSD-Beta Autopilot (with Traffic Light and Stop Sign Control toggled):
View attachment 916498

10.69 has the same warning. I wouldn’t expect anything different in the near term. They need to get the basics down before they can handle variables like rain and snow b

 

[LowlyOilBurner]

HW3 is so (shuffles cards….) February 2023. I’m waiting to see the exponential advancements of HW4 and magical 5.1mp cams.

 

[EVNow]

Has anybody noticed what FSD Beta 11.3.1 does in really bad weather? This video seems to keep "Full Self-Driving may be degraded / Poor weather detected" message up basically the whole drive:

I've seen this happen with 10.69 as well. Only in heavy ain it falls back to AP - or sometimes when you start in rain, it will be in AP for a while and then switch to FSDb.