FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Mardak]

It's been a while since this Tesla employee tweeted about FSD Beta and now showing 11.3 without specifically mentioning it:

https://twitter.com/x/status/1629381021836926976

Looks like crosswalks are represented with 2 lines instead of a gray region now.

 

[InductionBalls]

Those arcs in the visualization - is that vision-based USS essentially?

 

[JulienW]

Those arcs in the visualization - is that vision-based USS essentially?

My money is that the car has USS (not a post Sep 3 or Y) and that is just them being displayed as before. See pic above as probable proof that it is the same car (most likely).

For some reason the pic kinda looks "Photoshopped" to me. Looking at the Speed MPH the pic was taken is blurry but the car, arcs and tentacle are razor sharp.

EDIT/conjecture: If it is an overlay/Photoshopped and by not mentioning V11 he is probably skirting the NDA.

 

[JB47394]

The part I like to see is that the car is doing 22 at the entrance to the traffic circle, suggesting that it's not stopping.

 

[M109Rider]

I can’t wait to try the full stack out. Less blinks before the lane change on the highway would be nice.

 

[JulienW]

TeslaBull posted a new night drive video. The quality is getting a tad better, but had no other place to go. Looks like they found a camera mount and selecting Dark Mode for night so can see the screen a little better. That "music" track is to die for. And I mean actually die. Needs some narration and SPEED up at LONG red lights at a minimum. WEIRDEST thing is that they are driving with a seatbelt unbuckled.

EDIT: Looks like someone in the backseat (without seatbelt) is holding the camera.

 

[sleepydoc]

WEIRDEST thing is that they are driving without a seatbelt.

Are you sure? FSDb won’t activate if you don’t have your seatbelt on. I think the notice you’re seeing in the bottom left of the screen is because there’s a bag or something else heavy enough to trigger the sensor in another seat.

 

[JulienW]

Are you sure? FSDb won’t activate if you don’t have your seatbelt on. I think the notice you’re seeing in the bottom left of the screen is because there’s a bag or something else heavy enough to trigger the sensor in another seat.

Keep reading where I typed EDIT.

 

[sleepydoc]

The part I like to see is that the car is doing 22 at the entrance to the traffic circle, suggesting that it's not stopping.

Yes - everyone was talking about chuck’s unprotected left turns but right turns on yield (I.e. unprotected right turns) we’re just as much (if not more) of an issue IME. They are more common and in many ways more likely to cause accidents if the car suddenly stops unnecessarily at a right turn yield.

In the most basic sense, traffic circles are nothing but a group of unprotected rights, too.

 

[Mardak]

Looks like parking lot path jitter isn't much improved with 11.3:

 

[dachshund]

Yes - everyone was talking about chuck’s unprotected left turns but right turns on yield (I.e. unprotected right turns) we’re just as much (if not more) of an issue IME. They are more common and in many ways more likely to cause accidents if the car suddenly stops unnecessarily at a right turn yield.

In the most basic sense, traffic circles are nothing but a group of unprotected rights, too.

Chuck's unprotected left turn has a median - an extra lane, in effect. I rarely see that extra middle lane on my roads. An unprotected left is a left turn across all lanes to the left and at least one lane to the right. There's no safety lane to pause in while waiting for a gap in traffic from the right.

Does this make the chuck turn easier or harder for FSD? It seems like two separate operations, vs the "single operation" left turn where I'm located.

 

[ddeflyer]

Yes - everyone was talking about chuck’s unprotected left turns but right turns on yield (I.e. unprotected right turns) we’re just as much (if not more) of an issue IME. They are more common and in many ways more likely to cause accidents if the car suddenly stops unnecessarily at a right turn yield.

In the most basic sense, traffic circles are nothing but a group of unprotected rights, too.

Yeah, I had a scary one yesterday, was at a red stoplight and FSD decided to do a right turn on red while someone was doing a U-turn. Needless to say I slammed my brakes hard

I do find it ironic that FSD is terrified at stop signs but rambunctious at right on red

 

[Supcom]

Chuck's unprotected left turn has a median - an extra lane, in effect. I rarely see that extra middle lane on my roads. An unprotected left is a left turn across all lanes to the left and at least one lane to the right. There's no safety lane to pause in while waiting for a gap in traffic from the right.

Does this make the chuck turn easier or harder for FSD? It seems like two separate operations, vs the "single operation" left turn where I'm located.

I believe that the extra lane makes Chuck's turn quicker, but not necessarily easier. When there is an extra lane that the car will use as a stopping point part way through the turn, the car has the additional task of positioning itself in that lane so as not to block other traffic, yet have visibility to the right in order to complete the turn safely. It also may need to deal with other traffic using the median area as a stopping place.

Without the extra lane, the car needs to complete the turn at one go. This means that the car cannot start the turn until both directions have sufficient gaps. That's not harder for the car, but simply takes more waiting time on average, which is probably harder on the driver's patience than the car.

 

[powertoold]

Looks like parking lot path jitter isn't much improved with 11.3:

I think the most interesting thing about 11.3 for me is a total 180 on how the road geometry is predicted, according to the visualization.

Rather than predicting the road geometry based on an "HD map"-like memory, it is now predicting more based on what it actually sees. On 69.2, when you stopped at an intersection, the road geometry was very stable (almost as though it were static), but if you watch the 11.3 videos, the road geometry "pulsates" even when the car is stopped.

I had talked about this in a prior post about how I thought it was odd that Tesla was auto-labeling the entire road geometry (AI Day 2), regardless of whether the car actually saw the features that led to that prediction.

I find this intriguing because when Karpathy was at Tesla, in his presentations at CVPR (and others), he showed road geometry outputs that are more like 11.3, except 11.3 is more refined and trained longer:

I always thought the "HD-map"-like autolabeling was a crutch, plus it wasn't clear how they could store such a large memory of intersections that could run on HW3.

I don't know if I'm making sense on this. I don't work in the field, but this is what I'm seeing with 11.3. I think the best litmus test would be to drive Lombard St on 11.3.

 

[sleepydoc]

Looks like parking lot path jitter isn't much improved with 11.3:
View attachment 911597

In all fairness, parking lots are probably one of the most difficult areas to drive. I often have a lot of jitter when I’m trying to negotiate them!

Chuck's unprotected left turn has a median - an extra lane, in effect. I rarely see that extra middle lane on my roads. An unprotected left is a left turn across all lanes to the left and at least one lane to the right. There's no safety lane to pause in while waiting for a gap in traffic from the right.

Does this make the chuck turn easier or harder for FSD? It seems like two separate operations, vs the "single operation" left turn where I'm located.

The island nominally makes it easier since you only have to worry about half of road at a time, but the code needs to know how to deal with it.

Yeah, I had a scary one yesterday, was at a red stoplight and FSD decided to do a right turn on red while someone was doing a U-turn. Needless to say I slammed my brakes hard

I do find it ironic that FSD is terrified at stop signs but rambunctious at right on red

This happens on a regular basis when people make U turns at lights. In these cases, the person turning right has the right of way and the person making the U turn has to yield, even if the person turning right has a red light, so FSDb was in the right here.

 

[kabin]

TeslaBull posted a new night drive video. The quality is getting a tad better, but had no other place to go. Looks like they found a camera mount and selecting Dark Mode for night so can see the screen a little better. That "music" track is to die for. And I mean actually die. Needs some narration and SPEED up at LONG red lights at a minimum. WEIRDEST thing is that they are driving with a seatbelt unbuckled.

EDIT: Looks like someone in the backseat (without seatbelt) is holding the camera.

Some of the braking when turning into the intersection left turn lanes looked smoother. Maybe using regen more now?

 

[emmz0r]

https://twitter.com/x/status/1416511905515048966

Any sublime auto park and summon yet?

 

[Dan D.]

Any sublime auto park and summon yet?

Ah sublime:

to cause to pass directly to the vapor state

 

[thewishmaster]

TeslaBull posted a new night drive video. The quality is getting a tad better, but had no other place to go. Looks like they found a camera mount and selecting Dark Mode for night so can see the screen a little better. That "music" track is to die for. And I mean actually die. Needs some narration and SPEED up at LONG red lights at a minimum. WEIRDEST thing is that they are driving with a seatbelt unbuckled.

EDIT: Looks like someone in the backseat (without seatbelt) is holding the camera.

Anyone else notice that phantom brake at 3:05? Dips into regen briefly after this screenshot. The red line visualization makes these more obvious than before

 

[ddeflyer]

In all fairness, parking lots are probably one of the most difficult areas to drive. I often have a lot of jitter when I’m trying to negotiate them!

The island nominally makes it easier since you only have to worry about half of road at a time, but the code needs to know how to deal with it.

This happens on a regular basis when people make U turns at lights. In these cases, the person turning right has the right of way and the person making the U turn has to yield, even if the person turning right has a red light, so FSDb was in the right here.

I've always assumed the person doing a U-turn has the right of way and from a quick googling it looks like in California the vehicle making the U-turn has the right of way.