FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Dewg]

What do you think the current FSD Beta 10.x is lacking to replace standard Autopilot (in particular driving on highways)? I've used FSD Beta driving 65mph+ on highways, so both the FSD Beta perception and planning are in use already. But indeed I don't think I've had a situation where it needed to go around a pedestrian at those speeds to see how it behaves.

Tesla is probably well aware of phantom braking issues, and potentially false positive predictions on the production Tesla Vision stack are significantly reduced with FSD Beta 11 stack incorporating multiple camera videos and memory types. That could also be a reason to work on single stack and get it ready to deploy to the whole fleet to solve numerous issues with standard Autopilot.

I read a post from someone that said the CPU/RAM limitations of HW3 were the problem. I'm not sure if he was correct, but his argument was FSD Beta (city streets) uses much more RAM and CPU, especially dealing with far-away objects. It's not normally a problem because city streets don't go 80MPH. Put that same stack on freeways going 80MPH, and the system can't keep up.

I've heard Elon saying they are changing how cameras send data to the NNs, instead of processing raw video images, the system will process vectors instead. Perhaps that will allow the system to handle data at 80MPH with the same stack. I don't know, I'm just guessing here.

 

[Mardak]

I read a post from someone that said the CPU/RAM limitations of HW3 were the problem

Indeed this could be part of why FSD Beta 10.x stack with "bag of points" / "dense rasters" is getting replaced in FSD Beta 11, and some of it is being tested since 10.9's updating modeling of intersection areas from dense rasters to sparse instances. Karpathy raised the issue back on AI Day:

​

For example, our outputs are dense rasters, and it's actually pretty expensive to post-process some of these dense rasters in the car. And of course, we are under very strict latency requirements, so this is not ideal. We actually are looking into all kinds of ways of predicting just the sparse structure of the road -- maybe point-by-point or in some other fashion that doesn't require expensive post-processing.​

​

The specific quote was about roads, but it seems likely it'll get applied to other predictions like moving objects and vulnerable road users to improve perception and modeling of behaviors. With 10.10.2, I saw a car drive past a parked delivery truck, and the visualization showed the two merged together as a single large truck changing into a short truck approaching then finally as a car and truck.

 

[drtimhill]

What do you think the current FSD Beta 10.x is lacking to replace standard Autopilot (in particular driving on highways)? I've used FSD Beta driving 65mph+ on highways, so both the FSD Beta perception and planning are in use already. But indeed I don't think I've had a situation where it needed to go around a pedestrian at those speeds to see how it behaves.

Tesla is probably well aware of phantom braking issues, and potentially false positive predictions on the production Tesla Vision stack are significantly reduced with FSD Beta 11 stack incorporating multiple camera videos and memory types. That could also be a reason to work on single stack and get it ready to deploy to the whole fleet to solve numerous issues with standard Autopilot.

I think you've answered your own question.

It's also a matter of rationalizing. Tesla dont want to support two disparate NN stacks long-term, nor take the hit on running both on the cars computers (which are reaching a resource limit) at the same time. Using one stack means that vision improvements benefit all users o the car, from AP all the way to FSD. And FSD is far better at handling phantom braking type issues since the FSD stack has far more information about its surroundings and the projected paths of other vehicles.

 

[EVNow]

nor take the hit on running both on the cars computers

Only one set needs to run at a time ....

 

[drtimhill]

Only one set needs to run at a time ....

I thought about this, and I'm not so sure. The car switches back and forth between stacks pretty quickly (and drops back to the pre-FSD stack if road conditions look bad), and I'm not sure if it can swap the stacks in/out AND have time for the "incoming" stack to build up context from the cameras in this time. I suspect (and am happy to be proved wrong) that both stacks are co-resident and running all the time.

 

[Mardak]

nor take the hit on running both on the cars computers (which are reaching a resource limit) at the same time

Oh indeed, that would be in line with the various FSD Beta 11 improvements for performance (latency and throughput): replacing bag of points, switching to raw images, optimizing C->FSD computer compilers, etc. If we just randomly guess 20% single node capacity goes to running pre-FSD stack, that's quite a bit of additional performance from getting rid of the redundant stack -- maybe enough for FSD Beta to handle up to 90mph and 1 follow distance.

That probably makes "single stack" a hard requirement in that if there is no longer a fallback Autopilot stack, FSD Beta needs to be able to handle "everything" at a quality that is at least comparable in the especially tougher/more-dangerous corner cases, e.g., poor weather, high speed.

 

[EVNow]

That probably makes "single stack" a hard requirement in that if there is no longer a fallback Autopilot stack, FSD Beta needs to be able to handle "everything" at a quality that is at least comparable in the especially tougher/more-dangerous corner cases, e.g., poor weather, high speed.

In poor whether they degrade the features - FSD can definitely just keep lane and stop for lights and such In poor whether.

 

[Mardak]

In poor weather they degrade the features - FSD can definitely just keep lane and stop for lights…

Something changed with I believe FSD Beta 10.8 as I know with 10.3, it stayed active even though it was making all sorts of bad decisions when the roads were wet at night. Now FSD Beta reverts to standard Autopilot when it detects poor weather, but yes even then, Tesla could detect and fully degrade with flash big red "take over immediately" message if they don't want to partially degrade for now.

On a separate note for lights, I believe some visualizations, e.g., road markings, waste bins, traffic lights, are still powered by the old stack as part of "Full Self-Driving Visualization Preview." Notably, I've encountered multiple incorrectly visualized traffic lights where FSD Beta's driving behavior still correctly stops or goes. Similarly, turn arrows other visualization preview can show up but then get fully or partially painted over by FSD Beta visualizations drawing the ground at a different elevation.

 

[EVNow]

Something changed with I believe FSD Beta 10.8 as I know with 10.3, it stayed active even though it was making all sorts of bad decisions when the roads were wet at night. Now FSD Beta reverts to standard Autopilot when it detects poor weather, but yes even then, Tesla could detect and fully degrade with flash big red "take over immediately" message if they don't want to partially degrade for now.

Yes, I know what it does now.

All I'm saying is ... the AP functionality that they now fall back on - can be easily implemented in the single stack as the fall back mechanism. They are just deleting automatic navigation.

On a separate note for lights, I believe some visualizations, e.g., road markings, waste bins, traffic lights, are still powered by the old stack as part of "Full Self-Driving Visualization Preview." Notably, I've encountered multiple incorrectly visualized traffic lights where FSD Beta's driving behavior still correctly stops or goes. Similarly, turn arrows other visualization preview can show up but then get fully or partially painted over by FSD Beta visualizations drawing the ground at a different elevation.

I think the FSD perception stack still uses some of the old AP networks or is a copy of the AP NN that has been enhanced.

 

[Mardak]

can be easily implemented in the single stack as the fall back mechanism. They are just deleting automatic navigation

I would hope so yes, but if it's anything like FSD Beta 10.3 behavior I experienced in wet nighttime situations, it probably isn't safe to fall back to "just" lane keeping. It kept swerving when it should have kept straight often steering into medians and curbs (let alone suddenly exiting the lane into oncoming traffic).

My guess is that 10.3's road layout prediction related networks were recently retrained for some new architecture, and there was not as much training data yet for these situations. Hopefully they've collected more data since then to train the new FSD Beta 11 stack and don't regress even more, but curious how FSD Beta 10.x still falls back to old stack. That's why I was suggesting they might even completely degrade to human must drive.

It's not too surprising as what the cameras were seeing can look quite different:

 

[EVNow]

https://twitter.com/x/status/1491482919767924737

 

[mtbwalt]

Apologies if this is a well known concept, but seeing you mention “road layout prediction” made me wonder… does Tesla’s driving stack incorporate “memory” about past driving experience in any given location (either by the individual vehicle or all teslas in aggregate)?

I know that Elon has touted “pure vision” as how humans drive. But really, we use memory a whole lot in much of our driving. We have taken that same traffic circle or turn by the house hundreds of times and know exactly how to approach it. We humans are much more cautious and limited in driving in complex areas we have never driven before.

Is the pure vision FSD Tesla AI stack seeing that traffic circle for the first time every time, or is it incorporating past behaviors and flows of vehicles in determining how to proceed, mimicking the habit and memory of a human driving a familiar path?

 

[EVNow]

My interpretation - 11 isn’t coming soon … ?

https://twitter.com/x/status/1500329470300876801

 

[tmoz]

My interpretation - 11 isn’t coming soon … ?

https://twitter.com/x/status/1500329470300876801

I'm guessing he means "downtown-like orthogonally gridded" when he says "city streets". Which means no curvy roads, no flakey markings, no unprotected lefthand turns, etc. Similar to what Waymo deals with.

 

[EVNow]

I'm guessing he means "downtown-like orthogonally gridded" when he says "city streets". Which means no curvy roads, no flakey markings, no unprotected lefthand turns, etc. Similar to what Waymo deals with.

I posted this interpretation in the tweets thread.

To me it sounds like he is saying - beta not good enough to merge with highway stack - but close. So, they will wait till it’s good enough before 11.

Everything else is fluff.

 

[drtimhill]

Apologies if this is a well known concept, but seeing you mention “road layout prediction” made me wonder… does Tesla’s driving stack incorporate “memory” about past driving experience in any given location (either by the individual vehicle or all teslas in aggregate)?

I know that Elon has touted “pure vision” as how humans drive. But really, we use memory a whole lot in much of our driving. We have taken that same traffic circle or turn by the house hundreds of times and know exactly how to approach it. We humans are much more cautious and limited in driving in complex areas we have never driven before.

Is the pure vision FSD Tesla AI stack seeing that traffic circle for the first time every time, or is it incorporating past behaviors and flows of vehicles in determining how to proceed, mimicking the habit and memory of a human driving a familiar path?

Short answer .. no. You can expect the car to make the same mistakes at the same location every time (though the environment as well as some of the random jitter in some of the algorithms will lead to some variation). In the future its quite possible that the car may "learn" the layout of junctions as it approaches them multiple times from different angles (and, as you note, this would be ideal if it formed a collective memory across the fleet). However, this wont be coming any time soon (years at least).

 

[JulienW]

....Is the pure vision FSD Tesla AI stack seeing that traffic circle for the first time every time, or is it incorporating past behaviors and flows of vehicles in determining how to proceed, mimicking the habit and memory of a human driving a familiar path?

It does seem counterintuitive and VERY un-human like that it is seeing everything for the "first" time EVERY time and not learning but.......You also can't have each car learning on its own. This would mean some cars would be better at driving than other cars. That would make even less sense since choosing to buy a used car could be more expensive because it was a "smart" FSD driver or buying a new car that was "dumb" and yet to learn.

 

[sleepydoc]

Short answer .. no. You can expect the car to make the same mistakes at the same location every time (though the environment as well as some of the random jitter in some of the algorithms will lead to some variation). In the future its quite possible that the car may "learn" the layout of junctions as it approaches them multiple times from different angles (and, as you note, this would be ideal if it formed a collective memory across the fleet). However, this wont be coming any time soon (years at least).

Yes. Virtually all ‘artificial intelligence’ is not that at all, it’s computer pattern matching according to programmed parameters. ‘Learning’ implies pattern recognition or recognizing a situation as the same or similar to one you’ve faced before. Beyond that it requires remembering what worked and didn’t work in past attempts to improve the outcome. Like you said, there is a bit of ’fuzziness’ due to due to environmental variables but Tesla seems to fall pretty squarely in the pattern matching category.

 

[sleepydoc]

I posted this interpretation in the tweets thread.

Here’s a link: Elon: FSD Beta tweets

I posted this in that thread, too, but has anyone had FSD consistently navigate city intersections well? I’m not talking about stoplights but rather 4 way stops, 2 way stops, yields, etc? IME the car is so slow and unsure that the only way I’ll let it do it unaided is if there’s no traffic around. Otherwise I virtually always have to ’goose’ the accelerator. It seems like a little thing but is there’s more to it than it would seem and still means the car is failing.

 

[Mardak]

My interpretation - 11 isn’t coming soon … ?

https://twitter.com/x/status/1500329470300876801

Hmm… "We are almost at the point… that we can turn our attention to applying the code to highways" sounds like they haven't started or that it's not the top priority yet. There's definitely improvements to FSD Beta that would be quite nice for highway Autopilot:

• 10.3's "Allowed more acceleration when merging from on-ramps onto major roads and when lane changing from slow to fast lanes" would avoid needing to manually accelerate for interchanges
• 10.5's "Enabled behavior to lane change away from merges when safe to do so" instead of manually initiating lane changes from onramp traffic
• 10.6's "Allow more room for longitudinal alignment during merges by incorporating modelling of merge region end" helps avoid some incorrect map positioning of onramp merges (that currently result in unnecessary yielding / braking)
• 10.8's "Improved integration between lead vehicle overtake and lane change gap selection" should have smoother and more appropriate lane changes, and in general FSD Beta's better 360º perception especially with large trucks and changing lines that currently result in abort
• 10.9's "Made launches from stop more calm when there is an imminent slowdown nearby" hopefully means in stop-n-go traffic, it no longer uncomfortably jerks the passengers over and over again
• 10.10's "Smoother fork maneuvers and turn-lane selection using high fidelity trajectory primitives" will fix the unnecessary lateral jerk for taking offramps

All of those issues with highway Autopilot we encountered on a nearly 300 mile trip yesterday, and each time they happened made me want FSD Beta 11 single stack so much more realizing FSD Beta has had significant improvements even since October public beta 10.2.