FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[OncomingStorm]

Except that nothing like this have ever been mentioned at AI day or any other blogs/posts by Karpathy. And it is inconsistent with the deep learning model in use. Training is done in massive data centers and once complete, the static network is deployed to the fleet. There is no evidence to suggest any other architecture.

thats why i quoted learning lol as it wouldn't be learning per se, more the car noticing discrepancies in the data and tagging them for review, but depending on how thats implemented it could appear that the car "learned" something due to the annotations it made as part of that process resulting in minor changed behavior that may be transitory. again just random speculation but it makes sence, supports some anecdotal evidence, it wouldn't be something they'd brag about cause its not exactly revolutionary lol, and it wouldn't be contrary to any methodologies they currently employ.

 

[verygreen]

Here’s one spot - when heading northeast on 101 the car will very reliably enter the right turn lane, despite needing to go straight (and the map indicating that it’s going straight.) At the end of the turn lane it will sometimes veer back Into the driving lane or sometimes continue straight on the shoulder until the shoulder disappears for the next turn lane, then it gets confused and starts squaking.

The issues with turn lanes are pretty well known with 10.2 and a new bug but regardless, it hasn’t learned how to go straight.

Google Maps

Find local businesses, view maps and get driving directions in Google Maps.

www.google.com

View attachment 788247

yup, that's a common failure mode, I see those often too.

This sort of data is not stored in the temp bufffer though, the kind of data that's stored is mostly from the BEV nets that include drivable vs undrivable space.
So in this example the drivable turning lane confuses car forever, but in my example on pass2, the car already knows there's no more way forward so it takes a turn.

 

[boonedocks]

Yes, it’s very doubtful that the neural nets are actually learning anything in the car as (as in the neutral net weightings being changed). However, it would seem sensible for the car to locally store additional hints and enhanced map-related detail data that it “learns” while repetitively driving streets and intersections. I have no idea if it actually does this today.

Would be nice but it certainly does not do this.

 

[EVNow]

Would be nice but it certainly does not do this.

There certainly isn't any proff for what you say.

But there is some anecdotal proof for the opposite. Apart from what verygreen posted, there are some tweets from an industry consultant who did some experiments with multiple cars on the same route from a couple of years back - and I've posted those here. I've to dig up the tweets.

 

[boonedocks]

There certainly isn't any proff for what you say.

But there is some anecdotal proof for the opposite. Apart from what verygreen posted, there are some tweets from an industry consultant who did some experiments with multiple cars on the same route from a couple of years back - and I've posted those here. I've to dig up the tweets.

Just literal thousands of miles driven on it is all the proof I need.

Anecdotal is the placebo affect.

People really do see what they want to see not the reality. Especially if they are relying on what they “read” and not what they experience.

 

[GlmnAlyAirCar]

There certainly isn't any proff for what you say.

But there is some anecdotal proof for the opposite. Apart from what verygreen posted, there are some tweets from an industry consultant who did some experiments with multiple cars on the same route from a couple of years back - and I've posted those here. I've to dig up the tweets.

There was no FSD beta a couple of years back

 

[EVNow]

There was no FSD beta a couple of years back

AP was there ...

 

[EVNow]

Just literal thousands of miles driven on it is all the proof I need.

Anecdotal is the placebo affect.

People really do see what they want to see not the reality. Especially if they are relying on what they “read” and not what they experience.

But you are driving with the same vehicle. The experiment was with using a different vehicle (on the same firmware).

Get a different vehicle from a friend and drive your routes (assuming that vehicle doesn't use those routes) - see if there are any differences.

The claim is not that your car "learns" your route and drives it perfectly.

 

[momo3605]

But you are driving with the same vehicle. The experiment was with using a different vehicle (on the same firmware).

Get a different vehicle from a friend and drive your routes (assuming that vehicle doesn't use those routes) - see if there are any differences.

The claim is not that your car "learns" your route and drives it perfectly.

FSD Beta will fail certain intersections and turns over and over again. This is very easy to see and I’m sure you have the same experience. How would the car be “learning” if it continues to fail every pass through? Depending on conditions day or night or maybe having a lead car could lead to slightly different behavior.

Are you saying after driving on your normal routes after a couple times your car handles it perfectly and is fully autonomous with 0 disengagements or that every time you take the same route on the same version it does better than before?

 

[jeremymc7]

I would think that pretty the report button should cause the car AI to say to itself okay if I’m here going in this direction I need to do it slightly different and see if I get a report again.

But that’s obviously not the case.

 

[boonedocks]

I would think that pretty the report button should cause the car AI to say to itself okay if I’m here going in this direction I need to do it slightly different and see if I get a report again.

But that’s obviously not the case.

100% true. I hit that “report button” so many times that I have just given up on it. The “training” system either will get it right from all the submissions or it won’t.

Local vehicle learning obviously is not happening.

It is kind of comical to see the posts of people really believing that their individual car is actually “learning” and correcting future experience in the same spots that it doesn’t handle right in previous drives.

It will hopefully with future updates but not on the same software version at present.

 

[GlmnAlyAirCar]

AP was there ...

But that’s not we are talking about

 

[EVNow]

But that’s not we are talking about

FSD beta is an evolution of city AP.

 

[drtimhill]

I would think that pretty the report button should cause the car AI to say to itself okay if I’m here going in this direction I need to do it slightly different and see if I get a report again.

But that’s obviously not the case.

No it isnt, and it shouldn't .. the AI (which isnt "intelligence", AI should really be called Artificial Mimicry) simply obeys its training. In fact it would be very bad if it "learned" from its driver .. can you imagine the result in the hands of a teenager!

What happens is your reports go back, are triaged, and added to the corpus of training data used to train the next release of the NN (its more complex than this, but that is basically the result).

 

[EVNow]

I would think that pretty the report button should cause the car AI to say to itself okay if I’m here going in this direction I need to do it slightly different and see if I get a report again.

But that’s obviously not the case.

FSD Beta will fail certain intersections and turns over and over again. This is very easy to see and I’m sure you have the same experience.

I think you guys are approaching it the wrong way.

Nobody is claiming the car learns 100% to correct all the issues. It doesn't - though they could potentially correct a lot of things if they put their minds to it. But that is not "universal FSD" that Tesla apparently wants to develop.

But - here is the crux - some people report that Tesla does have some local metadata to help them. This is more like a "crutch" and they shouldn't even be using it ! I tend to believe they do use it (or used to) sparingly ... or may be they just experimented with it earlier.

 

[GlmnAlyAirCar]

FSD beta is an evolution of city AP.

Ok I need to stop wasting my time responding to you. First of all, there is no such thing as city AP. Highway AP that happens to work to some degree on surface streets. It is an entirely different design and implementation than standard AP. Different code base. Developed at different times but largely different teams.

 

[EVNow]

Ok I need to stop wasting my time responding to you. First of all, there is no such thing as city AP. Highway AP that happens to work to some degree on surface streets. It is an entirely different design and implementation than standard AP. Different code base. Developed at different times but largely different teams.

Ofcourse it is just highway AP. That’s why they highly publicized adding stop sign and traffic light to it. And even recognized revenue based on that.

Some people just want to read obsolete manuals and take that as the only truth.

 

[EVNow]

Are you saying after driving on your normal routes after a couple times your car handles it perfectly and is fully autonomous with 0 disengagements or that every time you take the same route on the same version it does better than before?

Here is the old anecdotal report I was mentioning.

The tweeter helps large investors (think $1B+) investigate and analyze companies before investing. He has a team of engineers who investigate all parts of the car (like Munro, but private).

"We observed one vehicle improving in ability when driven over the same roads over time"

I don't know how many of you follow SoylentBrown on twitter - but he had an interesting tweet the other day. Essentially it says, the car seems to learn the route it is driven on and drives better on AP after some time. From what is known generally about NN and what Tesla has told us, this seems...

teslamotorsclub.com

https://twitter.com/x/status/1162734085695070211

 

[Mardak]

there are multiple factors, maps influence things

Are there indications of less reliance on maps or a sense of lower map quality in different regions? FSD Beta 10.11 explicitly mentions "modeling intersection extents is now entirely based on network predictions and no longer uses map-based heuristics," and I would guess that was in preparation for expanding FSD Beta to Canada where maybe the map data is not as accurate? There are map inaccuracies in the US as well, but maybe it was bad enough for Canada and important enough to expand FSD Beta internationally in Q1 that these predictions got prioritized into a 10.x release pulled in from FSD Beta 11.

 

[Mardak]

FSD Beta 10.12 release notes seem to indicate a lot of neural network architecture changes that generally improve 3D/geometric understanding at higher resolutions and from further distances. These probably increased the compute requirements for the FSD Beta networks, but even then, they achieved a net +1.8 frames per second by removing 3 legacy networks.

As we've seen with other 10.x releases, there's plenty of technical improvements that are needed to achieve the product requirement for releasing FSD Beta 11 / Single Stack, which would most likely remove even more legacy networks to allow more compute for FSD Beta networks.

Removing legacy networks means they can't be used as an immediate fallback, e.g., currently FSD Beta can revert to basic Autopilot in poor weather and when switching to highway driving, so Tesla must believe/know the new networks are at least as good as what's replaced. This might have also resulted in the extended development time for 10.12 as removing safety-nets probably has a higher threshold.

@verygreen, do you have insights into which networks are (already?) removed? This could give a sense of where Tesla feels FSD Beta is already more capable as well as when 10.12 and future versions get released.