FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[tmoz]

And this is not encouraging, unless FSD is renamed FSDEFS (Except For Snow)

Robo Taxis from June to September when north of the 43rd parallel.

 

[Gbills]

And this is not encouraging, unless FSD is renamed FSDEFS (Except For Snow)

 

[gsmith123]

Yep, that video pretty much matches my experience in snow in Colorado. It's very hard to tell in video but the car drives far too fast in slippery conditions. I disengage for speed issues as often as path issues.

I'm actually not too discouraged by this. To me it's pretty obvious they haven't put much work into making the planner to handle low traction environments. The car clearly assumes full traction in the velocity profile for turns, for instance. In safe situations it can be rather funny when the car turns abruptly while trying to accelerate, slides, gets 5-10deg drift angle in the slide and promptly throws the red hands take over immediately. WEEEEEEEE!! (Be carefully playing with this!)

The perception is a lot better than the planner in snow. It does a fairly decent job of correctly perceiving drivable space, large snow piles, snow on the edge of the road, guessing where lane lines might be, etc. However when it's even a little unsure of the pathing you get pretty wild jerky decisions. In snow humans mostly pick a path and smoothly drive it, ignoring the lanes at times. FSD doesn't work like that at all.

When it's good in snow (which is... sometimes), it really is good which is pretty cool. I'm kind of amazed it works as well as it does for probably not having too much R&D effort focused specifically on snow yet.

 

[powertoold]

I've watch some snow fsd videos. You can just manually lower the max speed in most cases.

 

[MindBlend]

I've watch some snow fsd videos. You can just manually lower the max speed in most cases.

Sure, but every time you reengage FSD it jumps up to the posted speed and you've got to quickly lower it. That still doesn't change its tendency to be jerky and assume full traction.

 

[gsmith123]

I've watch some snow fsd videos. You can just manually lower the max speed in most cases.

You can and that works fine when FSD is engaged. What gets rather annoying is when you first engage FSD you have to franticly scroll down (or tap the speedometer) if it's accelerating up to 30mph in conditions it should go 15. And in really bad conditions where you're constantly disengaging you have to do this every time! It makes testing a pain!

When the car passes a speed limit change it also resets back to the speed limit and you have to scroll down again.

And if you disengage via steering wheel the car reverts back to TACC which will accelerate you back up to the speed limit too (I don't know why the speed adjustments don't stick in this case...) If you've slowed down FSD a lot on snow you really can't disengage via steering wheel only, gotta hit the brake or stalk. I've gotten into some slightly sketchy situations with this where the car drove off through deeper snow or slick patches on the edge of the road, I intervened via steering wheel, and it promptly gives move acceleration. Very annoying.

There's a lot of little quality of life stuff that's not really part of the core FSD mission but could make winter testing a lot better.

 

[powertoold]

could make winter testing a lot better.

Yes, although I think fsd driving well in snow is low on their priority list.

As for the speed limit thing, I think this is how it works:

1) When it see's a new speed limit (like 25 -> 30), it resets your max speed.
2) If it sees the same speed limit (25 -> 25), it keeps the same speed you set.
3) If you disengage, then it resets the limits

 

[Mardak]

in really bad conditions where you're constantly disengaging you have to do this every time! It makes testing a pain!

You can set the offset to be a relative percentage, e.g., +10%, or absolute difference, e.g., -10mph. If conditions are good enough to allow Autopilot to run, maybe try switching between relative and absolute as it'll remember each offset's settings, so you basically have a toggle for not-bad-weather and bad-weather.

My experience in driving in snow/ice is that FSD Beta perception does a surprisingly decent job treating plowed snow piles as road edges, and unplowed streets with lines covered are fine too. However, roads with inconsistent patches of road and/or snow can result in dangerous swerving (so keep your hands tight on the wheel to prevent any turning) because it's trying to decide if it's a wide road fully driveable vs narrow road following tire tracks. I believe the Occupancy network has detected piles of snow to go around even though they would have been fine to drive over. Of course, the speed control isn't great, so if the stopping area or turn is covered in ice, disengage if FSD Beta wouldn't slow down soon enough.

I think current FSD Beta would actually be better than the legacy highway stack in quite a few snowy conditions such as high speed driving where driving lanes are basically clear but lines are covered in snow and even lower speed situations where lanes are completely covered and people shouldn't be driving highway speeds anyway. Hopefully we'll be able to try it out with FSD Beta 11 soon.

 

[EVNow]

And this is not encouraging, unless FSD is renamed FSDEFS (Except For Snow)

Yet he doesn't even hold the wheel all the time !

 

[AlanSubie4Life]

And this is not encouraging, unless FSD is renamed FSDEFS (Except For Snow)

Don’t care at all that it cannot handle snow. No need to use it there.

However, the interesting thing here is that it does show the fragility of the current pattern matching and how inferior it is to human neural ability (no surprises there of course). It is surprisingly good at identifying road edges - but not good enough at all.

Definitely does not appear they will to get to human-level ability without some sort of revolutionary change (or much more hardware capability, maybe).

 

[Supcom]

Don’t care at all that it cannot handle snow. No need to use it there.

However, the interesting thing here is that it does show the fragility of the current pattern matching and how inferior it is to human neural ability (no surprises there of course). It is surprisingly good at identifying road edges - but not good enough at all.

Definitely does not appear they will to get to human-level ability without some sort of revolutionary change (or much more hardware capability, maybe).

Given that Tesla has not addressed FSD operation in snow as far as I can recall, any ability on snow would be more luck than anything.

 

[AlanSubie4Life]

Given that Tesla has not addressed FSD operation in snow as far as I can recall, any ability on snow would be more luck than anything.

Right. It was more interesting to see how it does in a situation where visibility or marking are very poor (one of the reasons results can vary so much from person to person and drive to drive).

 

[aerodyne]

Given that Tesla has not addressed FSD operation in snow as far as I can recall, any ability on snow would be more luck than anything.

They didn't say it couldn't either. Would put quite a ding in your robo taxi income if it did not work in winter in the snow belt.

 

[JulienW]

They didn't say it couldn't either. Would put quite a ding in your robo taxi income if it did not work in winter in the snow belt.

FSD Beta is a L2 system so it doesn't need to say anything since it is NOT driving the car.

First Step: Tesla needs to get a fair weather functioning L3 system and then work their way to a L4 system which could handle snow. So handing snow is still a ways away.

 

[AlanSubie4Life]

So handing snow is still a ways away.

For any manufacturer or robotaxi service it is probably many years away. Except for food (a problem which can be planned for in advance) for most people there is no longer any reason to drive in snow anyway. It’s generally not a good idea if you don’t have to.

Seems like a non-issue. Plenty of real problems to solve, like stopping and going. Remarkably, steering seems to be opening up a big lead on these other operations, in terms of success rate. Though maybe that’s just a false impression on my part since it happens less often.

In any case I don’t think the snow storm case will be addressed by v11.x. Nor does it need to be. Seems highly irrelevant except to the extent mentioned above where it reveals fragility.

 

[jebinc]

FSD Beta is a L2 system so it doesn't need to say anything since it is NOT driving the car.

First Step: Tesla needs to get a fair weather functioning L3 system and then work their way to a L4 system which could handle snow. So handing snow is still a ways away.

A very long ways…

 

[JulienW]

A very long ways…

....but which comes first?

• V11
• Tesla Bot Optimus
• Inclement/snow weather driving
• Roadster
• Robotaxies
• Cyber Truck
• Tweeter utopia/everyone happy/Musk back to Tesla
• TSLA most valuable company in the world

 

[jebinc]

....but which comes first?

• V11
• Tesla Bot Optimus
• Inclement/snow weather driving
• Roadster
• Robotaxies
• Cyber Truck
• Tweeter utopia/everyone happy/Musk back to Tesla
• TSLA most valuable company in the world

Forgot one…. Usable auto wipers…

 

[Jcl007]

Forgot one…. Usable auto wipers…

They usually work for me. Better then the Honda auto wipers.

 

[Mardak]

So handing snow is still a ways away

Yeah, I don't expect Tesla to put specific driving behavior changes for snow yet, but some snowy situations will be included as part of their training data collection. This means we'll need to keep disengaging to slow/stop for turns, etc. instead of letting it do its usual dry-roads behaviors.