FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[kabin]

I did

Based on my experience and many others, - it's nothing to do with a hardware issue, and everything to do with software. I don't remember which version it was where wipers became horrific for me, - but a couple of releases back. Until then, I'd had no real issues with the auto wipers. I knew, before I even updated my software, that people with the same car as me had complained about the wiper regression with that specific software. I still hit install on that update, and since that exact day, my wipers have been near unusable, either dry wiping or not wiping in the rain. In Seattle, it's total pita this time of year. Now, this software update may play differently with different hardware revisions installed on these cars, but it's still a root cause software issue, - no doubt in my mind.

Yep. I had to do some digging. Elon was talking about photon counts over a year ago but 10.69.25 it was finally released. I noticed some complaints about questionable 'degraded system performance' alerts thereafter. I've seen complaints for both HW3 and HW4 so that seems to be a red herring.

Interesting Elon apologized about autowipers in May 2023 but still no resolution.

 

[WilliamG]

I did


Yep. I had to do some digging. Elon was talking about photon counts over a year ago but 10.69.25 it was finally released. I noticed some complaints about questionable 'degraded system performance' alerts thereafter. I've seen complaints for both HW3 and HW4 so that seems to be a red herring.

Interesting Elon apologized about autowipers in May 2023 but still no resolution.

It's interesting that he apologized, but not at all surprising this happened in the first place. The vision system is fundamentally flawed. Model 3 was released in 2017, we're almost in 2024 and there's still not what I would consider satisfactory wiper performance in all conditions like you get with a "traditional" IR sensor automatic wipers should have. Elon can talk the talk about simplification, but with the wipers/vision system he got it wrong, and now can't backtrack without risking the wrath of millions of owners.

 

[kabin]

It's interesting that he apologized, but not at all surprising this happened in the first place. The vision system is fundamentally flawed. Model 3 was released in 2017, we're almost in 2024 and there's still not what I would consider satisfactory wiper performance in all conditions like you get with a "traditional" IR sensor automatic wipers should have. Elon can talk the talk about simplification, but with the wipers/vision system he got it wrong, and now can't backtrack without risking the wrath of millions of owners.

Yep. The team's apparent goal of using NNs for anything jumped the shark on this one. As they say, keep it simple stupid.

 

[WilliamG]

Yep. The team's apparent goal of using NNs for anything jumped the shark on this one. As they say, keep it simple stupid.

Yep, and keeping it simple would mean keeping the IR sensor for the wipers. That's a proven system, and there was no reason to replace it other than $. Then again, it is showing consistency for Tesla, - replacing a working technology with a work-in-progress. The number of instances where this is true is... well.., almost innumerable.

 

[AndrewZ]

For the wipers, we need to be careful to not miss the forest for the trees here. We're on the cusp of SAE Level 4-5 robotaxis, and a windshield won't even be necessary for vehicles that are being entirely driven by AI + Vision.

Next gen vehicle likely won't have a steering wheel, won't have pedals, and probably won't even have a windshield, so Tesla should direct its efforts thusly

 

[fasteddie7]

Wouldn’t the cost of paying engineers to develop and test far exceeded the cost of the sensors up until now and in the future until another revision did it really save anything?

 

[WilliamG]

Wouldn’t the cost of paying engineers to develop and test far exceeded the cost of the sensors up until now and in the future until another revision did it really save anything?

Not to mention the cost on service considering the customers going in to complain that wipers don't work.

 

[Tronguy]

Is there any hope for me and my 2020 Model X (2023.27.7 / FSD 11.4.7.3)? I have to tug on the wheel relentlessly. Just a wee bit too hard and it disconnects. Not enought and it penalizes you. Three of those and you're in jail. I've never been able to drive on the freeway even for more than an hour without being dinged. Almost useless. I don't f-around in the car I look out, no phone use, no looking at Spotify on the display, whatever. I don't think I have any way to send voice or video feedback from the system. What am I missing?

I've got two ways to solve the problem, both of which seem to work:

• Drive with one hand at either the 270 or 90 degree position. The jiggling that the car does as it goes over the ground will be enough to keep the car from complaining that one doesn't have ones hands on the wheel.
  • During a long drive, one hand on, say, the 270 degree position still results in the car complaining that ones hands aren't on the wheel. Switch, and it goes away, again, for a longish while.
  • It bugs me to drive one-handed, even if it is the car doing the driving. All those drivers' ed courses talk about safety being Two Hands, which brings us to the next item:
• Drive with two hands in their normal position. However, with breathing, first lean gently towards the left, then lean gently towards the right. Make it a habit. After doing this for a while, one doesn't think about it and the car won't complain about being hands-free.

On various versions of FSD-b, the algorithms used by the car to determine whether one's hands were present or not definitely went through revisions. Even with the above, in mid 2022, there were FSD-b versions that, seemingly, would get more and more sensitive over time, to the point where the window of torque to prevent the nag and the amount of torque where FSD-b was disengaged became zero. Thankfully, that got improved over the next couple of revs. I think Tesla was trying to build software to detect those "defeat" devices that put a weight on the wheel and was having trouble doing so.

 

[KLam]

I think the dry wipes have nothing to do with rain sensing.

 

[EVNow]

Wouldn’t the cost of paying engineers to develop and test far exceeded the cost of the sensors up until now and in the future until another revision did it really save anything?

Probably ... but gave them a lot of testing material for FSD/AP.

 

[AndrewZ]

There might be reasons beyond raw cost, like parts supply -- without knowing when exactly the wipers went Vision

But like for the ultrasonic sensor removal, I'm still in the school of thought it was necessary to navigate supply chain problems. On the surface it seems like a relatively small cost impact, but vehicles can't be delivered at all if they're sitting in lots waiting for a couple parts.

Pretty good position to be in where you can cut parts and promise some future magical OTA update that will restore functionality

 

[Ramphex]

Not necessarily FSD related but, my woman hates FSDj and so she won’t use AP in her car because of it. Convinced her to use AP today while driving and it was good for about 10 mins and she seemed to show interest in liking it. Then it jammed the brakes hard on Interstate 275 doing 70mph, woke up the baby in the back, and now I’m pretty sure she’ll never use it again

 

[ItsNotAboutTheMoney]

Not necessarily FSD related but, my woman hates FSDj and so she won’t use AP in her car because of it. Convinced her to use AP today while driving and it was good for about 10 mins and she seemed to show interest in liking it. Then it jammed the brakes hard on Interstate 275 doing 70mph, woke up the baby in the back, and now I’m pretty sure she’ll never use it again

Once again showing why they need a selectable dumb cruise mode as many other manufacturers have. The hubris is infuriating.

 

[powertoold]

Once again showing why they need a selectable dumb cruise mode as many other manufacturers have. The hubris is infuriating.

Dumb cruise bad idea in Tesla, confusion lead to accidents, oops thought I was in TACC

 

[EVNow]

Once again showing why they need a selectable dumb cruise mode as many other manufacturers have. The hubris is infuriating.

People still use that ?

Fundamentally the issue is collision avoidance. You don't want your car in 2023 to be hitting cars in the front like they did in 1923. But that collision avoidance comes with false positives ...

 

[Ramphex]

If there was a dumb cruise option that would just hold the lane for me, I would use that over the current crapshoot. Sending me into seat belt lock coming up on red lights is just impossible to “deal with”.

 

[Twiglett]

People still use that ?

Fundamentally the issue is collision avoidance. You don't want your car in 2023 to be hitting cars in the front like they did in 1923. But that collision avoidance comes with false positives ...

Even worst - that collision avoidance is built into the current safety testing so is essentially required to get good safety ratings.

 

[Bob Denny]

I've got two ways to solve the problem, both of which seem to work:

• [...]

On various versions of FSD-b, the algorithms used by the car to determine whether one's hands were present or not definitely went through revisions. Even with the above, in mid 2022, there were FSD-b versions that, seemingly, would get more and more sensitive over time, to the point where the window of torque to prevent the nag and the amount of torque where FSD-b was disengaged became zero. Thankfully, that got improved over the next couple of revs. I think Tesla was trying to build software to detect those "defeat" devices that put a weight on the wheel and was having trouble doing so.

Thanks for the suggestions. After the earlier reply I tried again and got dinged flashflash BEEBEE with my hand lightly on the wheel, etc. I might put a balaclava on and see if I notice a difference ha ha ha ha. Does my cabin camera see anything in detail???

 

[KLam]

Even worst - that collision avoidance is built into the current safety testing so is essentially required to get good safety ratings

Yup, its a question of reducing the amount of false positives. If not implemented correctly could result in less safe. i.e. when a real collision is not detected.
I would rather deal with some false positives.

 

[KLam]

Thanks for the suggestions. After the earlier reply I tried again and got dinged flashflash BEEBEE with my hand lightly on the wheel, etc. I might put a balaclava on and see if I notice a difference ha ha ha ha. Does my cabin camera see anything in detail???

Try this: Don't put your hands on the steering wheel when you engage FSD, do nothing while the system asked you to put hands on the steering wheel. The system should engage as if its OK. Then when warnings comes just touch the steering wheel lightly until you feel some resistance and the warning goes away.

Also don't keep your head still and stare to the front. Move your head frequently.

Hope this helps. If not, you may want to schedule a service to check on you front cameras, and the steering sensor. There were some reports of MY camera module not installed correctly.