FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[AlanSubie4Life]

I often use prescription sunglasses and FSD cannot determine if I'm paying attention or not. My understanding is it's likely unique to my sunglasses.

Oh it can tell. It’s actually quite unnatural to look at something elsewhere in the car without turning your head. Will it detect if you are dozing off? Maybe not, if your head stays in place.

 

[Bigship]

View attachment 927168

I am now on 11.3.6 and FSDb continues to put me in the wrong lane (as with previous version). When heading NW on lakeland hills, the plan is to turn Left on 69th. And instead of waiting for the correct left turn lane it always puts me in the opposing turn lane (oncoming). That lane has a hard yellow curb, so once you are in it is nearly impossible get out. Not sure if that is mapping data or just poor planning.

YIKES!!

 

[nvx1977]

View attachment 927168

I am now on 11.3.6 and FSDb continues to put me in the wrong lane (as with previous version). When heading NW on lakeland hills, the plan is to turn Left on 69th. And instead of waiting for the correct left turn lane it always puts me in the opposing turn lane (oncoming). That lane has a hard yellow curb, so once you are in it is nearly impossible get out. Not sure if that is mapping data or just poor planning.

i have a lot of these kinds of situations here, without the raised beds, dang. but yeah, seems like some low-def map data can definitely assist with things like this. Expecting vision to figure it out on the fly every time as if it was the first time going there seems like there will always be tons of intersections across the country that will always perform poorly. I'm really hoping that Tesla is already building crowdsourced low-def mapping. I thought I read somewhere that it was a work in progress.

 

[tivoboy]

Oh it can tell. It’s actually quite unnatural to look at something elsewhere in the car without turning your head. Will it detect if you are dozing off? Maybe not, if your head stays in place.

I think we’ve seen some details about recognizing dozing, or looking directly DOWN, but what Chaps my b…t is that this latest build 11.3.6 IMHO WAAY OVER detects usage of the main screen when trying to search for a location, supercharger, etc… even just spending more than about 7 seconds doing a search, looking at availability, etc. BOOM wheel warning. Something about that, more situational awareness of the outside environment and what the user is trying to do needs to be incorporated.

 

[AlanSubie4Life]

just spending more than about 7 seconds doing a search, looking at availability, etc. BOOM wheel warning. Something about that, more situational awareness of the outside environment and what the user is trying to do needs to be incorporated.

This seems perfectly well justified.

You should not be fiddling with the screen when driving. Use voice commands if you have cell service.

I have found that when I use the screen I can miss things happening on the road.

So it is encouraging good habits.

Both hands on wheel, eyes and head mostly straight ahead, all is well.

 

[tivoboy]

This seems perfectly well justified.

You should not be fiddling with the screen when driving. Use voice commands if you have cell service.

I have found that when I use the screen I can miss things happening on the road.

So it is encouraging good habits.

Both hands on wheel, eyes and head mostly straight ahead, all is well.

What is the point of FSD then?

I can’t get EIGHT seconds to check for nearby superchargers? Check vehicle performance and power usage while driving? Disable the in car browser, FINE, but simply trying to input alternate destinations, checking routes for traffic, that can take more than seven seconds to do. Even breaking it up into pieces seems to be considered by the camera and the nag timer.

And, by the way, this is looking OVER, then looking UP, then looking OVER again, then up, back and forth back and forth but the nag timer still interprets this as ‘distracted’ and not an active enough participant to allow to continue.

And I’ll add that this is HIGHWAY only, I don’t do this type of in car nav/system interaction while in local surface streets.. frankly I stopped using this build on local surface streets about 2 days into getting this latest poor release.

If Elon is hoping for FSD to save him (Tesla) I’m more doubtful than I was last Wednesday.

 

[AlanSubie4Life]

It feels like it’s almost easier to rest one hand on your leg and place it at 7 o’clock on the wheel and give it a-little resistance than keeping both hands on the wheel with no pressure.

I just rest one elbow on the door and one on the center armrest and I can just put my hands on the wheel quite comfortably. Biggest change for me over the last few months has been getting rid of torquing when using AP. This torquing was always problematic and it is great that it is not longer needed. Took a little while to unlearn the habit. Of course sometimes I have some torque on the wheel but I am not consciously applying it continuously.

 

[AlanSubie4Life]

What is the point of FSD then?

Ummm….it helps you drive and gets rid of a lot of the constant microcorrections. This mostly applies on the freeway.

Not sure what the point of it is on surface streets yet. It’s not good enough there to evaluate whether it will be useful. But possibly still could be quite useful with eyes on road and hands on wheel. May prevent some mistakes. But it creates mistakes too, so tricky. We’ll see.

In any case you are still the driver. Seems unlikely to change any time soon.

frankly I stopped using this build on local surface streets about 2 days into getting this latest poor release.

Sure. It isn’t great. I keep using it on occasion but only on known routes, because I know on a given route where to not use it.

I was just commenting in my original post that in general the nag reduction and the lack of torque requirement is great. Maybe it still nags when using the screen, but it also sometimes did that before if you weren’t torquing the wheel. It’s true that there are new nags for detected lack of gaze though, so now it probably nags when using the screen more than it used to.

Anyway still plenty of improvement needed but relying on the camera is clearly coming along and that increasingly gets rid of the silly torque requirement the better the camera gets. And it can still enforce hands on wheel, which is also great.

 

[swedge]

What is the point of FSD then?

We are not using FSD!

We are testing FSD beta.

This is an experimental prototype. It will do the wrong thing at the worst time. If you don't want to pay attention, don't use it. Or, don't pay attention and FSD beta will disable itself for you. This is what you agreed to when you first enabled it.

That said, I completely agree that the driver monitoring needs work, like many other aspects of FSD beta.

 

[aronth5]

Oh it can tell. It’s actually quite unnatural to look at something elsewhere in the car without turning your head. Will it detect if you are dozing off? Maybe not, if your head stays in place.

Highway drive.
I've had someone else in the car with me ready to alert me if I have to take over. FSD never warns me when I'm not paying attention, but as soon as I take the prescription glasses off I will get an alert pretty quickly. I'm talking looking at the display or anywhere but the road for minutes at a time and FSD just ignores me. Only exception is if I try and use my cell phone (as a test only), FSD will then warn me. Prescription sun glasses. No hat.

 

[JHCCAZ]

Highway drive.
I've had someone else in the car with me ready to alert me if I have to take over. FSD never warns me when I'm not paying attention, but as soon as I take the prescription glasses off I will get an alert pretty quickly. I'm talking looking at the display or anywhere but the road for minutes at a time and FSD just ignores me. Only exception is if I try and use my cell phone (as a test only), FSD will then warn me. Prescription sun glasses. No hat.

Interesting test and results. I've had the impression, though not carefully tested as you did, that if the system cannot see your eyes it will then fall back to the steering wheel torque nag. There's no question that under some circumstances I get nags and I can cancel them with wheel torque. In other cases, it seems very happy if I'm looking forward for a long time without any significant torque on the wheel.

in this test, can you say that when your sunglasses were on and you report no nags, that during that interval you were or were not holding the wheel with detectable torque input?

Also for clarification, when you had your sunglasses on and you were looking "anywhere but the road for minutes at a time", was this just eyeball movement with your nose pretty much pointed forward, or were you moving your head position to different directions with no objection from the system?

 

[RowdyMY]

I've noticed that during one of my frequent drive. No matter how I set my FSD, it always switches to the left lane, at the exact same spot. This is even with Minimize turned on. I assumed it must be a mapping info error. This is on a 4-way rural road with almost no traffic light

Same issues here. There are a couple spots where to-the-foot it tries to move out of the right lane and says "changing lanes to stay out of the rightmost lane".

All 4 spots are where it was previously using the "FSDb Stack" vs the "NoA Stack"

I don't have this issue on any of my drive where it was previously NoA.

For group edification and reminder - my commute is about 62 miles each way. All highway but a mix of restricted access and not restricted access state highway where it used to bounce between the two modes.

It seems to me that there are still a few hangover issues in some spots, though overall reduced from 10.x.

 

[RowdyMY]

Reference above - there's also a unique spot where a traffic light was removed about 6 weeks ago. Repaved and fresh lines with a jersey barrier in the middle now.

Both directions I get the "stopping for traffic control in 600, 500, 400, etc" and it applies the brakes until I stalk down or press the accelerator.

Seems as though map data is still king and the absence of a negative = default to a positive.

There is no infrastructure from the signals left. If you were driving it for the first time you would never have known there was a signal.

 

[thewishmaster]

Found one intersection where FSDb full-on crashed (red hands of death etc). Visualizations went back to regular AP ones, but with no way to enable TACC/AP. Was a while before FSDb came back. I went past the same intersection on the way back, and FSDb crashed again. After the second time I couldn't enable FSDb for the rest of the evening, even after parking for a few minutes. Hopefully it comes back tomorrow, after the car goes to sleep fully.

 

[JHCCAZ]

Found one intersection where FSDb full-on crashed (red hands of death etc). Visualizations went back to regular AP ones, but with no way to enable TACC/AP. Was a while before FSDb came back. I went past the same intersection on the way back, and FSDb crashed again. After the second time I couldn't enable FSDb for the rest of the evening, even after parking for a few minutes. Hopefully it comes back tomorrow, after the car goes to sleep fully.

If you go back there again, you could consider disengaging a block ahead and leaving a voice note that you are approaching an intersection that crashes FSD. Because I'm assuming that after it crashes there's no chance to do so. I know this is a lot of work for a dubious chance that Tesla will look at it and act on it; I'm just throwing it out there.

 

[arnolddeleon]

The Home Depot near where I live has some security cameras installed with bright flashing blue lights. I think the car mistook them for a police car and tried to pull over. I'm not sure and didn't get a chance to look at the screen to see if that really happened. I'm not as dedicated as others that setup a dedicated camera recording what is going on.

Driving to and from said Home Depot gave several repeated attempts on similar conditions:

FSD seems to be "not as good" at night compared to daytime.

Lots of random blinkers (not news).

Left turns are like a box of chocolates. As previously noted, the car "wants" to cut corners. What is odd is that I've seen it take a left turn "correctly" on the same build. An example is an unprotected left from our cul-de-sac to a street which has left turn only lane and straight through lane. The car recently has taken to making the left turn into the left turn only lane and then frantically diving over to the right as the route needed to make the immediate right turn.

Little hesitations here and there. There are times where it seems that car was going to stop for a random reason. I didn't find out, I pressed go. Some of it is predictable, for example the car makes a small steering wheel wiggle when passing a particular cross walk mid street.

It seems the "moves" are better but the car still feels "behind"

 

[thewishmaster]

If you go back there again, you could consider disengaging a block ahead and leaving a voice note that you are approaching an intersection that crashes FSD. Because I'm assuming that after it crashes there's no chance to do so. I know this is a lot of work for a dubious chance that Tesla will look at it and act on it; I'm just throwing it out there.

Yep, no chance to report after the crash (no voice note popup). I hope the crashes are automatically reported together with a snapshot of the last state, but who knows.

 

[LowlyOilBurner]

What happened to 11.4.X?

 

[JulienW]

What happened to 11.4.X?

Still internal testing.

 

[JohnB007]

I often use prescription sunglasses and FSD cannot determine if I'm paying attention or not. My understanding is it's likely unique to my sunglasses.

I use a clip on glasses so I suppose it's kinda like prescription sunglasses... ?

It works well for me though. Even during dusk as sun sets and the sun constantly appears and disappears depending on what's in front of me.

Just to make things a bit more interesting, I use transition lenses, so often when I get back into my car, my lenses are already dark. Then I put the clip-on on top. It's a double dark layer between my eyes and the camera. Still haven't noticed camera acting strangely for me.