FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[KArnold]

Approaching a brand new traffic light that was recently installed but is inactive (there are bags covering the lights), FSD Beta detected the traffic light and stopped. I got the "stopping for traffic light" message

I've had that happen with planned lights that have the poles erected but no traffic lights yet, in addition to bagged lights similar to your example. I assume it's map-based which seems like a very fast update.

But there is also a 10-month old redesigned intersection where v11 incorrectly insists on using the old straight lane which is now wrong. That also appears map- based.

So I honestly don't know. My guess is map errors, not FSD itself, is going to be one of the biggest problems to solve going forward.

 

[Larry Hutchinson]

Approaching a brand new traffic light that was recently installed but is inactive (there are bags covering the lights), FSD Beta detected the traffic light and stopped. I got the "stopping for traffic light" message on the screen. I needed to press the accelerator to make the car go. So FSD beta 11 knows there is a traffic light from the cameras but does not know that the traffic light is inactive and should be ignored.

Has happened multiple times to me. Each time, even though I override , my wife exclaims “I hate your car.”

 

[FOOLSLFDRVNG]

Has happened multiple times to me. Each time, even though I override , my wife exclaims “I hate your car.”

My safety score is low and the aggressive turns are always in the red, so the choice for her is FSD or the fool driving.

 

[Supcom]

On the current V11 version, I noticed two interesting behaviors:

1) Approaching a brand new traffic light that was recently installed but is inactive (there are bags covering the lights), FSD Beta detected the traffic light and stopped. I got the "stopping for traffic light" message on the screen. I needed to press the accelerator to make the car go. So FSD beta 11 knows there is a traffic light from the cameras but does not know that the traffic light is inactive and should be ignored.

2) When approaching a railroad crossing, with no traffic, no train coming, sometimes FSD beta stops at the stop line for the railroad crossing and sometimes it just keeps going. I have no idea why FSD beta sometimes stops and sometimes does not stop as the conditions seem identical in both cases.

For safety reasons, FSD is likely trained to stop if it sees a traffic light with none of the lights illuminated. Stopping would be the safest approach and since you, the driver, are paying attention, you can easily use the accelerator to override this, when appropriate.

I have had FSD interpret a railroad signal as a red traffic light when the sun reflects off the red lenses just right.

A long time annoyance for me are flashing yellow lights strung above an intersection. FSD will often attempt to stop for these. Perhaps it interprets them as a traffic light about to turn red? This is something that has come and gone with various versions of FSD over the last couple years. It's one of the many things I look forward to testing with V12.

 

[aataskin]

Horrible driver error. Going 2MPH gives the driver over a second to react. Clearly it was not lined up for the spot or anything else. I "pressed the brake" the first time I watched it. Uncomfortable = disengage

More like horrible design error. Blind spot (no camera(s) in front) + no ultrasonic sensors + the car cannot map reliably what it saw a second ago.

 

[MP3Mike]

More like horrible design error. Blind spot (no camera(s) in front) + no ultrasonic sensors + the car cannot map reliably what it saw a second ago.

But that car has ultrasonic sensors...

 

[willow_hiller]

But that car has ultrasonic sensors...

Yep. The guy who posted it on Twitter said the guy had HW3 with ultrasonics. We know that V12 doesn't make use of the USS, but the alarms must have been furiously dinging at that driver up until the moment of impact.

 

[JulienW]

More like horrible design error. Blind spot (no camera(s) in front) + no ultrasonic sensors + the car cannot map reliably what it saw a second ago.

I don't disagree that the cameras are not the most ideal placement. But this is a L2 system and you are the driver and the driver is responsible for the car. So my point is the driver ALLOWED the car to hit the other car.

Obviously something in the FSD software and/or hardware failed BUT the driver is in control.

Also as pointed out it does have USS and they had to be blaring. Of course it could have been a newer car sans the USS and same results. So removing them is obtuse IMO.

 

[Dewg]

Can we confirm that HW3 and/or HW4 have actual blind spots off the front corner bumpers? If so, this seems like a big issue. If not, this is a planner issue and not a perception issue.

 

[Knightshade]

Can we confirm that HW3 and/or HW4 have actual blind spots off the front corner bumpers? If so, this seems like a big issue. If not, this is a planner issue and not a perception issue.

You can confirm it just by looking at the car.

There are no cameras physically located to see below the hoodline and in front of the front fender cams.

This is also why the car can't provide a genuine 360 parking view- it physically can not see close to the car in that area.

USS was a solution for this for years but newer cars don't have them, and V12 FSD wouldn't be using them since an increasingly large % of the fleet lacks the hardware.

Telsa has suggested their solution would be the car using cameras when it DOES see objects to accurately measure distance to them and remember where they are when they move into blind spots. This video suggests they haven't actually solved one (or both) of those issues (and of course anything that moves while in the blind spot is impossible for the system to recognize)

 

[JulienW]

Can we confirm that HW3 and/or HW4 have actual blind spots off the front corner bumpers? If so, this seems like a big issue. If not, this is a planner issue and not a perception issue.

Menu>Service>Camera Preview and you can see what your cameras see and DON'T see.

 

[Dewg]

You can confirm it just by looking at the car.

There are no cameras physically located to see below the hoodline and in front of the front fender cams.

This is also why the car can't provide a genuine 360 parking view- it physically can not see close to the car in that area.

USS was a solution for this for years but newer cars don't have them, and V12 FSD wouldn't be using them since an increasingly large % of the fleet lacks the hardware.

Telsa has suggested their solution would be the car using cameras when it DOES see objects to accurately measure distance to them and remember where they are when they move into blind spots. This video suggests they haven't actually solved one (or both) of those issues (and of course anything that moves while in the blind spot is impossible for the system to recognize)

Yes yes, we all know about the below the bumper issues, but this was parking, and the car it ran into was clearly visible to both the front standard and wide-angle camera, and as it turned, the B-pillar camera. What was hiding below the bumper?

 

[Knightshade]

Yes yes, we all know about the below the bumper issues, but this was parking, and the car it ran into was clearly visible to both the front standard and wide-angle camera, and as it turned, the B-pillar camera. What was hiding below the bumper?

My previous post cites a twitter thread discussing exactly that- the front camera can see "a car" but it can't see the part of that car it actually hit... here it is again.

https://twitter.com/x/status/1761817429594755293

 

[Dewg]

My previous post cites a twitter thread discussing exactly that- the front camera can see "a car" but it can't see the part of that car it actually hit... here it is again.

https://twitter.com/x/status/1761817429594755293

Still not buying it. The car drives through parking lots and in residential neighborhoods with parked cars on the sides, and makes left and right turns with parked cars around the corners. Never hits em. If this was a blind spot, an issue with perception, we'd be seeing cars hitting curbs, parked cars around corners, parking lot cars, etc. This must be a planner issue with V12

 

[uscbucsfan]

Still not buying it. The car drives through parking lots and in residential neighborhoods with parked cars on the sides, and makes left and right turns with parked cars around the corners. Never hits em. If this was a blind spot, an issue with perception, we'd be seeing cars hitting curbs, parked cars around corners, parking lot cars, etc. This must be a planner issue with V12

To be fair, there are quite a few FSD videos where the car hits curbs.

 

[Dewg]

To be fair, there are quite a few FSD videos where the car hits curbs.

Recently? I remember issues in early v10, but in recent versions of v11?

 

[FOOLSLFDRVNG]

More like horrible design error. Blind spot (no camera(s) in front) + no ultrasonic sensors + the car cannot map reliably what it saw a second ago.

I believe it is visible through the fender camera. If you go into settings the camera seems to cover the sides 90 degrees.

 

[Knightshade]

I believe it is visible through the fender camera. If you go into settings the camera seems to cover the sides 90 degrees.

huh? The fender cams face backward-- they're what you see when you put your turn signal on and are visible in the picture showing 0 part of the vehicle that was hit because they face backward, not forward.

 

[JulienW]

huh? The fender cams face backward-- they're what you see when you put your turn signal on and are visible in the picture showing 0 part of the vehicle that was hit because they face backward, not forward.

Also we can't see what the wide angle (or Narrow) front or the B-Pillar camera saw since they are not recorded.

 

[AlanSubie4Life]

I believe it is visible through the fender camera. If you go into settings the camera seems to cover the sides 90 degrees.

You have gone into settings so you know how to find the info. You’ll find there are clearly things in the vicinity of the front bumper and to the sides of the front bumper which cannot be seen. They have to be low enough of course, otherwise they are visible. Unfortunately a bumper is too low!

Of course a human with the camera view would have just extrapolated the vehicle bumper position, and if they were familiar with the extents of the Tesla and the camera coverage, would have clearly determined it was a no-go situation. And of course humans would have memory.