FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Tronguy]

Anyone else experiencing this side-to-side wobble? See video. It's pretty bad! It started with 11.3.6 and no change with recent update to 11.4.2

Only when the car is driving. I tried both FSDb or TACC same thing.
If I disengage I don't feel anything our of the ordinary. Even without touching the wheel the car goes straight as an arrow.

Below 50 mph hardly any wobble at all, but above 65 mph it gets really noticeable.
The faster I go and the more it wobbles. It's making my GF sick so this is serious!!

Only on AP/FSDb. Driving manual ... smooth as butter.

Had an alignment, made no difference.
Tires are in good shape; rear are new and front have lots of thread left... suspension inspected and all is tight ... no idea what to do

I've been zipping around Central Jersey on and off interstates, up to 70 mph or thereabouts on the 65 mph roads, and no wobbles. Switches lanes back and forth as advertised.

Somebody said something about camera calibration.. dunno.

 

[Twiglett]

not sure if this has been posted in here before. Joe does a series of comparisons between an older Model S and M3P both on 11.4.2
Both the cars seem to behave pretty much the same, but it really shows the crazy lane selection regressions.

https://twitter.com/x/status/1665154467883302915

 

[powertoold]

But if you're saying the car is not downloading any helpful information (beyond its builtl-in general map) during the routing calculation - well then I'd have to disagree with that. I think it's become fairly clear that this is happening and it's mostly helpful - but occasionally detrimental, if the added details aren't quite right.

The main argument that would make me retract that assertion, would be that FSD in the car is now building its own individualized route experience and acting upon that across various drives. That may happen someday, but pretty much everyone says it's impossible with the present in-car NN architecture.

It's definitely downloading some lane guidance hints; Tesla says as much in the release notes.

But lane / map guidance is not the same kind of data we're talking about wrt the road geometry predictions.

I'm sure you've witnesses the +-1 foot accurate parking lot predictions on the visualization. It basically looks like the whole parking lot was downloaded as an elevation map with curbs, islands, through roads, and everything. This would be considered an HD map in most circles, as it includes accurate curbs and widths and distances and all.

 

[EVNow]

But that’s the point: the light is flickering at a rate faster than a human can perceive it: the car is sampling at a rate that violates the Nyquist limit for the system; and the offset in frequency between the two rates makes the aliasing happen, causing the digital filter in the Tesla to see a slowly blinking red light where there isn’t one.

We’re probably saying the same thing in different ways. And I’m not surprised that there would be a paper on the subject.

Thanks for the refresher - I knew the 2x sample rate like all EEs do and the low pass filter all DACs use etc- but had forgotten about aliasing.

BUT - that doesn’t explain why the visualization shows a solid red when the car runs the red light. When the car shows solid red in visualization- that’s what the vision NN is detecting.

ps : We used to joke EE college was four years of Fourier transformation

 

[rlsd]

Map NA-2022.44-14515 is worse than the previous version in some areas. There is a 50mph street near my area. The previous version shows 50 as speed limit correctly. NA-2022.4-14515 shows 35 as speed limit for a very short distance (about 50 yards). There is no reason for that speed limit drop.

 

[AlanSubie4Life]

Possibly a flicker rate issue with the specific LED lights used in the traffic light. If the flicker rate is close to the camera frame rate, it could be seen as a flashing red instead of solid red. FSD might then stop, then attempt to go when the intersection looks clear.

Might even seem random if the sync between the camera and the light need to be just right.

Never had FSD go on a flashing red, for example Miramar Rd at night due to Pure Water toilet to tap.

It just sits there if it is a regular traffic light in normal times. (Presumably very different than behavior at a flashing red light under normal conditions.) Of course I have not waited a long time. Maybe it eventually goes.

 

[tivoboy]

My GOD, why on earth at this point does the FSDb still sometimes want to go THROUGH A RED LIGHT! I can see in the UI on screen that it SEES THE RED LIGHT, all three red lights in this case, and yet it proceeds forward, I disengage and stop the car and then re-engage FSDb and boom, wants to go through the RED LIGHT again. This is IMHO totally unacceptable.

Part of me really thinks maybe there is a problem with my forward looking cameras, (may 2018 M3), but frankly if there IS something wrong with them, shouldn’t the system recognize that during calibration and give a warning?

 

[enemji]

Never had FSD go on a flashing red, for example Miramar Rd at night due to Pure Water toilet to tap.

It just sits there if it is a regular traffic light in normal times. (Presumably very different than behavior at a flashing red light under normal conditions.) Of course I have not waited a long time. Maybe it eventually goes.

Mine treated the flashing red as a STOP sign. Perfect.

 

[rlsd]

Mine treated the flashing red as a STOP sign. Perfect.

I don't have any issue with red/yellow lights so far. FSD works perfectly. The only concern I have is FSD moves the car to the intersection during green light and will block the intersection when the leading cars can't move because of slow traffic.

 

[valem]

Not good. You didn't mention performing a camera calibration, not sure that would help but it's worth a try. I haven't needed to do it at all, but it's an available procedure in the service menu.

Camera calibration was the first thing I did before the alignment (since that was free lol)

I am thinking it has to be something in the steering servo overreacting ... if it was mechanical/structural I would have to feel something while driving manually

 

[AlanSubie4Life]

Mine treated the flashing red as a STOP sign. Perfect.

For a disabled light?

A normal flashing red presumably works fine (I’ve never encountered one of course).

Over many versions I have encountered many flashing-red disabled lights and not once has FSD proceeded. As mentioned, caveat is that I have a limited time to wait since rarely have I encountered one with zero traffic. Maybe after 10-15 seconds it would go (I have no way to know).

 

[willow_hiller]

For a disabled light?

A normal flashing red presumably works fine (I’ve never encountered one of course).

Over many versions I have encountered many flashing-red disabled lights and not once has FSD proceeded. As mentioned, caveat is that I have a limited time to wait since rarely have I encountered one with zero traffic.

FSD Beta stops for all situations for me. I've never seen it proceed on a flashing red, and it's also tried to stop multiple times for lights in situations where human drivers do not treat it as a 4-way stop (e.g. traffic lights being added, but not yet enabled to an intersection).

The only situations I've seen where FSD Beta may have been confused into running a red was when there are adjacent lights at acute angles that could be confused to be controlling the lane.

 

[AlanSubie4Life]

I've never seen it proceed on a flashing red,

Yes, same experience (though never encountered a true flashing red as I said). I have only seen it proceed on solid red (as documented on video posted - been a few months), just a regular red, normal, unconfusing intersection, but second in a series (location provided previously).

But flashing red (disabled traffic light) - nothing. Always stopped. And remained stopped.

 

[JulienW]

Got a map update this afternoon, pretty sure all my FSD drives will be flawless from now on.

I have a traffic island that since 10.2 has misidentified as a traffic circle about 25% of the time. Since the new map data update it now misidentified it 100% of the time. Keep'em coming.

 

[LowlyOilBurner]

I have a traffic island that since 10.2 has misidentified as a traffic circle about 25% of the time. Since the new map data update it now misidentified it 100% of the time. Keep'em coming.

Yeah, this map update is for the worse. As mentioned earlier, speed limits are now broken on some roads, I’ve had “changing lane to follow route”, where there is absolutely no need to leave the lane I’m in. This has made FSDb(J) worse for me. I have some more drives scheduled this afternoon, which prior to this map update were fine, so I’ll have to see.

 

[Twiglett]

Yeah, this map update is for the worse. As mentioned earlier, speed limits are now broken on some roads, I’ve had “changing lane to follow route”, where there is absolutely no need to leave the lane I’m in. This has made FSDb(J) worse for me. I have some more drives scheduled this afternoon, which prior to this map update were fine, so I’ll have to see.

yep - 11.4.2 is significantly worse for this, I though 11.3.6 was quite a lot better but 11.4.2 is at least as bad as 10.11 or so. Although now I type that I'm thinking back and maybe that's doing a disservice to 10.11

 

[sleepydoc]

But that’s the point: the light is flickering at a rate faster than a human can perceive it: the car is sampling at a rate that violates the Nyquist limit for the system; and the offset in frequency between the two rates makes the aliasing happen, causing the digital filter in the Tesla to see a slowly blinking red light where there isn’t one.

We’re probably saying the same thing in different ways. And I’m not surprised that there would be a paper on the subject.

The other issue is the ‘shutter speed’ for the sample which would also act as a pseudo-low pass filter

 

[LowlyOilBurner]

yep - 11.4.2 is significantly worse for this, I though 11.3.6 was quite a lot better but 11.4.2 is at least as bad as 10.11 or so. Although now I type that I'm thinking back and maybe that's doing a disservice to 10.11

Honestly, the best update I’ve had was 11.3.2.

 

[sleepydoc]

Mine treated the flashing red as a STOP sign. Perfect.

I’ve had very inconsistent behavior with blinking red lights. Occasionally it will handle them appropriately. Most times it will simply stop and not proceed. I haven’t been able to figure out why.

FSDb does reliably seem to handle flashing yellow lights, so they must have some code to deal with them, unless all FSD doesn’t see the blinking at all and sees a solid yellow.

 

[Ramphex]

Sounds like you might have a Model S/X lol

@Dave EV do you not have a model s/x? You disliked without replying…