FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[kabin]

The turns my car struggle with are not blind, you can clearly see traffic on either side. The car still makes unsafe turn decisions. Why does it do this if the car is accurately modeling a 3d vector space around it? It baffles me every time.

The poor lane decisions, bad speed limits, and other various quirks are all understandable disengagements and failures. But safety related such as UPL's and right hand turns really lower the credibility of FSD ever becoming a reality. The car does now what it should have done 5 years ago, but still has fatal flaws where everyone initially expected.

Yep, there seems to be plenty of vision capability. I suspect vision quality is a factor especially when there's movement between ego and targets which is almost all the time. So far FSD still struggles to estimate target vehicles kinematics and paths. But also something dramatic falls short from time to time. Pulling out in front of semi trucks suggests there's occasional data latency, processing lag, or a system losing sanity.

I'm not sure TSLA's L2 will ever be accepted if it regularly causes near misses, attempts to run into stationary obstacles, phantom brakes, inadequately models VRUs/moving targets, fails to handle roundabouts, etc.

 

[aronth5]

Yep, there seems to be plenty of vision capability. I suspect vision quality is a factor especially when there's movement between ego and targets which is almost all the time. So far FSD still struggles to estimate target vehicles kinematics and paths. But also something dramatic falls short from time to time. Pulling out in front of semi trucks suggests there's occasional data latency, processing lag, or a system losing sanity.

I'm not sure TSLA's L2 will ever be accepted if it regularly causes near misses, attempts to run into stationary obstacles, phantom brakes, inadequately models VRUs/moving targets, fails to handle roundabouts, etc.

There is plenty of vision capability for many of the problems people report. I don't believe this is a true statement for the B-pillar cameras. I have had FSD more than once start to pull out when the vehicle is 110 feet away. By just pulling forward 2-3 feet where my head is when I lean forward that distance increases significantly. With crossing roads with no shoulder and no room to creep out far enough you have a significant problem. The problem is when you have obstructed views to the side. If I let FSD creep so far it forces oncoming vehicles to cross the yellow center it will gladly mess with crossing traffic which of course is stupid.

 

[mgs333]

I suppose it’s relative. I think it’s just a fun and cool thing to have in my opinion and well worth the money as I enjoy feeling like I’m in the future even though it’s far from perfect. I spent twice the cost of FSD on a Mac Pro a few years back and ended up not feeling like I got my money’s worth from it even though others with a similar configuration of Mac were much happier than I was using it.

As both an Apple and Tesla shareholder, I thank you.

 

[jebinc]

The turns my car struggle with are not blind, you can clearly see traffic on either side. The car still makes unsafe turn decisions. Why does it do this if the car is accurately modeling a 3d vector space around it? It baffles me every time.

The poor lane decisions, bad speed limits, and other various quirks are all understandable disengagements and failures. But safety related such as UPL's and right hand turns really lower the credibility of FSD ever becoming a reality. The car does now what it should have done 5 years ago, but still has fatal flaws where everyone initially expected.

And people pay $15,000 for this. It's one of the biggest scams you could purchase.

Agreed.

 

[kabin]

There is plenty of vision capability for many of the problems people report. I don't believe this is a true statement for the B-pillar cameras. I have had FSD more than once start to pull out when the vehicle is 110 feet away. By just pulling forward 2-3 feet where my head is when I lean forward that distance increases significantly. With crossing roads with no shoulder and no room to creep out far enough you have a significant problem. The problem is when you have obstructed views to the side. If I let FSD creep so far it forces oncoming vehicles to cross the yellow center it will gladly mess with crossing traffic which of course is stupid.

Yep, B pillar camera location isn't ideal. And it doesn't help to stop short at intersections and gamble one's safety with goofy lunges into inappropriate traffic openings. If FSDj can't handle a scenario it needs to hand off before causing a potential accident. And that's the rub! Unfortunately Tesla's way is to fake it until they make it.

 

[powertoold]

The side repeater cameras also see cross traffic, especially on unprotected rights.

 

[fasteddie7]

Yep, B pillar camera location isn't ideal. And it doesn't help to stop short at intersections and gamble one's safety with goofy lunges into inappropriate traffic openings.

That camera placement is now roughly 7 years old. One would think at some point the realization that there would be a better placement for optics was made, what was the motivation for staying the course instead of cutting losses at one of the revision points over the years? The same talk track always seems to be used when a revision comes out, the hardware can do it but the new hardware can do it X% better than a person, so why wait just to have to do it later when it would have been better when the software was $2000 and early as opposed to doubling down several times on $15k?

 

[D Good]

When we get updated to the latest software, it will be interesting use the new camera previews to see what the cameras can actually see.

 

[eli_]

I'm not sure which build this changed, maybe it was the first "single stack" release. But heavy rain no longer disables the FSD visualization, it would always revert back to old school AP viz whenever the wipers sped up to continuous speeds, and then refuse to engage FSD.

 

[fasteddie7]

I'm not sure which build this changed, maybe it was the first "single stack" release. But heavy rain no longer disables the FSD visualization, it would always revert back to old school AP viz whenever the wipers sped up to continuous speeds, and then refuse to engage FSD.

At least the 11.4.1 build as I had a trip in heavy rain after I got it and it was there.

 

[kabin]

That camera placement is now roughly 7 years old. One would think at some point the realization that there would be a better placement for optics was made, what was the motivation for staying the course instead of cutting losses at one of the revision points over the years? The same talk track always seems to be used when a revision comes out, the hardware can do it but the new hardware can do it X% better than a person, so why wait just to have to do it later when it would have been better when the software was $2000 and early as opposed to doubling down several times on $15k?

Aside from optics the bean counters win with the B pillar placement by eliminating fender housings. And then less concern for debris splatter, ever so slightly better aerodynamics, and a B pillar camera height might see better than a fender mount when the view is blocked by an adjacent vehicle. Maybe the best placement is the upper A frame corners?

 

[aronth5]

The side repeater cameras also see cross traffic, especially on unprotected rights.

Try backing your car up and you will see how limited the side repeater cameras are for viewing cross traffic. Then turn your car like you would to get ready for a turn and the view is still very limited. It looks like HW4 has a larger view but until Tesla puts a camera further forward I am doubtful FSD is as safe as a human driver who leans way forward.

Now there is a chance what we see now when backing up is less than what FSD processes. If true this concern could be mute.

 

[swedge]

I've been keeping a list (link) of a dozen or so spots where I must routinely disengage FSDβ, and I check each new release to see if any of these have been fixed. Probably none of them are particularly dangerous, but each would certainly confuse or piss off other drivers and/or scare my passengers.

On the progress side, 11.4.4 is much better at keeping FSD available than was 11.3.6 - much less blinking grey steering wheel icon and therefor far fewer failures to engage. Also there seems to be fewer random blinker activations. While somewhat subjective, 11.4.4 also seems to have balanced the quickness of responding to requesting a lane change with the blinker, and the delay between the warning and blinker before making an automatic lane change. Now the requested lane change starts promptly, and the automatic one gives enough time to cancel without zig-zagging.

As for regressions, 11.3.6 had fixed one long standing issue fixed, but it has returned from the dead bug graveyard: abrupt firm braking far before one particular stop sign. Sadly 11.4.2 brought it back, and it persists in 11.4.4. This is less than a quarter mile from home, and results in passengers and unsecured contents getting tossed forward. This makes FSD demo drives awkward, instant proof that it is not yet ready for prime time.

Based on only one short drive, 11.4.2 appears to have retained the other problems as well. There are a couple I have not checked yet.

One which the 11.4.4 release notes suggested may have improved relates to merging out of an ending lane. In this case, a freeway onramp leads to a new lane which then shortly becomes into an exit only lane and off-ramp. The car should turn on the blinker and change into the thru lane right away, or else it is forced to Take the exit. Today, 11.4.4 did manage, but only just barely, as it delayed the blinker and lane change till the last second, and had to cross the triangular exit lane gore to make its abrupt merge.

There is a complication at this spot in that the car comes down the entire straight on-ramp thinking the speed limit is 30mph, clearly a map error. Next time I'll dial up the speed to see if it can handle the merge at a normal speed. Does anyone know who to contact to get the ramp speed fixed on the map?

I guess the wait for 11.420 begins today...

 

[jabloomf1230]

The side repeater cameras also see cross traffic, especially on unprotected rights.

They can see some cross traffic. If the cross traffic speeds are modest, a side obstruction is less important. However if the cross traffic speeds are 100 kph or more, by the time that the repeater cam sees the cross traffic, it's too late.

 

[scottf200]

Aside from optics the bean counters win with the B pillar placement by eliminating fender housings. And then less concern for debris splatter, ever so slightly better aerodynamics, and a B pillar camera height might see better than a fender mount when the view is blocked by an adjacent vehicle. Maybe the best placement is the upper A frame corners?

See "Side Front Camera" - farther ahead angle than a human even leaning forward can get.

Camera POV - Mobileyes SuperVision requirements on their partner Zeekr

Via: Mobileye CES talk (see bottom of this post) A pretty interesting graphic of the camera placement and degrees of visibility (FOV) of Mobileyes SuperVision requirements on their partner Zeekr. - Of note: front camera (2 vs 3), Side Front Camera on mirror vs B-pillar. - "7 long range cameras...

teslamotorsclub.com

 

[aronth5]

They can see some cross traffic. If the cross traffic speeds are modest, a side obstruction is less important. However if the cross traffic speeds are 100 kph or more, by the time that the repeater cam sees the cross traffic, it's too late.

Back up and check out the side camera view and you'll realize it's very limited. I know the Tesla engineers are much smarter than I am but I keep wondering what do they know that we don't when it comes to camera placement? Or is this an example of one of Elons guiding principles that admitting you're wrong can sometimes be extremely difficult?

 

[jabloomf1230]

Back up and check out the side camera view and you'll realize it's very limited. I know the Tesla engineers are much smarter than I am but I keep wondering what do they know that we don't when it comes to camera placement? Or is this an example of one of Elons guiding principles that admitting you're wrong can sometimes be extremely difficult?

It's a cost issue. A side-facing set of cameras mounted on the front bumper would get dirty really quick. That means that they would need a cleaning system. All that complexity violates Elon's Razor.

 

[fasteddie7]

Looks like actually smart wipers and actually smart summon are coming in one month and three months respectively, they either really believe these cameras can do it all, or they are doing the best they can with an impossible task.

 

[powertoold]

they either really believe these cameras can do it all, or they are doing the best they can with an impossible task.

Why would Tesla set up billions of dollars in infrastructure and commit to millions of cars with the same sensor locations if they didn't think about this for at least a few weeks (ie much longer than we have and with simulations and all)

 

[D Good]

11.420. Ahh. How much longer will we be waiting for the “big one”? I thought that v10 was that? Or maybe v11? Wait, V11 is the worst update we have ever seen so far…
The point is is that there is no “fix-all” update. That is not how neural networks work. We are years and years and years away from true level 3 FSD at this rate of improvement. There is no sign of “exponential improvement” outside of Elon’s twitter world.
Ever notice how as soon as one (as always, bad) release comes out, Elon then mentions the next one, which will be or “The big one”? It’s all hype to keep the rolling in and the lawsuits away.