FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[jeremymc7]

As am I. But it’s going to be quite some time before they add the improved / additional cameras, HD radar, update body panels (as necessarily) for radar and cameras, get the software optimized for all above.

I’m thinking at least another year or two. The hardware might all be in there before but as usual it’s going to take a while to get all the coding complete.

 

[johnm]

HW3's last great SW hope (V11)

There are plenty of natural numbers greater than 11.

 

[Knightshade]

As am I. But it’s going to be quite some time before they add the improved / additional cameras, HD radar, update body panels (as necessarily) for radar and cameras, get the software optimized for all above.

I’m thinking at least another year or two. The hardware might all be in there before but as usual it’s going to take a while to get all the coding complete.

huh?

The improved cameras and HD radar are already there in HW4 cars. There's no body panel updates anybody's "waiting" for. And they're not coming to HW3 cars.

The question of added cameras is outstanding, but it's possible they'll only be appearing in other models (cybertruck for example that might have greater need of low cameras for off-roading).


I do agree it'll likely take some time for significant SW differences to emerge though- same was true when they went HW2.x to HW3 and that was a much smaller change.

 

[jeremymc7]

huh?

The improved cameras and HD radar are already there in HW4 cars. There's no body panel updates anybody's "waiting" for. And they're not coming to HW3 cars.

The question of added cameras is outstanding, but it's possible they'll only be appearing in other models (cybertruck for example that might have greater need of low cameras for off-roading).


I do agree it'll likely take some time for significant SW differences to emerge though- same was true when they went HW2.x to HW3 and that was a much smaller change.

Improved cameras yea. But not the additional cameras that are being discussed in the forum.

If HD radar is in there then great. But I’ve brought this up many times and everyone so far has said the HD radar sent in yet.

The additional cameras not rolled out yet will require some physical changes somewhere. If the front corners cameras end up in the headlights then new headlights might be plug and play and not require new panels.

It is possible there may be no one xtra cameras for the cars and only the truck. However there are photos flying around of 3/Y headlights with cameras in new headlights.

I made no mention of HW3. Not sure where that comment came from.

I’m simply pointing out for those that are waiting on a car that has everything they need now isn’t the right time to buy in. At a minimum it’s not official what the final HW4 equipment will be is still up in the air. Some say only the computer is in the current cars, other says the cameras as well, while yourself says also the radar.

From what I can see Tesla isn’t mentioning anything related to HW4 on the website yet. Even the photo they have of the FSD on the Model X looks like the HW3 computer.

My latest car still has the old radar ‘(no longer used?) and USS (still used on current beta?) but it seems more feature complete than current cars with HW4 computer and no USS. Though some argument could be made that new cameras if on current cars might provide better imaging; provided the beta software has been coded to utilize the improvements.

Once Tesla officially starts detailing everything HW4 and updates the website and all new cars are at least hardware complete I’ll likely be trading in. Till then I’ll be holding off. Depending on how functional the software is at that point I might still hold out longer if my current car is more feature complete.

Either way now is not a time to make that decision; at least for me.

If someone absolutely needs a car now; buy it now. You can always wait for the next thing.

 

[Scott Chapman]

Has anybody noticed what FSD Beta 11.3.1 does in really bad weather? This video seems to keep "Full Self-Driving may be degraded / Poor weather detected" message up basically the whole drive:

I haven't seen "improve weather" in any of the release notes, but that feature would certainly benefit both Tesla & SpaceX so who knows?!

 

[sleepydoc]

Improved cameras yea. But not the additional cameras that are being discussed in the forum.

How do you know those are even being considered? Don’t confuse opinions on a forum with anything official from Tesla.

 

[Knightshade]

Improved cameras yea. But not the additional cameras that are being discussed in the forum.

Which might never be coming to existing vehicles at all.

If HD radar is in there then great. But I’ve brought this up many times and everyone so far has said the HD radar sent in yet.

Who is "everyone"? Do any of them actually have the HW4 cars that have been delivered?

The additional cameras not rolled out yet will require some physical changes somewhere.

Again, not if they're only coming to new models rather than existing ones.

It is possible there may be no one xtra cameras for the cars and only the truck. However there are photos flying around of 3/Y headlights with cameras in new headlights.

There's been random speculation that something in a random test pic might be a camera. But it'd make little sense to have swapped in the new cameras with plans to retrofit them with more later... not to mention if they WERE in the headlights there'd be no need for the "revised" panels or bumpers you mention, you'd just swap the headlight units.

From what I can see Tesla isn’t mentioning anything related to HW4 on the website yet.

Typically they do not do so until all production has switched over.... currently only S/X have gotten HW4 to my knowledge.

I haven't seen "improve weather" in any of the release notes, but that feature would certainly benefit both Tesla & SpaceX so who knows?!

They HAD a weather dominator, but the Joes broke it into three pieces....

 

[jeremymc7]

Which might never be coming to existing vehicles at all.

Like with many HW4 nothing is officially known 100%

Who is "everyone"? Do any of them actually have the HW4 cars that have been delivered?

Several people with HW4 but no one I’ve heard has mentioned HD Radar or would know where / how to check.

Again, not if they're only coming to new models rather than existing ones.

It seems more likely than not there would be two additional cameras for front bumper corners. This has been highly demanded and several people have done test mules that show it could greatly help FAD not only into pulling out into intersections, but especially when making turns - notably on 90% variants that beta would have significantly better visibility in these areas where beta does a lot of slowwwww creeping.


There's been random speculation that something in a random test pic might be a camera. But it'd make little sense to have swapped in the new cameras with plans to retrofit them with more later... not to mention if they WERE in the headlights there'd be no need for the "revised" panels or bumpers you mention, you'd just swap the headlight units.

Depending it’s the space and mount are exactly the same as current units or there would need to be metal work changes. As before it seems logical there will be front side cameras; especially with the change of 3 windshield units or varying focal reused to one; logically picked up by two for a wider area.


Typically they do not do so until all production has switched over.... currently only S/X have gotten HW4 to my knowledge.

Yes. So as before. Anyone who doesn’t need a car now is best suited to wait until it’s ordinal, documented on the website, and 100% shipping on all of your model - at the least. Once we get to the first point it’s then a matter of if all functions are working in software and if not is it a feature you use on your current HW3 which if the case might be justification of yet waiting a little till SW is up to your level. Lastly they’re the old tale of don’t buy the first year; wait for the second year until all the bugs and fixed. Plus there’s always stuff they couldn’t get out in time for the first year.

Yes I understand Tesla and model year don’t mean what they do in the rest of the industry. It’s just a reference.

 

[powertoold]

This is the kind of stop sign confusion that happens all the time if fsd stops completely to 0mph in urban areas. Watch all the stops:

 

[jeremymc7]

Human drivers will never drive like their Robot counterparts.

Robot drivers can’t predictably and shouldn’t anyway be like human drivers.

Finding a middle area that fits within the letter of the law, is acceptable to human drivers, and can be coded reliably & predictably is going to be the never ending nightmare.

 

[EVNow]

Every one of the 30+ roundabouts near my house () are mapped in Google Maps, TomTom and Open Street Maps. I've never noticed FSDb able to traverse any of them properly if there was even the slightest amount of traffic. Roundabouts are probably one of the reasons why there is no FSDb availability in Europe.

Can you explain what you mean by "I've never noticed FSDb able to traverse any of them properly if there was even the slightest amount of traffic."

I've a new thread to document and track FSDb behavior at roundabouts.

Roundabouts !

It is high time roundabouts got their own thread. Here are all the issues I see / used to see - along with what seems to be currently fixed. - Doesn't wait for the vehicle coming from the left.

teslamotorsclub.com

 

[AlanSubie4Life]

This is the kind of stop sign confusion that happens all the time if fsd stops completely to 0mph in urban areas. Watch all the stops:

Of course, it has nothing to do with the vehicle stopping to 0mph.

 

[zoomer0056]

How do you know those are even being considered? Don’t confuse opinions on a forum with anything official from Tesla.

The only pre-facts we have are those discussed at TMC. Talk enough and they become mainstream. Then they get referenced and become facts.

 

[kriskros91]

This is the kind of stop sign confusion that happens all the time if fsd stops completely to 0mph in urban areas. Watch all the stops:

I think I agree with @AlanSubie4Life that this doesn't necessarily have to do with stopping at 0 MPH. It is in fact the car not realizing that it has already "stopped" for this stop sign while the pedestrians are crossing.

In this clip the ego car appears to have stopped a little short of the true "stop" position at the intersection and is waiting for pedestrians to cross while another car is very obviously second to the multi-way stop sign. It is an easy assumption at this intersection that if the pedestrians have fully left the path of the ego car while the other car is still in the process of stopping for their stop sign, that then the ego car should proceed through the full intersection and the other car should wait for this to be completed before proceeding. Human drivers in other cars expect this behavior. This situation was not good because after the pedestrians cleared the intersection, the ego car wanted to AGAIN stop at the "correct" stopping position and then proceed. It had some acceleration to make it look like the car would proceed through the intersection without a clear response to the ego driver that it was going to stop again. The bug in his visualization may have contributed to this because maybe there would have been a comment like "stopping for stop sign" if it was working correctly. So not only is it a surprise to the other human drivers at the intersection but also a surprise to the ego driver that another stop was going to happen. Then the car sees that the other car has stopped in a confused way and tries to proceed while the other human thinks that the ego driver has given right of way to him. Then it's a "no you go, no you go" situation that requires human to human interaction. Awkward and not at all normal observation of typical traffic rules. This is why robotaxi interactions with humans will be such a HUGE challenge and will be super annoying in the early stages for all parties.

 

[JulienW]

Some may find interesting. An interview of Chuck Cook with some diving clips analyzed.

 

[powertoold]

Of course, it has nothing to do with the vehicle stopping to 0mph.

I think it does. Fsd is required to stop to 0mph exactly at the line, every time. That's confusing to all those other drivers in that video that were creeping at 2-3mph into the intersection without completely stopping. You can actually see multiple instances of confusion in the video, not only the pedestrian case.

I'm not saying that fsdb is wrong in all these cases. I'm just pointing out the cause of confusion.

 

[powertoold]

ego car wanted to AGAIN stop at the "correct" stopping position and then proceed.

It's required to do that. Complete 0mph exactly at the stop line, every time.

 

[EVNow]

Interesting thing is - so far, no reports of any major regressions on highways.

This leads me to believe some of the delays in the last month or so are because of recall rather than highway regressions.

 

[Supcom]

Interesting thing is - so far, no reports of any major regressions on highways.

This leads me to believe some of the delays in the last month or so are because of recall rather than highway regressions.

As are some of the delays in the next month or so...

 

[gsmith123]

Fsd is required to stop to 0mph exactly at the line, every time.

Is the "exactly at the line" part confirmed?

It's clear NHTSA requires a full, perceptible stop, but it's not clear if they have strict requirements on where that is. In the real world stop signs can occasionally be placed pretty far back. Does a stop sign legally mean "stop exactly parallel to this sign" or does it mean "stop at the edge of the intersection (which might be a couple feet further forward)"?

If you spend a significant amount of time driving on FSD Beta it becomes pretty obvious that the difference between those can be substantial!

In my opinion, FSD today uses pretty rudimentary stop sign handling (lets call it V0.1). At a stop sign there's three points (maybe four) that are important to the car:

1. Where the stop sign is
2. Where the stop line, or logical "edge of the intersection" is located
3. How far the car needs to creep for visibility
4. How far the car can creep for visibility without potentially blocking cross traffic (the creep limit the car visualizes today)

Stop sign logic V0.1 does this:
- Stop at #1
- Creeps forward slowly until it reaches #3 (never exceeding #4)
- Proceeds when clear

Humans generally make their initial stop somewhere around #3, but short of #4. Quite a bit further forward than FSD.

I'm guessing Tesla is pretty aware of the limitations of the current design. They're probably trying to get it compliant with NHTSA requirements first (by just forcing the car to stop for 1-200ms), and then working on a better V1 version.

A better implementation would:

- Approach the intersection a little bit faster (i.e. brake a little more assertively between 3 mph down to zero)
- Pick an initial stopping point that ideally is near #3 (maybe at #2 if NHTSA requires the initial stop to be completely clear of the intersection including not protruding into any crosswalks even when obviously clear of humans). The idea is the initial stop should be as far forward as possible to reduce creeping.
- Then creep (and for weird intersections where the creep is really long, creep a little faster at the beginning). The further from #4 the faster the creep should be.
- Accelerate more assertively once reaching a "go" decision.


I'm very curious how NHTSA is thinking about the second step (where the stop is).


I'm hoping this logic, combined with high deceleration and acceleration (to reduce the overall time it takes to go through the intersection) can satisfy the perceptual stop requirements and be reasonable clear to other cars and comfortable for the driver. Better cross traffic visibility cameras helps a lot too by reducing the required creep.