FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Mardak]

HD map means higher resolution i.e. say if standard map is accurate to one foot, HD would be accurate to one inch or one mm

Whatever Tesla is using isn't even accurate to 1 foot or even 10 feet in many cases. Here's an updated BBBike.org comparison for this same area of Saluda NC from 2 years ago (from "FSD Beta: TomTom Map, Roundabouts and the Speedlimit"):

There are differences among these standard maps "just" showing general road location and connectivity. Notably look at Bing Map and Michelin Map at the overlaid pointer/cursor position that should be where 11.4.1's Tesla GPS is drawing the red arrow in the vehicle matching up with Google Map:

I would guess that's a difference of at least 50 feet between the car's actual position versus what its map/navigation data believes where the road is (the filled blue line in the right-side map area). The small driveway shown splitting off to the right in the left-side visualization is probably partially visible to the vehicle cameras, and I would expect this vehicle sending back video snapshots since 2020 would have resulted in autolabeled training data to have higher-than-average ability for the neural networks to predict this particular driveway.

 

[Ravioli]

Do you ever feel like your car is driving too close to the curb?

Normally, my car does a great job of staying within the lane. However, there are times when I get a bit nervous because in certain areas, the car seems very close to the curb or cars parked on the street. This tends to be more noticeable on narrow streets or streets with sharp curves. When I check the camera view, it always shows that the car is perfectly centered in the lane. But personally, I would feel more comfortable with a greater safety buffer, especially when there's enough space on the other side.

 

[Knightshade]

It's cool bro. When I said, "There are not 5 different answers". I wasn't saying that random forum posters wouldn't give differing answers. I was referring to the AV and HD Map industry at large. Basically I was saying there are not 5 different answers even though 5 people might give you 5 different answers.

Ok.

Except even AMONG the "AV and HD map industry at large" there are multiple, different, definitions of what an "HD map" is.

HD Map | TomTom

TomTom HD Map that enable accurate and precise localization for autonomous vehicles, key element for precise lateral and longitudinal position. Discover More.

www.tomtom.com

This cites cm scale accuracy for example.

The Crucial Differences Between AV Maps and HD Maps | Mobileye Blog

Powered by REM™, the Mobileye Roadbook™ provides AVs with the accurate, up-to-date information they need to operate effectively and safely.

www.mobileye.com

This cites 10 cm accuracy

https://www.diva-portal.org/smash/get/diva2:1578797/FULLTEXT01.pdf

This cites "between 10 and 20 cm"

I suspect if I invest more than 30 seconds into looking I'll find at least 2 more different definitions so it appears you can get at least 5 different answers even after moving the goalposts as far from my original statement as you did.

But by all means, if you believe you know of a SINGLE "correct" and complete definition of HD maps that everyone in the industry agrees on, please provide it, with a source.

 

[Knightshade]

Where/when has Tesla said that they are partnering with TomTom to build HD maps? Was that way back when AP started and Tesla went down the road of HD maps being a big driver for FSD, for which they failed, and abandoned that solution? (Bladerskb's quote didn't appear to be from Tesla, and had no link to verify it is even real. I did some quick Internet searches and couldn't find that quote anywhere.)

The quote IS from Tesla-- it's from all the FOIA stuff that plainsite published a while back....you can see the exact screen shot of text Bladerksb reposted without attribution here as posted by Green in 2021.

https://twitter.com/x/status/1399913190964842497

 

[diplomat33]

But I don't think this is the kind of centimeter level HD map that is used for the "virtual rails" robotaxi approach.

First, I really do not like that term. Waymo and other robotaxis do not use "virtual rails". cm level HD maps are not "virtual rails". That is a mischaracterization of their approach. Waymo has stated that their HD maps are used as priors only, not to actively guide or drive the car. So "virtual rails" is not a correct description. And the robotaxis rely on their cameras, lidar and radar for the real-time perception and driving. Additionally, there are many instances where Waymo or other robotaxis will leave their lane to perform a special maneuver like to go around a double parked vehicle or navigating a construction zone. The fact that they know how to leave their lane based on real-time perception that differs from their HD map, proves that they are not on "rails".

I'm going to ping @diplomat33 because he's reported about this on and off, including some posts a year or two back about Medium Density (or Medium Definition) maps.

Yes, there are 3 basic types of maps based on the amount of detail and precision they have:

Standard Definition maps are 2D, with an accuracy of +/- 10 m. And they have a total of about 50 attributes. They are used in your basic GPS navigation to help humans navigate, ie turn left in 100 m to take Rose Rd.

At the other extreme, you have your High Definition (HD) maps. They are 3D, with an accuracy of +/- 10 cm. They can have over 3000 total attributes. HD maps are used to support L4 autonomous driving by providing precise localization, perception of static world, planning and safety.

But some companies felt like HD maps are too costly and too much work for what they need, especially if they are doing some level of autonomous driving for consumer cars and don't want to HD map millions of miles of roads. But at the same time, they want maps that have many of the useful traits of HD maps to help with autonomous driving. So they wanted maps that are more detailed than standard maps but not as detailed as HD maps. So they came up with Medium Definition maps that are in between.

Medium Definition maps are 2D with an accuracy of +/- 3 m. They have about 100-500 total attributes, including rich lane and rules-of-the road metadata. These maps can be used for localization, perception, planning and safety. You can think of them as a HD lite.

Source: The Mapping Singularity Is Near

 

[jebinc]

It seems like 11.4.1 is being deployed at a slow but steady rate, and definitely beyond just employees and You-Tubers. I see from TeslaFi that there have been a couple hundred cars receiving it so far (or queued to receive it). Hope that means it will go wide soon! I know we would normally expect to see an 11.4.2 version soon that would be for a wider audience, but 11.4.1 seems to be going out fairly widely already?

I’m hoping a .2 version will come to mine.

 

[BBTX]

While we're on the general topic of the maps: in the long run, Tesla will probably always have to rely on some level of mapping (of at least "medium" definition per above). I think it's important that they pursue map-less, vision-based autonomy: it's the harder problem, and it's more general, and doesn't suffer in out of the way places with less-frequent map updates (or none at all). But there are definitely situations in which only a medium+ map will lead to good driving behavior.

What comes to mind is situations where the turn lanes (say 2 left turn lanes, two straight, then one right+straight) are only marked by paint on the surface, and you can't see the painted signage around a corner as you turn towards it heading at the "wrong" lane, and/or cars are blocking being able to see the paint at all in traffic. Humans learn about these situations that are local to them, so they don't screw them up often enough to matter, but you can't have all the Teslas in the area screwing them up every single time through the same spot. You can call them design mistakes by the highway department or whatever, but many such dumb design mistakes exist today and will likely exist far into the future, so they're just part of what you have to deal with to succeed.

I think for now the maps can make decent sense of these situations, but in the long run I think/hope Tesla will slowly remove their dependence on shoddy, slowly-updated 3rd-party maps and start harvesting their own fleet data instead, at least for higher-traffic areas. They've got a lot of cars on the road with 8 cameras on them, and they could correct even for new or temporary construction issues in their own maps very quickly from their own data.

 

[AlanSubie4Life]

I’m hoping a .2 version will come to mine.

0.22 over here.

 

[jebinc]

0.22 over here.

Typo. I meant to say, “2023.7.2j”

 

[diplomat33]

I would add that nobody builds autonomous driving that relies exclusively on maps for all the driving. Maps are a prior, meaning they are a guide to help the car. All autonomous driving must rely on real-time perception, ie cameras and other sensors, since the environment is changing. So certainly, you need to focus on vision (and lidar/radar) to make sure the car can drive without maps. But maps can be extremely useful as an aid. There can be weird roads that are difficult for vision to understand. There can be special rules of the road that will not be obvious to vision. And there can be instances where vision is not able to see far enough ahead due to trees blocking the view or a hill etc... So you build autonomous driving with vision (and lidar/radar) but you add maps as a prior. Everybody that has reliable autonomous driving has some type of map as a prior. The debate is how detailed does the map need to be.

 

[archae86]

there is no way that I would trust FSD enough to not pay close attention to what it is doing.

Good that you pay attention. Sooner or later it will do something much more alarming than anything you've seen so far, so you'll be glad you were on guard.

 

[JHCCAZ]

First, I really do not like that term.... The fact that they know how to leave their lane based on real-time perception that differs from their HD map, proves that they are not on "rails".

Fair enough. I was really only trying to reference categories of mapping and their usage that I had picked up from reading the Autonomous Car Progress thread.c I actually had a feeling you might respond something like that, as I think it's pretty obvious that the cars have to deal with actual facts on the ground.

But for clarification, not to annoy you any further - is it not the case that a roughly centimeter-level virtual track is the software's starting point, the baseline path and infrastructure features to which the real-time perception is compared? And if not, then what is the original purpose of the HD map database?

My point in asking is not to attack Waymo or anyone else. I'm genuinely interested in the pros and cons of different system design approaches. For example, it could be that the HD map and the virtual rails concept (that I know you don't like) was indeed a way to get things with prior less mature software, while knowing full well that the car could not be over-eliant on it. It's just one of several possible starting frameworks.

Thanks also for the comments on MD maps, which I remembered that you had followed. And for the newer message that I just saw before hitting the post button here.

 

[Madmolecule]

Maps should provide Pre-Vision. So the cameras, radar and ultrasonics, etc can no what to expect and verify and react. My car does not appear to know what to expect and does not have time to react. the computing might also not be fast enough. sometime I feel like I am getting the hourglass while its thinking at an intersection. Humans have more senses to help with driving than a camera on a gimbal, we also use sound smell and temperature and weather conditions.

still no discussion on Liability on FSD! Once FSD is no longer beta, like that will ever come, who is liable for bad Maps or stupid AI decisions when your Tesla is Fully Self Driving

 

[JulienW]

...still no discussion on Liability on FSD! Once FSD is no longer beta, like that will ever come, who is liable for bad Maps or stupid AI decisions when your Tesla is Fully Self Driving

Liability is simple. As long as it is L2 driver assist system then you are 100% responsible no matter if labeled Beta or not. If it is ever (looking kinda doubtful) certified as a L3 or higher system then the driver will be the car (not you) and Tesla will be responsible when the system is on.

 

[uscbucsfan]

still no discussion on Liability on FSD! Once FSD is no longer beta, like that will ever come, who is liable for bad Maps or stupid AI decisions when your Tesla is Fully Self Driving

Isn't that like asking who will be plumbers on Mars?

 

[JHCCAZ]

...I think for now the maps can make decent sense of these situations, but in the long run I think/hope Tesla will slowly remove their dependence on shoddy, slowly-updated 3rd-party maps and start harvesting their own fleet data instead, at least for higher-traffic areas. They've got a lot of cars on the road with 8 cameras on them, and they could correct even for new or temporary construction issues in their own maps very quickly from their own data.

I generally agree with your whole post, but specifically to the last paragraph here, I think the good news is that this does seem to be the plan as of now. I'm quite sure we haven't seen a very mature version of it yet, and I hope it gets decent ongoing attention.

Even as Tesla's perception capabilities improve in the direction of what a human can understand about the driving environment, we can appreciate that it's usually a better drive, on a given route, if it's not the first drive. It's been a common observation that Tesla FSD sees every scene for the first time, doesn't learn from having been there yesterday and doesn't learn from corrections you provide to it. The downloaded map and environment details is a potentially major step to improve those things.

 

[Ramphex]

Damn I’m behind on 4 pages

Can I get spark notes?

 

[jebinc]

Damn I’m behind on 4 pages

Can I get spark notes?

Four pages of map debates. You missed nothing.

 

[ItsNotAboutTheMoney]

Isn't that like asking who will be plumbers on Mars?

Isn't it obvious? Mario on Mars. Luigi will be the plumber on the moon

 

[diplomat33]

But for clarification, not to annoy you any further - is it not the case that a roughly centimeter-level virtual track is the software's starting point, the baseline path and infrastructure features to which the real-time perception is compared? And if not, then what is the original purpose of the HD map database?

You are not annoying. You are asking good questions.

I think you might be thinking of localization. Every autonomous vehicle has to start with localization. The idea of localization is that the car has to know where it is in the world. So part of the purpose of any map, whether it is HD or not, is to give the car a "picture" of the static world around it so it can localize itself in that world. AVs will use their real-time sensors like cameras, radar, lidar as well as other devices like a gyroscope and accelerometers to localize itself on a map. And you need to maintain precise localization over time otherwise the localization could degrade. So you need to accurately measure the vehicle's speed and direction.

But localization is different from "rails". Localization just means knowing where you are in reference to your environment. Localization does not mean that you are stuck on a path that you have to take. That is why I don't like the term "virtual rails" since it falsely implies that the AVs are stuck on a path that they can never deviate from. Once the AV is localized, AVs will rely on real-time perception like cameras to do the actual driving. The planning stack will decide the appropriate path based on real-time perception and behavior prediction. And the car will often update its planned path in real-time based on its changing environment.

I would add that AV companies might have relied on HD maps more in the past when perception was less reliable. But I don't think any smart AV company thought that they could rely on HD maps alone to drive. Every AV company worked to make their real-time perception like camera vision better.

In closing, I would say the original purpose of HD maps was to help with localization as well as perception of the static world. Now, HD maps are still used for localization but they used more for information that perception cannot provide like implied rules of the road.