FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[aronth5]

Not really sure where to post this but updated maps would certainly be helpful.
@greentheonly has a well deserved reputation for posting upcoming Tesla software/hardware changes.

https://twitter.com/x/status/1482631022302875649

 

[Mardak]

At CVPR'21 in June last year, Karpathy had a slide showing "8 cameras measuring [1280 x 960 x 3] uint8 @ 36Hz" presumably referring to the 3 Y,U,V-converted channels each supporting up to 256 values for ~17M colors (where some say humans generally perceive ~10M colors). Whereas at AI Day just 2 months later, the slides show "raw 1280x960 12-Bit (HDR) @ 36Hz" where assuming these are 3 channels with 4096 values each, that's now a range of ~69B colors (3 channels * 4 extra bits = 4096x).

Hmm… Looks like 10-bit made it in to 10.x probably ported/uplifted from Beta 11?

 

[EVNow]

Hmm… Looks like 10-bit made it in to 10.x probably ported/uplifted from Beta 11?

View attachment 756266

Not just 10 bit - probably the bigger news is doing some of the 3D environment building in NN instead of C++ code. That is the first point.

 

[EVNow]

Not really sure where to post this but updated maps would certainly be helpful.
@greentheonly has a well deserved reputation for posting upcoming Tesla software/hardware changes.

https://twitter.com/x/status/1482631022302875649

This would be a very interesting update - which can tell us whether the TomTom maps updates we have done are flowing into the car.

 

[EVNow]

Hmm… Looks like 10-bit made it in to 10.x probably ported/uplifted from Beta 11?

View attachment 756266

Here is some interesting discussion about this on Reddit. Follow the thread from the post below.

https://www.reddit.com/r/teslamotors/comments/s5eor0/_/hsx4itd

 

[Mardak]

Here is some interesting discussion about this on Reddit

https://www.reddit.com/r/teslamotors/comments/s5eor0/_/hsx4itd

Decades designing CCD cameras sounds like for human use and not designed for neural networks. In particular "debayering… otherwise the data is pure crap" is probably true for "normal" use as the mosaic pattern does make it hard for humans to perceive what's going on. Here I've rendered HSL gradient across hue and lightness with what a RCCB pattern might see (2x2: top left red, bottom right blue, others clear):

Especially for this gradient pattern without "sharp" edges, one could somewhat recreate the human pleasing image on the left from the data on the right (except for the lack of "green filter" with "clear" helping detect low light instead). To be clear, the right side of the image is only reds and blues and gray, but at some zoom levels, it looks like more than those colors.

Neural networks trained on this unprocessed data can still learn the patterns for what to predict as it's just a big matrix of numbers anyway.

 

[Dutchy Ron]

This would be a very interesting update - which can tell us whether the TomTom maps updates we have done are flowing into the car.

Tesla to update navigationa maps soon [now live]

Tesla uses a combination of Google data and Tesla's own generated data to provide a robust map and routing experience. While some map-related informatio

www.notateslaapp.com

According to this website Tesla is only using Google Maps and their own Maps.

I wish they used the (ADAS/HD) maps like Lexus and Genesis do

“Tesla uses a combination of Google data and Tesla's own generated data to provide a robust map and routing experience.”

 

[JulienW]

Tesla to update navigationa maps soon [now live]

Tesla uses a combination of Google data and Tesla's own generated data to provide a robust map and routing experience. While some map-related informatio

www.notateslaapp.com

According to this website Tesla is only using Google Maps and their own Maps.

I wish they used the (ADAS/HD) maps like Lexus and Genesis do

“Tesla uses a combination of Google data and Tesla's own generated data to provide a robust map and routing experience.”

Don't believe that is correct or complete info. Tesla likely aggregates map info from TomTom, Open Street Maps and probably several other sources. Not sure if they use Google Maps data but I think (could be wrong) Google requires exclusivity (no aggregating with other sources) when using their data. However the Navigation Database is then compiled by Tesla and that is the update we will be getting. Happens about 2x a year and is usually around 2 to 3GB in the US.

 

[Mardak]

Tesla likely aggregates map info from TomTom, Open Street Maps and probably several other sources. Not sure if they use Google Maps data…

It does seems like Tesla maps/navigation functionality uses multiple sources. Even for the same feature such as the navigation route, it seems like there's differences in offline and online behaviors (more than just avoiding traffic) where I've noticed an initial "good route" get replaced by a bad one when connectivity is restored.

• Google Maps is clearly shown for the map display of roads, point of interest pins, etc. Seems like it (or at least something remote/Tesla server) is used for search and address lookup (for long press or selecting a pin).
• TomTom seems to be used for road connectivity, and these can be seen with the blue lines drawn on top of the map during navigation. It's most obvious when these lines don't match up with the Google Map road data.
• OpenStreetMap type data seems to be used for additional road attributes as a crutch for FSD Beta. This can show up as messages such as "switching lanes to follow route" as it believes there might be fewer lanes ahead or your lane might be forced to turn. Could also be for various other highway features such as stop signs and traffic lights where messages indicate Autopilot is slowing down for a signal that is not possible to be seen, e.g., just behind a bend.

FSD Beta has gradually been adding capabilities to at least improve road attribute predictions such as intersection, static objects and lane properties. Some of these have required fundamental changes to the neural network architecture such as adding spatial memory. I would assume at least some of FSD Beta 11 will continue improving the learning capability of the network structure in addition to training on more variety of data.

 

[gmoe]

How do you do a reset? My score is always at 95 because of following to close. I live in Atlanta and there is no way i can I can not follow close on the highway

 

[JulienW]

.....Google Maps is clearly shown for the map display of roads, point of interest pins, etc. Seems like it (or at least something remote/Tesla server) is used for search and address lookup (for long press or selecting a pin).....

But this could just be UI/satellite image overlay only. Google (and Apple) allows apps to use their Map images. Tesla even uses Apple Map images in iOS App under Location.

 

[VValleyEV]

How do you do a reset? My score is always at 95 because of following to close. I live in Atlanta and there is no way i can I can not follow close on the highway

Try using cruise control without the lane keep. You still get credit for the safety of your driving, but the car will be in control of following distance not you. If someone cuts in front of you, try to resist the temptation to touch the brake, letting the cruise do the braking.

[EDIT] Also, resist the temptation to move to the passing lane close behind the first car in line as it passes you (you will need to exit cruise because it will not auto-lane change). Model the behavior of what you see from FSD, don’t move into that lane until you have a good following distance, and if that means you miss squeezing in at the first opportunity, live with it, remember that makes you a safer driver (yeah, this one was difficult for me).

This helped get me through to the FSD beta, but also taught me to keep more distance, which I suspect is possible even in Atlanta.

 

[gmoe]

ill try it thanks

 

[VValleyEV]

ill try it thanks

See an update in an edit to my post, advice which I recall being critical, and a good lesson in moderating my aggressive driving habits.

 

[EVNow]

Tesla to update navigationa maps soon [now live]

Tesla uses a combination of Google data and Tesla's own generated data to provide a robust map and routing experience. While some map-related informatio

www.notateslaapp.com

According to this website Tesla is only using Google Maps and their own Maps.

I wish they used the (ADAS/HD) maps like Lexus and Genesis do

“Tesla uses a combination of Google data and Tesla's own generated data to provide a robust map and routing experience.”

That guy has no special knowledge about maps than us.

Tesla has never acknowledged using others maps (apart from Google onscreen). They might be found some kind of integration on their own - we’ll get a confirmation after the latest map update.

 

[Mardak]

probably the bigger news is doing some of the 3D environment building in NN instead of C++ code. That is the first point

The first point for improved intersections has multiple items that Elon Musk mentioned in the Lex Fridman interview including the bag of points and compiler. But I think more interesting is "… first model deployed with an auto-regressive architecture…" which might be describing what Karpathy called "Spatial RNN" at AI Day:

You can image that we've now given the power to the neural network to actually selectively read and write to this memory. So for example, if there's a car right next to us and is occluding some parts of the road, then now the network has the ability to not write to those locations, but when the car goes away and we have a really good view, then the recurring neural net can say "okay, we have very clear visibility; we definitely want to write information about what's in that part of space."​

So to better predict intersections (and down the line moving and static objects), this new auto-regressive architecture internally improves its understanding dynamically. This is more than "just" memory and seems to allow for some amount of "reasoning" to progressively combine a partial view a few seconds/meters ago with what's partially visible now to guess at what the whole view should be.

I don't know if this is entirely comparable, but it reminds me of having multiple perspectives / 2D shadows to reason about what's the actual 3D shape:

 

[Mardak]

@greentheonly has a well deserved reputation for posting upcoming Tesla software/hardware changes.

https://twitter.com/x/status/1482631022302875649

@verygreen are you able to view individual road attributes for the map data, and if so could you share what values they have, e.g., Way: ‪Deer Creek Road‬ (‪393469133‬) | OpenStreetMap at Tesla's Palo Alto old headquarters, or some examples of tags/attributes changed between map versions?

 

[Raurele]

How do you do a reset? My score is always at 95 because of following to close. I live in Atlanta and there is no way i can I can not follow close on the highway

Put it on AP when on the on ramp before you hit 50, let it take you off the freeway and dont turn off AP until you're below 50. you will get 0.0 close follow every time. I live in the Bay Area where its even harder when everyones going 80.

 

[LastGas]

How do you do a reset? My score is always at 95 because of following to close. I live in Atlanta and there is no way i can I can not follow close on the highway

Following too close only applies to speeds 50 mph and over. The score is based on a ratio of the time driving under 1 second to the time driving 1-3 seconds behind the other car. To get a good score you have to have a small number on the top and a big number on the bottom.

What screws you is Autopilot. Neither the bad driving nor the good driving accumulates when Autopilot is active. If you use Autopilot "all the time" like I would if I were driving the Interstate in Atlanta, the only time that counts is the scraps when Autopilot gets turned off, such as when you're making a difficult maneuver.

If you want a good following score, turn off Autopilot and let long stretches of normal driving add up in the divisor. As long as you can keep it over 1 second a fair amount of the time, you're cool.

I turned off Autopilot and my safety score jumped from all over the map to a consistent 100. When the average hit 97, I got the Beta.

 

[LastGas]

Put it on AP when on the on ramp before you hit 50, let it take you off the freeway and dont turn off AP until you're below 50. you will get 0.0 close follow every time. I live in the Bay Area where its even harder when everyones going 80.

That would only work if you could drive without ever turning AP off. At least for me something is going to happen where I have to intervene and that ruins the score because with AP on, you never accumulate any good driving to compensate for the exceptions.