FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[willow_hiller]

I'm guessing employees weren't meant to share these again. The original source of pages 1/2 has deleted their tweets. An employee on Reddit posted page 3. Backing it up here:

Edit: full text notes here: FSD Beta 11.3 (2022.45.5) Official Tesla Release Notes - Software Updates

 

[sleepydoc]

The better question to ask is, how many rolling stops has fsdb done in its 90+ million miles of driving, and how many of those rolling stops led to an accident. The recall release doesn't mention any accidents related to rolling stops, only a "risk."

Still don’t get it, do you? (and still not the right question to ask)

 

[MP3Mike]

What I find most interesting is that the nhtsa made Tesla stop at stop signs to 0mph, and now, Tesla has made it 1mph, which isn't following the law to the letter, and the nhtsa has yet to reenforce 0mph.

I don't think the "1 MPH" stop is a rolling stop. It is just how Tesla implemented the requirement that you can never report slower than the actual speed. Elon even mentioned that they had to record video showing that the wheels weren't moving to prove it to NHTSA.

 

[sleepydoc]

I don't think the "1 MPH" stop is a rolling stop. It is just how Tesla implemented the requirement that you can never report slower than the actual speed. Elon even mentioned that they had to record video showing that the wheels weren't moving to prove it to NHTSA.

IME it’s all irrelevant anyway since FSD takes so long to figure out what it’s doing at stop signs. Even at an empty 4-way stop with completely clear sight lines it has to stop and think about what it’s doing.

 

[JHCCAZ]

- Improved lane changes, including: earlier detection and handling for simultaneous lane changes, better gap selection when approaching deadlines, better integration between speed-based and nav-based lane change decisions and more differentiation between the FSD driving profiles with respect to speed lane changes.

This should be a great help in smartening up the misbehavior that causes the majority of my interventions and lane-change overrides. I've been wanting a toggle that would essentially disable speed-based lane changes, but this will mostly be accomplished if the Chill setting starts actually doing its job.

What's not clearly referenced here are the mysterious phantom lane change threats that inappropriately activate the blinker, confusing everyone, and then disappear after three or four ticks. I think these have more to do with confusion in identifying turn vs. through lanes, so some of the other lane-related release notes will hopefully address those.

I'll be following the reports of the early-access influencer users with great interest.

 

[EVNow]

I'm guessing employees weren't meant to share these again. The original source of pages 1/2 has deleted their tweets. An employee on Reddit posted page 3. Backing it up here:

View attachment 909196

Edit: full text notes here: FSD Beta 11.3 (2022.45.5) Official Tesla Release Notes - Software Updates

Wow … lots of nice updates.

 

[JulienW]

Wow … lots of nice updates.

Wonder if this is an employee only feature and won't be in the consumer release/Notes? No way would they remove Camera data feedback and give 400k the ability to record "analog" messages that MUST be listened to then "decoded", tagged and labeled by humans. That would requires 10s of thousands of human employees to accomplish.

 

[TexasDuke]

I wonder at what point where autonomous driving will be able to see the potholes and be able to maneuver around them.

 

[JulienW]

I wonder at what point where autonomous driving will be able to see the potholes and be able to maneuver around them.

Potholes are a lot like porn. We all know hardcore porn when we see it but what about when you get into the gray areas? A large pothole with no cars around is easy to identify and avoid but what as the potholes become smaller and the identification becomes later and more jerk needed to avoid. How to handle if there are cars and/or VRUs around? At what point is avoiding more dangerous that going through. Lots of subjective behavior that will be hard to fine tune to satisfy most people.

EDIT: A good analogy is Speed bumps. Beta is coded to identify them but it is a hodgepodge of responses and non responses to them.

 

[willow_hiller]

This version must be going very wide to employees. A solar roof installer on Reddit just reported receiving it:

https://www.reddit.com/r/teslamotors/comments/116ux0r/_/j98xmg8

https://www.reddit.com/r/teslamotors/comments/116ux0r/_/j98zmin

 

[jabloomf1230]

Potholes are a lot like porn. We all know hardcore porn when we see it but what about when you get into the gray areas? A large pothole with no cars around is easy to identify and avoid but what as the potholes become smaller and the identification becomes later and more jerk needed to avoid. How to handle if there are cars and/or VRUs around? At what point is avoiding more dangerous that going through. Lots of subjective behavior that will be hard to fine tune to satisfy most people.

EDIT: A good analogy is Speed bumps. Beta is coded to identify them but it is a hodgepodge of responses and non responses to them.

Maybe the voice note option in 11.3 could be a precursor to a hazard reporting system. Eh, probably not.

 

[powertoold]

I don't think the "1 MPH" stop is a rolling stop. It is just how Tesla implemented the requirement that you can never report slower than the actual speed. Elon even mentioned that they had to record video showing that the wheels weren't moving to prove it to NHTSA.

Are you sure this isn't related to the first recall of rolling stops (a year ago)?

Tesla to disable ‘rolling stop’ feature after NHTSA says it can ‘increase the risk of a crash’

The update will be applied to nearly 54,000 vehicles

www.theverge.com

 

[Supcom]

Wonder if this is an employee only feature and won't be in the consumer release/Notes? No way would they remove Camera data feedback and give 400k the ability to record "analog" messages that MUST be listened to then "decoded", tagged and labeled by humans. That would requires 10s of thousands of human employees to accomplish.

View attachment 909272

If it were only for employees, then why make it anonymous?

 

[Gbills]

Wonder if this is an employee only feature and won't be in the consumer release/Notes? No way would they remove Camera data feedback and give 400k the ability to record "analog" messages that MUST be listened to then "decoded", tagged and labeled by humans. That would requires 10s of thousands of human employees to accomplish.

View attachment 909272

Maybe AI listens categorize and passes the results along.

 

[Knightshade]

If it were only for employees, then why make it anonymous?

1000% speculation- but Tesla can't guarantee only an employees voice is ever captured so might be some CYA regarding data privacy?

 

[Mardak]

No way would they remove Camera data feedback and give 400k the ability to record "analog" messages that MUST be listened to then "decoded", tagged and labeled by humans

I'm sure the AI team is well aware of the improvements to automatic speech recognition such as OpenAI's Whisper. If 100% accuracy isn't needed yet and data collection is broad categorization of types of issues to get a sense of progress, e.g., 50% of reports about lane changes dropping to 5% over future 11.x releases, this seems like it would be almost fully automated without much ongoing human effort.

 

[Mardak]

the mysterious phantom lane change threats that inappropriately activate the blinker, confusing everyone, and then disappear after three or four ticks

Just making sure, you described these as phantom as the car doesn't actually change lanes? Do they happen relatively consistently at the same locations where there's a "fork" such as a temporary right or left turn lane probably near a curving road? The "forked lane" could potentially even be a driveway.

There's at least one that is confusing for others around me as a right turn lane appears just before an intersection that curves left, so FSD Beta signals left to stay in the current lane. Especially confusing is a new lane starts just past the intersection, so FSD Beta sometimes actually ends up switching right during the intersection after just signaling left but not signaling right as now it's "going straight." So I'll be checking for changes with 11.x.

 

[legendsk]

Potholes are a lot like porn. We all know hardcore porn when we see it but what about when you get into the gray areas? A large pothole with no cars around is easy to identify and avoid but what as the potholes become smaller and the identification becomes later and more jerk needed to avoid. How to handle if there are cars and/or VRUs around? At what point is avoiding more dangerous that going through. Lots of subjective behavior that will be hard to fine tune to satisfy most people.

EDIT: A good analogy is Speed bumps. Beta is coded to identify them but it is a hodgepodge of responses and non responses to them.

Add light show to head lights - now auto high/low beam selection no longer works right. Add rainbow road coloring - now windshield wiper rain sensing no longer works right and wants to wipe constantly a dry windshield on a sunny day. Add Apple music - music on USB no longer works. Add fart noises - I haven't identified the casualties from that.

It already slams on brakes and comes to full stop in high speed traffic lane because a bird flew over and the shadow crossed the road. Add identify and classify potholes and automatically take evasive maneuvers to avoid them? No thanks!

 

[Mardak]

View attachment 909196

• Improved longitudinal control response smoothness when following lead vehicles by better modeling the possible effect of lead vehicles’ brake lights on their future speed profiles.
• Improved decision making at crosswalks by leveraging neural network based ego trajectory estimation in place of approximated kinematic models.
• Improved reliability and smoothness of merge control, by deprecating legacy merge region tasks in favor of merge topologies derived from vector lanes.

Seems like at least these items are part of moving some of "control" to neural networks instead of legacy C++ heuristic approximations. Broadly these were controlling when and how much braking to apply in certain detected scenarios, so there's potential for even more of these to be converted for other situations.

It'll be interesting to see if people notice control improvements more than what has been listed so far or we'll need to wait for future 11.x. In particular, I'll be checking if 11.3 avoids braking for an intersection under a bridge.

 

[Dan D.]

I don't think the "1 MPH" stop is a rolling stop. It is just how Tesla implemented the requirement that you can never report slower than the actual speed. Elon even mentioned that they had to record video showing that the wheels weren't moving to prove it to NHTSA.

If the logs or the speedometer say 1 mph but the car is actually stopped (wheels aren't moving) then Tesla still has no defense. They must stop longer to allow the logs & speedometer to say 0 mph. You can't say "but 1 mph is stopped", after all you could actually be doing 1 mph too. Logs are the evidence and saying 1 mph is evidence of 1 mph.