FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[willow_hiller]

When do you guys think we'll have school zone awareness and/or no turn on red? Feels like those are easy enough to implement and absolutely necessary to fix ASAP because you cannot be a full self driving car without understanding these (non) edge cases.

Part of me wonders if Tesla is intentionally leaving out features in order to remain classified as level 2 for as long as possible while working on the harder fundamentals.

From Tesla's communications with the California DMV in late 2020:

"That said, we do not expect significant enhancements in OEDR or other changes to the feature that would shift the responsibility for the entire DDT to the system. As such, a final release of City Streets will continue to be an SAE Level 2, advanced driver-assistance feature."

So the crux of their argument for remaining classified as Level 2 is that they are not expecting to add additional Object and Event Detection and Response (OEDR). It's still Level 2 because you are required to intervene for things like No Right On Red and school zones. And as soon as Tesla starts recognizing and responding to those objects, California is going to expect Tesla to report all of their disengagement data to the DMV.

Worth noting that the same email ends with:

"Please note that Tesla’s development of true autonomous features (SAE Levels 3+) will follow our iterative process (development, validation, early release, etc.) and any such features will not be released to the general public until we have fully validated them and received any required regulatory permits or approvals."

So they are eventually planning on more OEDR, but maybe not until the overall stack is ready for the driver to fully take their hands off the wheel and eyes off the road.

 

[EVNow]

I still don’t use it around a school zone at all, and certainly not during the morning or afternoon coming and goings - if I’ve been so stupid as to come and go at the same time!

Yes - school zones are a problem. Usually there will be lead cars so the car slows down (and I reduce speed limit).

If there is no lead car, you have to reduce speed quite a bit in advance sine FSDb takes a long time to reduce speed.

 

[Twiglett]

I've been saying this for some time - Beta is for a certain type of person and not for everyone. It takes a commitment to the process, and a desire to test, report, and help improve it over time. You "just don't get how people enjoy navigating on FSD beta when it requires consistent intervention". It's because we want to be Beta testers and we want to help improve the system for people like you who benefit from our time and effort.

I agree with your statement - except that FSD is no longer in what most people would call a beta process or one that has us reporting issues.
Its now in the same beta phase that general AutoPilot has been ever since it was introduced. Tesla are saying its available to anyone who wants it with zero entry requirement other than owning FSD in some form.
There is no "report"
Many would take issue with Tesla using the Beta tag at all in the context of delivering to the general public.

 

[JulienW]

Part of me wonders if Tesla is intentionally leaving out features in order to remain classified as level 2 for as long as possible while working on the harder fundamentals....

Not really because Tesla could have L5 equivalent driving but as long as they don't apply it will remain regulated as L2 and driver responsible.

I suspect it is because School Zones and No Turn on Red signs are extremely varied, often hard to identify and often difficult to understand.

 

[EVNow]

Yes, you can see "Roadway Type" and probably infer from "Posted Speed Limit (MPH)" and some other data. You can download the csv data with the "Level 2 ADAS Incident Report Data" button from Standing General Order on Crash Reporting | NHTSA

I believe Report ID 13781-4338 (currently row 50) is the San Francisco Bay Bridge crash that FSD Beta 11 single stack might have avoided.

I downloaded the csv and I'm looking at it. There are a number of fields as you note that give some clue. I'll start a new thread to talk about the accident report.

 

[JulienW]

Elon is getting really bad since he is now missing his "V11 in a week or so" posts. Elon should at least give us a "Will have an update on the release time line for V11 in a week or two" Tweet.

 

[unk45]

Same here. After you gain the confidence in its abilities you learn to relax but stay observant and ready to intervene. Just like airline pilots use autopilot. Not at the beginning or end of flights. For now anyway.

FWIW most large airplanes and many smaller ones have ‘Autoland’. The details go OT. However, crew training and certification is required. Even though autotonomous drones have existed for years, in nearly all deployment a remote pilot does monitor and guide operations.

It seems to me that technologies have been safe enough for some time to eliminate human pilots. Today most aircraft accidents result from human error, not machine error.
Just as with road going vehicles an accident while fully automated would be catastrophic. Human error is somehow more tolerable.
Realistically we are nearly at that point. FWIW fully automated trains are not uncommon and not even controversial, but not really promoted as such.

in context I have flown aircraft with autoland capability and compare those with FSD. The aircraft to which I refer was among the earlier ones with that capability. In both cases the workload in trips is much reduced compared with only human. In both cases failures would have probably caused accidents without my intervention. ‘Probably’ because both might have corrected incipient errors.

True Level 5 is unlikely to happen very soon, I think, because there are nearly infinite edge cases. Aircraft are easier simply because there are precise measurements of virtually every plausible path. Short of a míssel attack or catastrophic failure automation is ‘easy’.
Without quite major advances that will not be true for cars. Some kind of major AI breakthroughs are happening, certainly, but regulatory lags will even inhibit true FSD for a long time.

I’ve paid for FSD three times and will continue. I just will not be disappointed if it doesn’t happen as level 5 in my lifetime.
I hope Elon will be correct. I think he will be. As usual timing is a different issue.

 

[mtbwalt]

No turn on red may be easy enough, presuming FSD is reading all signs or maybe can crowdsource the frequency of turns on red by human drivers.

School zones seem more complicated. Different states and municipalities have differing signage and rules which can make it hard. Some signs say things like “School Days/7:30-8:30/2:30-3:30” and others “when flashing” etc. others borrow the symbols from school zone signs for library “No bully” signs. Definitely a trickier situation to implement. Not sure anyone outside of the dev team knows where they are on that.

 

[Twiglett]

For me the question on the unified stack has been raised in other threads - which set of options will we get.
Will it be the AP options where we can set follow distance, lane change behavior like current AP
or will we be stuck with the appalling FSD not-controls where there is no noticeable difference between Chill and Assertive.
FSD lane changes are its weakest feature, were you have one or two indicator blinks before it just starts moving and you have to make a snap decision on whether to let it happen or not. Worst is that we have no control over that behavior, unlike AP.
My hope is that we get AP style controls, but we will probably get FSD style and no control.

 

[sleepydoc]

I suspect it is because School Zones and No Turn on Red signs are extremely varied, often hard to identify and often difficult to understand.

Yes. There was another discussion on this some time ago and particularly with school zones there appears to be no consistent standard on how they are designated. Even within a state there will be variability. The way I see it, if I have a hard time figuring out whether school zone limits apply how can I expect FSD to know?

No turn on red should be simpler as there’s less variability in the signage. There’s still a fair amount, though.

 

[EVNow]

No turn on red should be simpler as there’s less variability in the signage. There’s still a fair amount, though.

One big variability is the way it is observed. Most people in my area just ignore no turn on red. Atleast at most intersections - they consider it just a yield. If you wait at a no-turn-on-red you will get a honk !

 

[Dewg]

Yes. There was another discussion on this some time ago and particularly with school zones there appears to be no consistent standard on how they are designated. Even within a state there will be variability. The way I see it, if I have a hard time figuring out whether school zone limits apply how can I expect FSD to know?

No turn on red should be simpler as there’s less variability in the signage. There’s still a fair amount, though.

 

[Zane Liu]

View attachment 901068

Honestly even as a human driver I wouldn’t be able to figure that out mid drive

 

[FloridaJohn]

Yes. There was another discussion on this some time ago and particularly with school zones there appears to be no consistent standard on how they are designated. Even within a state there will be variability. The way I see it, if I have a hard time figuring out whether school zone limits apply how can I expect FSD to know?

No turn on red should be simpler as there’s less variability in the signage. There’s still a fair amount, though.

Seems like something like that would be pretty easy to catalog in a database.

 

[tivoboy]

No turn on red may be easy enough, presuming FSD is reading all signs or maybe can crowdsource the frequency of turns on red by human drivers.

School zones seem more complicated. Different states and municipalities have differing signage and rules which can make it hard. Some signs say things like “School Days/7:30-8:30/2:30-3:30” and others “when flashing” etc. others borrow the symbols from school zone signs for library “No bully” signs. Definitely a trickier situation to implement. Not sure anyone outside of the dev team knows where they are on that.

No turn on red (more specifically LEFT turn on red) would have to be incorporating geographic location as well as this is state by state at this point in the USA. Certainly that is possible, but I haven’t seen anything that indicates that current FSDb does state differentiation of state laws yet?

anyone seen something like this?

 

[JulienW]

I believe we are officially out of Musk's latest V11 to be available Tweets time line. Since the last few have been based on a "week or two week" phrasing and have overlapped what kind of time line will he offer this week? Come on Elon we are thirsty and need a BIG swig of that oh'so delicious Tweeter artificially flavored Kool-Aid.

 

[VanFriscia]

View attachment 901068

Honestly even as a human driver I wouldn’t be able to figure that out mid drive

Agreed. Every time I drive through unfamiliar parts, I'm reminded of how awful our traffic system is (we just get used to the local awfulness because we drive it every day!) I'm actually hopeful that the push for autonomous driving might eventually help clean this stuff up and standardize things a bit. Do we really need 47 different ways to say no turn on red? I know this gets into the weeds with federal vs. state laws, but come on - there's gotta be a better way than this.

 

[tmoz]

View attachment 901068

One could crash while reading that. 6:45 to 9:00 and 2:00 to 4:30 would be simpler.

 

[heapmaster]

One could crash while reading that. 6:45 to 9:00 and 2:00 to 4:30 would be simpler.

lol ya, by the time you pass it going 40, you have to figure out what time it it exactly to the second, and then try to find that time in a time range of 50 options, this is the worse sign ever. It would be best to just remove the times and use "when flashing", simple for humans and self driving, the flashing light box does all the work.

 

[Gilliland]

I often see school zone signs that say "Speed Limit 20 when children are present". That's a tough one to solve.