FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Dewg]

Let's rephrase it this way.
If today Tesla said they have introduced a driver assist feature called Stop sign detect and Auto stop. This feature could be subject to recall if it failed sometime to detect Stop sign and stop? If that's the case, why would car manufacturers introduce more advanced driver assist features.

I think yes, that would be the case. It's an unusual ODD for an L1 or L2 feature, so not sure why it would be implemented, but yes, it would be a safety issue and could be recalled.

 

[swedge]

Remind me not to try to post from my iPhone.

 

[MP3Mike]

I suspect if Tesla did the iniital rolls at say 1-2 mph we'd probably be here with 1-2mph rolling stops. Tesla most likely created this problem themselves.

I doubt it, I'm sure the TSLAQ people would have still hounded the NHTSA with complaints about it rolling stop signs.

 

[AlanSubie4Life]

This whole “rolling stops should be allowed, blame the NHTSA” fake outrage is so silly.

That’s not the problem with FSD. The problems are:

1) Too slow on approach.
2) Doesn’t stop quickly enough when it comes to an actual stop. Takes forever…just apply the brakes gently at the stop line!
3) Stops at the line which is often inappropriate. Though required by law, and it is still possible to have a decent stop doing this. This is something that humans definitely don’t do (and it is not even clear that people get ticketed for it, since stop lines are often poorly placed, and it is unsafe to stop at that line) so the stops will never be as good as long as this persists. But it could be ok.
4) Creeps interminably after the stop line looking for visibility. This is the biggest issue. It used to just go (often disconcerting!). I think it should creep, really quickly. Then go. As is it is really slow, and it takes a fairly long time to begin the creep.

At no point is the issue the actual stopping (except of course as outlined the amount of time it takes to actually halt).

In any case it is well within Tesla’s control to make it decent. It should be possible to emulate a human stopping at the stop line.

Disagree from @Dewg, who doesn’t have v11 yet, AFAIK? Anyway the behavior is quite different in v11. Part of it is the extended stopping to 0mph (which happens slowly), but the creeping after the stop is very different than 10.69. Fortunately I have video evidence which eventually I will post, and we can time each portion of the process and compare different versions.

 

[EVNow]

Simple solution is Tesla open sources their code ala Open Pilot, and if you want to roll stops like any sane person then you can go and rebuild it with ROLL_STOPS=1 and NHTSA can't do anything about it.

They don't need to open source - we know they already have many of these as configurable parameters (including stop at yield, rolling stop etc). Currently they are all available only in "dev" mode .... they just need to have a "secret" code that we can use to tinker with these settings.

 

[PianoAl]

As a man?

Disguised as Elon musk.

 

[MARKM3]

Any news yet for 2023 branch and V11 merge?

 

[snorp]

Just finished a 1000 mile roadtrip on 11.3. This is the first time I feel that AP1 is no longer the best option for highway driving.

The biggest improvements for me were the gradual accelerations and decelerations, moving over in my lane for semis, and moving to the left lane when coming up on slower traffic without slowing down first. The most annoying thing was the large following distances that allowed people to pass me in the right lane while in traffic, and then I had to pass them again once traffic cleared up.

Opportunities for improvement include merging right when a left travel lane ends, merging at the last minute for an exit, using turn signals when a travel lane splits and can go two ways, and maintaining a relatively constant distance behind another vehicle.

It was a relatively stress free drive on the highway.

 

[Dewg]

Just finished a 1000 mile roadtrip on 11.3. This is the first time I feel that AP1 is no longer the best option for highway driving.

The biggest improvements for me were the gradual accelerations and decelerations, moving over in my lane for semis, and moving to the left lane when coming up on slower traffic without slowing down first. The most annoying thing was the large following distances that allowed people to pass me in the right lane while in traffic, and then I had to pass them again once traffic cleared up.

Opportunities for improvement include merging right when a left travel lane ends, merging at the last minute for an exit, using turn signals when a travel lane splits and can go two ways, and maintaining a relatively constant distance behind another vehicle.

It was a relatively stress free drive on the highway.

Did you play with the 3 modes during your drive? Chill, Average, Assertive? I'm curious how the follow distance would change for you.

 

[snorp]

I left it in assertive and did not select limit lane changes. I didn't change it throughout the trip.

Now that I think about it, it never once moved out of the left lane on its own unless the right lane was moving faster. It doesn't bother me as I prefer to drive in the left lane and will move over if I see someone coming up on me. The road noise is usually much less in the left lane, I presume due to semis tearing up the right lanes. I also drive fast so i don't have to move over that often.

 

[SSDTester#1]

I left it in assertive and did not select limit lane changes. I didn't change it throughout the trip.

Now that I think about it, it never once moved out of the left lane on its own unless the right lane was moving faster. It doesn't bother me as I prefer to drive in the left lane and will move over if I see someone coming up on me. The road noise is usually much less in the left lane, I presume due to semis tearing up the right lanes. I also drive fast so i don't have to move over that often.

You can tell a difference in noise levels between lanes?

 

[PianoAl]

Has anyone here heard an update on the supposed 10k miles restriction? Elon tweeted that those with 10k autopilot miles can get the DM/hand req. removed but I can’t find anything recent on it.

Perhaps they changed their mind on that, and they should. I don't like nags, but I need them to keep me from getting distracted. I doubt they have evidence that the nags don't help.

 

[sandman1330]

Imagine the AP needed you to jerk off the yoke while staring out of the window nonstop

What, in your opinion, do you think is more impressive, Tesla’s FSDb or Cirrus’ Safe Return?

Solid point about the Safe Return - I've read about it but I'd forgotten that existed. It's a neat function and probably a good comparison.

I think the difference comes down, again, to traffic. On empty roads, I'll let FSDb do it's thing. It will hesitate, stop where it shouldn't (like the middle of intersections), swerve, etc - but none of that matters on empty roads. Yes, FSDb can safely and intervention free get me where I want to go - on empty roads. Cirrus' safe return is similar. As soon as the system changes the squawk to 7700 and declares the mayday, air traffic control will remove everyone and everything from it's path. And yes, it can then safely land the aircraft. But it would not, in it's current form, be capable of integrating itself into a busy traffic pattern and safely landing in the presence of other aircraft. I think it's a great parallel with FSDb. Another key element here is that the mayday declaration, by law, allows the aircraft (pilot?) to deviate from rules because of the emergency situation. FSDb still needs to adhere to all those rules. It would be as if you put police lights and sirens on the car then let FSD drive it - would it do better with everyone moving out of it's way and no requirement to obey traffic laws?

On the subject of safety interventions vs quality of life interventions, I had an epiphany driving on FSDb just a half hour ago, to which I'll apply another aviation comparison. I'm not just a professional pilot, but I'm a check pilot - that means I conduct annual and category upgrade check flights on other pilots. Depending on the type of check, the most basic principles we look for are two criteria that must be fulfilled to be considered a pass:

1. Was it safe;
2. Was it effective (in other words, did they achieve the mission or successfully complete the task?)

FSDb, I believe, will most times pass on point number one. However, on point #2, it fails miserably if presented with anything but empty streets. It is like a pilot who is so overly cautious that he's afraid to touch down on landing and keeps going around, or afraid to get right down to minimum descent altitudes on an instrument approach for fear of dipping below them and failing #1. He might be safe, but he's still failing the checkride on point #2. I feel like this is the stage FSDb is in right now. It's soooooo overly cautious, all the time, that it fails to be actually effective in driving the car. The vast vast majority of my (many) interventions are to keep it actually on task and avoid upsetting others on the road. While it's approaching safe, it's still very far from being effective.

 

[snorp]

You can tell a difference in noise levels between lanes?

Absolutely. There are a lot of highways out there that have chewed up right-hand lanes that are visibly missing asphalt where the tires meet the road. When there is a noise difference, it is generally a very large difference.

I find this more common on 2-lane interstates than 3-lane interstates.

 

[Dewg]

I left it in assertive and did not select limit lane changes. I didn't change it throughout the trip.

Now that I think about it, it never once moved out of the left lane on its own unless the right lane was moving faster. It doesn't bother me as I prefer to drive in the left lane and will move over if I see someone coming up on me. The road noise is usually much less in the left lane, I presume due to semis tearing up the right lanes. I also drive fast so i don't have to move over that often.

What is the follow distance on Assertive? Can you provide a time, and an approximate car length? I've read that Chill is about 1.5 seconds. Assuming 65MPH that's about 140 feet of distance in 1.5 seconds, which is about 9 car lengths (assuming average car length is about 15 feet).

 

[PianoAl]

I like reporting serious problems with the disengagement what-went-wrong speech option and have done so about ten times in the last week. But how can engineers handle the many reports they must be getting?

 

[snorp]

Can you provide a time, and an approximate car length?

Not sure about that, but it definitely seemed to other drivers, and I agree with them, that I was just holding down the left lane with no intention of keeping up with traffic. People passed me on the right and merged in front of me with ease. They didn't have to squeeze into a tight spot.

They were probably pleased that I didn't speed up to close the gap, but I wanted to see how the software would perform. And in the grand scheme of things, a couple cars ahead of me that I would have to pass again later didn't materially impact my travel time.

 

[Dennisis]

The most annoying thing was the large following distances that allowed people to pass me in the right lane while in traffic, and then I had to pass them again once traffic cleared up.

I’m talking about traffic is slowed and I’m not at set speed and waiting for traffic to clear so we can all get up to speed. Except I’m stuck too far back so people get frustrated behind me and zip around me just to fill the gap! They don’t need to change lanes they just want me to keep up with the cars ahead and if I won’t do it they will. Not talking anything malicious here.

Exactly! Same as I posted last week, annoying as hell. Some, including those that don't yet have the update, keep referencing seconds following at different speeds. In real life you're too far back in traffic, while just cruising on the open road it's fine. But most of us have traffic to deal with.

 

[AlanSubie4Life]

What is the follow distance on Assertive? Can you provide a time, and an approximate car length? I've read that Chill is about 1.5 seconds. Assuming 65MPH that's about 140 feet of distance in 1.5 seconds, which is about 9 car lengths (assuming average car length is about 15 feet).

It’s highly variable and dependent on changing speeds ahead. No mode keeps up as robustly as manually driving with the traffic ahead. It seems improved vs. the wide NOA release though; it is not so slow to respond.

Assertive seems to be about 1.5 seconds minimum (excluding cut-ins). Chill seemed to be about two seconds this morning but it was variable and it seemed to allow greater distances before closing up again.

This was at around 70mph.

A little unclear what the steady state is (have not been able to get behind someone on a straight freeway who is using cruise control). Again, the distances are highly variable if the traffic speeds are not constant, a big difference vs. manual driving, which can smoothly track such changes much more effectively.

That’s the biggest issue and I don’t expect it to be fixed because they cannot figure out how to be smooth.

The closest following distance is plenty close and people cutting into that decent-sized gap will likely not be dissuaded by even shorter and unsafe gaps. The bigger issue, where cut-ins occur probably more frequently, is when the speed fails to adjust quickly to accelerating traffic ahead. Again, it is better than NOA was. But still lacks alacrity. (This of course results in the very poor behavior referenced earlier when traffic slows ahead too quickly and exceeds the control limits.)

Steady state, on Assertive, no one is going to accuse you of not keeping up with traffic, and there will be no reason to cut in for that reason (though many will, especially when traffic is moving around to get to appropriate lane, new entrants are coming up to speed, etc.)

 

[JB47394]

I like reporting serious problems with the disengagement what-went-wrong speech option and have done so about ten times in the last week. But how can engineers handle the many reports they must be getting?

It's just data. I'm sure they convert it to text. There are myriad data analysis techniques that could be applied to it. Imagine creating a word cloud from the reports. That might be very informative to the developers. They can also search for words used as a means of further understanding the situation. For example, it's a lot easier to search for reports including the word "traffic" than it is to go through all the reports and figure out how many cars are nearby in each one. I'm sure I'm just scratching the surface of the possibilities.

If you're wondering if people are watching our reports and listening to our comments, then the answer is a definite no.