FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[powertoold]

Watch at 8 minutes at a 4 Way Stop. It hesitates and 4 cars cut in front of it.

That's not a big problem from a robotaxi standpoint, as long as it's safe. These little details will be worked out, and most of these issues stem from the 0mph recall. If you stop to 0mph, others will take your right of way.

 

[JulienW]

That's not a big problem from a robotaxi standpoint, as long as it's safe. These little details will be worked out, and most of these issues stem from the 0mph recall. If you stop to 0mph, others will take your right of way.

Don't disagree at all but as an in city driver (and sufferer of "get off my lawn" disease ) I can't stand sitting and watching others take advantage of "my" timidness.

 

[kabin]

Watch at 8 minutes at a 4 Way Stop. It hesitates and 4 cars cut in front of it.

And it reduces the utility of FSD for those that don't want to clog up normal traffic flow. For me the jury is still out whether this systemic sluggishness can be solved. The more things change the more it seems to remain the same.

 

[kabin]

Don't remember seeing this "influencer" before.

Not many too many followers but he comes up on twitter occasionally.

 

[JulienW]

And it reduces the utility of FSD for those that don't want to clog up normal traffic flow. For me the jury is still out whether this systemic sluggishness can be solved. The more things change the more it seems to remain the same.

Kinda reminds me of a video or podcast from a few years ago. Can't remember details and this is based off my "bad" memory but the person was an expert on autonomy and had consulted with some sociologists and the consensus was that once most cars were autonomous that drivers would cut them off at every opportunity or cut in front of them knowing they were passive and would simply yield. Also when all cars were autonomous that city traffic would come to a near stand still. VRUs would simply take over the streets and never look or care since all the cars would yield to them without passion or aggression.

 

[sleepydoc]

Wonder which one is being displayed
If the "regular" walking human wasn't there would it "do further damage"

Via DirtyTesla video

Yeah, I noticed that, too. Seemed to have a hard time recognizing the person on crutches. I wasn’t sure if it was the crutches, the way she was walking with them or because there were two people close together.

Or maybe Tesla thought it could get double points by knocking one crutch out without actually hitting the person.

 

[Ramphex]

Not sure if that is supposed to be sarcasm or true but either are incorrect. So far looks like a stereotypical rollout of a major point release. We should expect it to go a little slower since it is a full version release and not just a point release. Elon even said it "needs more polish" before going wider.

It’s 21st century, I can be impressed by anything! Are you shaming me?!?

 

[sleepydoc]

Yes. The speed limit and max speed changed at the sign, and the car did begin to slow down (just as it did in earlier versions). But the deceleration was very gradual, and the driver could have easily received a ticket before the car reached the speed limit.

Yup - looks like the identical behavior from 10.69. I assume they probably ported over the same code but it’s frustrating they didn’t make such a simple fix.

 

[sleepydoc]

Don't remember seeing this "influencer" before.

He’s from Minneapolis! Elon finally cares about someone who’s not in California!

He actually has a thread here on TMC and periodically posts drive videos. The one’s I’ve seen are good (i.e. Balanced and honest)

 

[JulienW]

It’s 21st century, I can be impressed by anything! Are you shaming me?!?

Now I'm envious. Was impressed by almost anything in the 20th century but now I'm more jaded. Looks like you have shamed me.

 

[kabin]

Kinda reminds me of a video or podcast from a few years ago. Can't remember details and this is based off my "bad" memory but the person was an expert on autonomy and had consulted with some sociologists and the consensus was that once most cars were autonomous that drivers would cut them off at every opportunity or cut in front of them knowing they were passive and would simply yield. Also when all cars were autonomous that city traffic would come to a near stand still. VRUs would simply take over the streets and never look or care since all the cars would yield to them without passion or aggression.

Exactly. There are online videos of robotaxis acting like timid deer in the headlights so surrounding traffic pick them off with little regard. Robotaxis will need to be safe and as aggressive as a normal driver to gain respect otherwise customers will be pushing red buttons to exit robotaxis rides in the middle of busy traffic.

 

[jabloomf1230]

Don't remember seeing this "influencer" before.

Isn't that the guy from MN? He's posted videos before.

Minnesota - FSD Beta Videos

Video #79 - Tesla - FSD Beta 10.69.25.2 - Brief Drive Through St. Paul This video contains footage recorded on the evening of January 19th, 2023 of FSD Beta 10.69.25.2 briefly driving me through Downtown St. Paul. St. Paul is known for it's interesting street layout and this is the first of...

teslamotorsclub.com

 

[aronth5]

Don't remember seeing this "influencer" before.

He has posted a lot of videos and is a good "influencer" to watch.

 

[AlanSubie4Life]

If you stop to 0mph, others will take your right of way.

What on earth are you talking about?

People get very upset if you don’t stop at a 4-way stop. It is expected and part of the process. The only one who will take right of way will be someone who does not stop.

4-way stops work really well. The only thing that really causes problems are people being too timid and not going quickly when it is their turn.

Stopping to 0mph is obviously not a problem. Never causes problems.

 

[powertoold]

What on earth are you talking about?

People get very upset if you don’t stop at a 4-way stop. It is expected and part of the process. The only one who will take right of way will be someone who does not stop.

4-way stops work really well. The only thing that really causes problems are people being too timid and not going quickly when it is their turn.

Stopping to 0mph is obviously not a problem. Never causes problems.

If you and another person approach a stop sign close to the same time, but you are first and stop to 0mph, it largely increases the likelihood of stop sign confusion, where the other person will creep up because they think you're too slow. In the case of fsd, when the other car creeps, then it thinks other car might go, so fsd stops (because it's conservative) and the cycle continues. It happens literally all the time in urban centers. Not sure if you drive in those environments much.

 

[AlanSubie4Life]

If you and another person approach a stop sign close to the same time, but you are first and stop to 0mph, it largely increases the likelihood of stop sign confusion, where the other person will creep up because they think you're too slow. In the case of fsd, when the other car creeps, then it thinks other car might go, so fsd stops (because it's conservative) and the cycle continues. It happens literally all the time in urban centers. Not sure if you drive in those environments much.

Lol. Take a video.

You have expressed that you get honked at at 4-way stops so I am curious about the dynamics here. I have not had this problem.

I have never had a problem with what is described.

One stops, then one goes cautiously. If someone is very assertive about their turn, you let them continue through the intersection, as you continue to proceed into the intersection. They’ll be quick about it anyway.

If you’re definitively first, you stop, then you go, assertively, stopping well into the intersection if someone asserts their turn incorrectly (then you rapidly proceed).

People go out of turn all the time. Not a big deal. Just keep things moving! The most important thing is to proceed rapidly when it is your turn.

4-way stops are not complicated. Just go after stopping, and don’t run into anyone. Particularly easy for a computer which can see all three other positions precisely and knows exactly who was first.

I’m not responsible for FSD stopping when someone else starts to proceed. Sounds like they got the coding wrong there - the car should definitely keep going! Once you have stopped you definitely need to get moving. The safety driver is there to intervene in any especially tricky situations so not really any risk in just going. (Reality is there is zero risk since things are low speed and reaction times are zero when properly coded - but the car must proceed when others are proceeding -this is critical of course.)

4-way stops are one of the areas where autonomous vehicles can be a lot more aggressive than humans, because they have all the information and can process the entire scene (including pedestrians who may block some people from proceeding, etc.) without error, so it is a shame to see it being thought of as a special challenge or a place where humans can take advantage- they won’t be able to; the AVs will dominate these scenarios.

 

[sleepydoc]

If you and another person approach a stop sign close to the same time, but you are first and stop to 0mph, it largely increases the likelihood of stop sign confusion, where the other person will creep up because they think you're too slow.

actually I find the reverse - if you can see one car clearly stop as evidenced but he slight backward lurch it makes it more clear that that car should proceed first. I will often intentionally fully stop sooner or later so the other car can tell which order to proceed in.

4 way stop signs are actually a very difficult problem for FSD since there are a lot of unspoken visual cues between drivers. I would also imagine that these cues vary somewhat from region to region.

 

[powertoold]

One stops, then one goes cautiously. If someone is very assertive about their turn, you let them continue through the intersection, as you continue to proceed into the intersection. They’ll be quick about it anyway.

That's exactly what fsd is contending with. The other car will be more assertive than the one stopping to 0mph if both approach the stop at similar times. Even if the 0mpher is first to approach, the other car creeps forward, more likely causing a dance of confusion about who's gonna go (especially with fsd systems where it's more conservative and would pause at any signs of forward creep).

 

[AlanSubie4Life]

That's exactly what fsd is contending with. The other car will be more assertive than the one stopping to 0mph if both approach the stop at similar times. Even if the 0mpher is first to approach, the other car creeps forward, more likely causing a dance of confusion about who's gonna go.

I think you are thinking that stopping to 0mph is the problem, when the real issue is that FSD is too cautious and generally incorrectly coded (stopping to 0mph being one of the few pieces of the puzzle which is correctly executed).

The piece you have highlighted involves an issue after the stop. Not the stop. At that point the Tesla will be well into the intersection aggressively moving forward, and potentially stopping.

 

[powertoold]

the real issue is that FSD is too cautious and generally incorrectly coded (stopping to 0mph being one of the few pieces of the puzzle which is correctly executed).

The real issue is that in SF (and other dense areas), in the situations I talked about, no one assumes the other car will completely stop (because less than 5% of people do that, I'd say it's around 2% lol).