FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[sleepydoc]

Yes, those were the days. V11, software that time forgot and releases seemingly cannot improve.

Well, since we've been waiting for the fabled v11 for a year now...

 

[AlanSubie4Life]

I think it's obvious that the car can perform zero intervention drives. It's just a matter of getting lucky

Obviously of sufficiently short length this is possible. Hence the requested 30-minute interval. I concede it might be possible, on trivial streets, no traffic, good markings, doing a carefully planned route where failure cases have been pre-screened and avoided, etc. Haven’t seen it though!

One of my most common interventions is to goose the accelerator at a right turn on red or yield.

Yes, this is one common one, and the sort of thing I am talking about. It’s all fine and good possibly if there is zero traffic! But yeah, it is very normal to press the accelerator, and this limits utility for sure.

you would fail FSD if it didn't perform exactly as a human would (and by human you typically mean exactly how you would.)

This isn’t true at all. I just am applying normal standards for typical driving. In the discussion here I’m not failing it for unpleasant braking behavior or slow accelerating behavior unless it is impeding traffic or creating a hazard - places where a person would intervene!

 

[willow_hiller]

Obviously of sufficiently short length this is possible. Hence the requested 30-minute interval. I concede it might be possible, on trivial streets, no traffic, good markings, doing a carefully planned route where failure cases have been pre-screened and avoided, etc. Haven’t seen it though!

I don't know about where you guys live, but personally I would find it hard to find a route that takes 30 minutes and avoids freeways/highways. And any location where you're forced to drive that long without being able to take a freeway or highway probably has poorly designed roads that would negatively impact FSDb's performance.

Just testing a hypothetical route in Google Maps from one corner of DC to the other, and it always routes to the 395 freeway unless I check "Avoid highways," which I don't think is an option in the Tesla nav. Might be possible by forcing it onto city streets via waypoints.

 

[AlanSubie4Life]

I don't know about where you guys live, but personally I would find it hard to find a route that takes 30 minutes and avoids freeways/highways.

Sure, I don’t typically if I am headed somewhere, but that is neither here nor there. It’s not actually hard to do - you just have to avoid the freeways.

 

[Dewg]

It's super easy, barely an inconvenience to satisfy Alan's request. Find a long road with no intersections, no turns, no stops and no curves. The car can usually go straight pretty well, according to him.

 

[nvx1977]

You're going off the rails, @AlanSubie4Life. Your original postulation is that the car behaves more or less the same for everyone (very imperfectly), and the differences in perception between people here are due mainly to tolerances (and maybe some geographical variances). So assuming this, why are you looking for a 30-minute perfect drive? Because some people claim to have zero-intervention drives?

I started to get my first intervention-free drives around 10.10. Since then, it's gotten better. Sure it still F's up hard in places, and I have to disengage. But there are people here who describe FSDb as utterly unusable and useless, and to me that perception is so far from mine that I seriously doubt it's just tolerance.

It's pretty clear from your posts that you have a very, very low tolerance for FSDb deviating from your mental model of an objective ideal. And I get that; I would love for my car to drive the way you want it to drive. But I don't think you can really explain how I find lots of utility with zero-intervention drives compared with the people who find the system so horrible that they deem it useless.

To me, exurban/suburban/urban is the biggest determination of experience. I know because I can barely use FSDb when I'm within Boston city limit. It's a totally different experience between where I live and where I have to commute to. Same car, same firmware, same tolerance.

 

[AlanSubie4Life]

It's super easy, barely an inconvenience to satisfy Alan's request. Find a long road with no intersections, no turns, no stops and no curves. The car can usually go straight pretty well, according to him.

This would likely work but not sure what it would demonstrate!

Anyway the lack of videos of that length kind of demonstrates the point.

In any case, it is surprisingly capable now, and the latest version really is quite good in many cases. There are just quite a few easy maneuvers it cannot be trusted to do, like ULTs as seen.

 

[EVNow]

It is well established that Whole Mars “zero-intervention” videos have occasional interventions because he accomplishes them by allowing the car to do things extraordinarily slowly, or incorrectly, as illustrated above.

Just like Cruise !

 

[AlanSubie4Life]

So assuming this, why are you looking for a 30-minute perfect drive? Because some people claim to have zero-intervention drives?

There were claims about zero-intervention drives, so just wanted to find an example of one. It’s not really for any reason other than that.

I agree it depends on the scenario.

Anyway there seemed to be confusion about FSD “varying capability” and it really is very consistent from user to user. Everyone has essentially the same experience in similar scenarios.

That was really the point of the discussion. There are not any mythical “spectacular” intervention-free experiences here caused by extraordinary cars.

FSD is quite capable at many maneuvers, but it is not that good! Not yet.

It's pretty clear from your posts that you have a very, very low tolerance for FSDb deviating from your mental model of an objective ideal.

Not really. Just looking for normal driving that won’t frustrate other drivers (dangerous!). This is quite clearly distinct from preferences about stopping acceleration profiles, which is a comfort issue.

We are not talking about my propensity to disengage at every stop or not use FSD at all. There is annoyance, and then there is the question of it can complete maneuvers to pass a driving test without accelerator or steering or scroll wheel adjustment.

 

[nvx1977]

There were claims about zero-intervention drives, so just wanted to find an example of one. It’s not really for any reason other than that.

I agree it depends on the scenario.

Anyway there seemed to be confusion about FSD “varying capability” and it really is very consistent from user to user. Everyone has essentially the same experience in similar scenarios.

That was really the point of the discussion. There are not any mythical “spectacular” intervention-free experiences here caused by extraordinary cars.

FSD is quite capable at many maneuvers, but it is not that good! Not yet.

Not really. Just looking for normal driving that won’t frustrate other drivers (dangerous!). This is quite clearly distinct from preferences about stopping acceleration profiles, which is a comfort issue.

We are not talking about my propensity to disengage at every stop or not use FSD at all. There is annoyance, and then there is the question of it can complete maneuvers to pass a driving test without accelerator or steering or scroll wheel adjustment.

sometimes I read your posts like this one, and I just can't figure out really what your point is. Here you explicitly say:

"Anyway there seemed to be confusion about FSD “varying capability” and it really is very consistent from user to user. Everyone has essentially the same experience in similar scenarios. That was really the point of the discussion."

and yet it was not clear you were trying to say that in your numerous previous posts. So when I try to address something you said, it feels like I'm the one moving the goalposts. It's a really weird way to communicate. Haven't quite figured it out. Sometimes it seems like we're exactly on the same page, and sometimes we're just talking past each other.

so in this case, I THINK we're on the same page. Because if you're saying "Everyone has essentially the same experience in similar scenarios," then that means that the best way to explain the huge discrepancy in experiences (useful vs useless) comes down to what scenarios we all encounter. If they are very different, then the experience varies widely. This is why I think rural/suburban/urban is the biggest factor in the discrepancy.

 

[Supcom]

I think it is extremely objective and verifiable. Do we really need take the time to define something when we “know it when we see it?”

In any case I hope we can agree that the car failing to take a trivial mapped right turn (8:50) would count as an intervention. The car did great in the first 9+ minutes of this video - overall impressive (minimal traffic of course which does make it easier). But I think most would not have the patience for the stops. It’s really the sort of thing you have to be in the car for to understand how slow they are - it’s weird how it is not more obvious when watching, actually.

Let me know when someone can point to an actual intervention-free example of significant length. It will happen eventually!

If more than one person is going to have the same "know it when we see it" result, then yes, it must be defined. That is, except for binary cases, like failing to take a turn. But, when you get into, what you feel is unacceptable braking or acceleration performance, yes, the performance should be defined. Otherwise, there will be more multipage arguments over how many intervention a drive should have had.

BTW, I don't know what constitutes a "trivial" mapped right turn, vs a non-trivial one. Nonetheless, in that case, simply failure to make a required turn would be objective and sufficient. After all, failure to make a non-trivial turn is still a failure. No need to qualify it with ambiguous terms.

 

[EVNow]

If more than one person is going to have the same "know it when we see it" result, then yes, it must be defined. That is, except for binary cases, like failing to take a turn. But, when you get into, what you feel is unacceptable braking or acceleration performance, yes, the performance should be defined. Otherwise, there will be more multipage arguments over how many intervention a drive should have had.

BTW, I don't know what constitutes a "trivial" mapped right turn, vs a non-trivial one. Nonetheless, in that case, simply failure to make a required turn would be objective and sufficient. After all, failure to make a non-trivial turn is still a failure. No need to qualify it with ambiguous terms.

The problem is how do you define “failure”.

Yes, in some cases it is obvious- like if FSDB gives up and shows red wheels. In other cases, how long of a wait is a failure, how jittery is a failure, how close to other vehicles is a failure ? @AlanSubie4Life has a lot of failures because he defines it as anything different than he would personally do. I look at it as anything even my wife won’t do

For eg. on Friday - in one of the right turns, FSD did about 20% of the turn and stopped - because someone was making a ULT from the opposite direction. I just disengaged and drove ahead of the other vehicle. I think it would have been perfectly safe to wait there and let the other car go first.

The reason people have such varying experiences is that they have different thresholds for waiting, how long they will let FSD do it’s thing before disengaging etc. Ofcourse there is wide variation between cities, routes and traffic conditions.

 

[JulienW]

The problem is how do you define “failure”.......

Um....well......saying you are going to release V11 in Jan of 2022 could qualify.

 

[AlanSubie4Life]

@AlanSubie4Life has a lot of failures because he defines it as anything different than he would personally do.

It mystifies me that anyone could get this impression.

I think it is very clear from my posts what the distinctions are here! It’s not complicated!

sometimes I read your posts like this one, and I just can't figure out really what your point is.

Sorry. My point is that FSD needs a lot of improvements in v11 to get to a point where it is a useful feature Yu that can be used with passengers. Since you have to pay attention, it has to be extremely good (a lot better than it needs to be if you don’t have to pay attention!).

Hopefully v11 won’t regress too much. Seems like a huge task, with what has been claimed thus far.

 

[powertoold]

Some V11 fails starting here:

 

[AlanSubie4Life]

Some V11 fails starting here:

This seemed fine. It’s like the approach to stop lights in v10.

I guess they haven’t added cut-in logic yet! They’re probably working on figuring out how to stop still, would be my guess. Definitely a very hard problem.

(And if course cut-in logic is not good enough - it has to know who is going to be changing lanes before there is any hint of that happening (a relatively easy problem I guess).)

 

[Supcom]

The problem is how do you define “failure”.

Yes, in some cases it is obvious- like if FSDB gives up and shows red wheels. In other cases, how long of a wait is a failure, how jittery is a failure, how close to other vehicles is a failure ? @AlanSubie4Life has a lot of failures because he defines it as anything different than he would personally do. I look at it as anything even my wife won’t do

For eg. on Friday - in one of the right turns, FSD did about 20% of the turn and stopped - because someone was making a ULT from the opposite direction. I just disengaged and drove ahead of the other vehicle. I think it would have been perfectly safe to wait there and let the other car go first.

The reason people have such varying experiences is that they have different thresholds for waiting, how long they will let FSD do it’s thing before disengaging etc. Ofcourse there is wide variation between cities, routes and traffic conditions.

Yes, it gets pretty messy when we try to evaluate driving without a good set of objective standards and measured data. This is one of the challenges that Tesla has in training NNs. Unlike us, they cannot use ambiguous standards. They need to define such things as acceleration limits, jerk limits, distances between other cars ahead, behind and to each side. They must define acceptable wait times, minimum gap distance for turns, etc.

 

[AlanSubie4Life]

Yes, it gets pretty messy when we try to evaluate driving without a good set of objective standards and measured data. This is one of the challenges that Tesla has in training NNs. Unlike us, they cannot use ambiguous standards. They need to define such things as acceleration limits, jerk limits, distances between other cars ahead, behind and to each side. They must define acceptable wait times, minimum gap distance for turns, etc.

Yes, it is very odd the stuff they do get right…and then somehow it is interspersed with a lot of undesired behavior in their framework. Somehow they have not mastered how a human drives. It is harder than it seems, I guess.

 

[powertoold]

Yes, it is very odd the stuff they do get right…and then somehow it is interspersed with a lot of undesired behavior in their framework. Somehow they have not mastered how a human drives. It is harder than it seems, I guess.

I've been in many many cars where the human drives a lot choppier than FSD, especially when I was in high school and college.

Heck, many manual transmission cars drive choppy in general. Although, there aren't many of those anymore.

 

[AlanSubie4Life]

I've been in many many cars where the human drives a lot choppier than FSD, especially when I was in high school and college.

Heck, many manual transmission cars drive choppy in general. Although, there aren't many of those anymore.

There is definitely no need to apologize for FSD! It’s important when it happens, not just what happens.

Lots of issues. Same issues much less of an issue for an L3+ system of course.