FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Tronguy]

actually I find the reverse - if you can see one car clearly stop as evidenced but he slight backward lurch it makes it more clear that that car should proceed first. I will often intentionally fully stop sooner or later so the other car can tell which order to proceed in.

4 way stop signs are actually a very difficult problem for FSD since there are a lot of unspoken visual cues between drivers. I would also imagine that these cues vary somewhat from region to region.

Once upon a time in Newton Centre, Massachusetts, there was this intersection (still there) between Center and Beacon streets, both major (mostly) two-lane thoroughfares through the center of the city. As one might imagine, left turn lights, left turn only lanes, two lanes on Center going to one on the far side, and all that jazz. I'm sure we all have seen worse and perhaps better. On an average 3 p.m. to 7 p.m. period one would wait an average of a minute or three to get through, whichever direction one was traveling, because there was No Way one was going to arrive at this intersection on a green. Possible, but statistically impossible.

One Fine Day something in the control system for the lights went on the fritz and all the lights, in every direction, started blinking red. In a place like Boston or maybe Brooklyn some traffic cop would show up and start doing hand motions at people. No such luck; the Newton cops were oblivious.

And... traffic picked up and moved through that intersection. Blinking red, in MA, means the same as a Stop Sign: Come to a full halt, check out everybody else, and go.

And, I swear, everybody entering that intersection was making serious eye contact with everybody else entering the intersection. Head motions, eyeblinks, the occasional hand popping up off the steering wheel and making motions, all that non-verbal stuff. No accidents, nobody doing something stupid (a miracle!), and traffic just flew through the place. I went through.. and was tempted to go find somewhere and park, just so I could come back and watch the show.

I mean, the traffic patterns were new, different, and in response to a fault, so everybody in creation was Paying Attention for once; I wouldn't trust Lack of Lights on any kind of a long term basis, there'll be a not-paying-attention, bored idiot with an emergency in the cockpit (crying child, spilt milkshake, ringing phone) along sooner than later.

And every time I think hard about autonomous vehicles negotiating something like that: Nope. Not until the AI gets good enough to pick up on human non-verbal communications, which it's not there yet upon. Level 5 autonomy? Um. Maybe if one waits a quarter hour for a break, with people honking all over? Just.. nope.

 

[powertoold]

At that point the Tesla will be well into the intersection aggressively moving forward, and potentially stopping.

Why would a safety-based FSD system be aggressively moving into the intersection when the other car is already creeping forward without stopping to 0mph?

 

[AlanSubie4Life]

The real issue is that in SF (and other dense areas), in the situations I talked about, no one assumes the other car will completely stop (because less than 5% of people do that, I'd say it's around 2% lol).

Would need a video. I have never experienced this in the city. The only issue I have ever encountered is not knowing who was first, and at that point you are rolling forward.

 

[powertoold]

Would need a video. I have never experienced this in the city. The only issue I have ever encountered is not knowing who was first, and at that point you are rolling forward.

The proof is that people will honk at you for stopping to 0mph when it's "clearly" your turn to go. Dirty Tesla and aidrvr commented on behind cars being annoyed at the 0mph rule.

People will honk at you for 1-2 second delays. A car that completely stops at 0mph for 1 second will get honked at often by the behind cars. If that's the case, the other cars at a 4-way stop will also display a similar expectation of not stopping to 0mph.

Anyway, when you get 11.3.2, see how it does at dense stop signs.

 

[AlanSubie4Life]

People will honk at you for 1-2 second delays.

Yes they will. That is a very excessively long stop if you should be going!

Dirty Tesla and aidrvr commented on behind cars being annoyed at the 0mph rule.

They’re misassigning the issue. The issue is the extremely slow approach and ponderous proceeding. Not the stop. That is what annoys people.

 

[JulienW]

Chuck's UPLs

 

[dachshund]

Exactly. There are online videos of robotaxis acting like timid deer in the headlights so surrounding traffic pick them off with little regard. Robotaxis will need to be safe and as aggressive as a normal driver to gain respect otherwise customers will be pushing red buttons to exit robotaxis rides in the middle of busy traffic.

Ah... memories of owning a 2006 Prius. My next car was a Mini Cooper because I was so tired of people cutting me off. "Prius!! Quick - get in front!"

Elon keeps talking about the robotaxi. FSD is designed for his robotaxi, and not for people who drive. Literally: the idea is to talk to your kids, play with your phone, and ignore traffic. I believe this is the goal. Therefore it has to be extremely safe. In this scenario, no one cares if 10 cars cut in front of you. This what people are testing in FSDb. Not a system that will please drivers, especially anyone in a hurry.

Yes, it's designed to be a timid deer.

 

[Dewg]

Exactly. There are online videos of robotaxis acting like timid deer in the headlights so surrounding traffic pick them off with little regard. Robotaxis will need to be safe and as aggressive as a normal driver to gain respect otherwise customers will be pushing red buttons to exit robotaxis rides in the middle of busy traffic.

I'm afraid you may be wrong on this. AVs are damned if they do and damned if they don't. If they are too timid, they will get complaints from people who are in a hurry and don't want to wait for an AV. claiming they are blocking or hindering traffic. If they are aggressive, they will get complaints from people who find their behavior unsafe and yell about an AV "cutting them off", demanding they are removed from the road as safety hazards.

You can please some of the people some of the time...

 

[diplomat33]

FSD beta 11 fails to merge unto busy highway.

https://twitter.com/x/status/1634159666111426560

It looked like there was a gap early on that FSD Beta could have merged into if it had been more assertive but it waited too long. This is a scenario that FSD beta still needs to work on because it seems to wait too long and misses its chance to merge.

 

[powertoold]

It looked like there was a gap early on that FSD Beta could have merged into if it had been more assertive but it waited too long. This is a scenario that FSD beta still needs to work on because it seems to wait too long and misses its chance to merge.

I think it was mostly due to poor planning for 18-wheeler trucks on merge. This has been an issue since forever.

 

[MP3Mike]

The proof is that people will honk at you for stopping to 0mph when it's "clearly" your turn to go.

Technically it isn't your turn to go until after you have come to a complete stop.

 

[AlanSubie4Life]

Chuck's UPLs

Ouch. Either 2/7 or 1/7 depending on how you count. Roughly. Some dangerous maneuvers too - no way should the car go when someone is turning left!!!

All sorts of problems, same issues as before. It is doing about the same, but the sliders have clearly been adjusted. There were not very many challenging 10.69.25.x tests, if any.

This was testing under difficult conditions and the results were predictable.

 

[EVNow]

Chuck's UPLs

So 2 failures out of ... I lost count.

 

[AlanSubie4Life]

So 2 failures out of ... I lost count.

Two successes. Arguably just one - the first one. The second success was questionable due to an excessively long wait when traffic was clear. That could cause problems with trailing traffic so you must go.

 

[EVNow]

Two successes. Arguably just one - the first one. The second success was questionable due to an excessively long wait when traffic was clear. That could cause problems with trailing traffic so you must go.

I just go by what Chuck says - he is in the driver's seat !

 

[AlanSubie4Life]

I just go by what Chuck says - he is in the driver's seat !

He intervened three times. At least. Talked about how the car should be getting out of the way of other traffic, at other times.

 

[kabin]

I'm afraid you may be wrong on this. AVs are damned if they do and damned if they don't. If they are too timid, they will get complaints from people who are in a hurry and don't want to wait for an AV. claiming they are blocking or hindering traffic. If they are aggressive, they will get complaints from people who find their behavior unsafe and yell about an AV "cutting them off", demanding they are removed from the road as safety hazards.

You can please some of the people some of the time...

Yes, assertive is probably a better way to say it. Aggressive is more likely to be avoided.

 

[EVNow]

He intervened three times. At least. Talked about how the car should be getting out of the way of other traffic, at other times.

Getting out of the way ... not failures. Most humans don't get out of the way (too busy looking at the phones at stops) !

 

[kabin]

Chuck's UPLs

Sounds like it isn't turning right as much before turning left which is good for all the other scenarios where UPL logic creeps in.

 

[AlanSubie4Life]

Sounds like it isn't turning right as much before turning left which is good for all the other scenarios where UPL logic creeps in.

Bad for me though since this bizarre rightward turn before turning left is currently the only thing which keeps the car from completely cutting off left turning traffic for my ULT which has no marked double yellow center line. It is bad enough as is! Usually why I intervene; barely get any chances to try it as a result.

Someday they will get positioning right! Maybe.