FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[TK211X]

Yea I’ve had FSD complete a U turn as well multiple times. It never does it confidently and freaks out about what lane it’ll settle on.

 

[jeewee3000]

Question from someone that has no access to FSDbeta (Europe), since you're talking about U-turns:

Does FSDbeta ever reverse? (i.e. three-point-turn) That's what I do when I can't make a U-turn in one go.

 

[LowlyOilBurner]

Question from someone that has no access to FSDbeta (Europe), since you're talking about U-turns:

Does FSDbeta ever reverse? (i.e. three-point-turn) That's what I do when I can't make a U-turn in one go.

It does not reverse.

 

[syncword]

I'm not currently subscribed to FSDb, but a couple months ago when I was - with 11.4.4, it definitely did a rather nice U-turn, following nav. No other traffic around. I had read here it wouldn't, then decided to see for myself how it would fail - and I could have done no better job. N. Courtenay pkwy S. of Kennedy Space Ctr. with crossover spots every mile or so, 55 mph speed limit in that section.

 

[rlsd]

I am sure FSD can make U turn on Texas freeways.

Texas U-turn - Wikipedia

en.m.wikipedia.org

 

[sleepydoc]

I am sure FSD can make U turn on Texas freeways.

Texas U-turn - Wikipedia

en.m.wikipedia.org

True, but that's not really a U turn

 

[FSDtester#1]

Chuck Cook keeps trying to have FSD make a U-turn, but it hasn't happened yet.

I have had it do it 1 time by itself about 6 months ago.
Tried it several times at the same place since, and it never worked again

 

[cbdream99]

I am sure FSD can make U turn on Texas freeways.

Texas U-turn - Wikipedia

en.m.wikipedia.org

Yes, but this is relatively easy, even a caveman can do it, LOL. After existing the highway, stay on the left most lane and then make the U turn to the opposite side, it is usually well protected, just be careful when exiting on the other side as there may be traffic coming. FSD can handle it. The U turn in the city is a little different, some junction actually is illegal and may be too tight.

 

[rlsd]

I wish California has that kind of U turn.

 

[jebinc]

I have had it do it 1 time by itself about 6 months ago.
Tried it several times at the same place since, and it never worked again

Repeatability, dependability and reliability. Yep, not with this product.

 

[Dewg]

Repeatability, dependability and reliability. Yep, not with this product.

Try to remember that this is a neural net, which is the attempt to simulate a biological brain. It's not a hard-coded system (IF, THEN, ELSE). So repeatability is almost guaranteed to never happen. If I ask you to walk a straight line, and precisely measure your footfalls and their exact locations - then asked you to walk that exact same line again, you wouldn't be able to walk the exact same way. There would be variations to your stride, location of your footfalls, etc. Same with the car - if you drove the same street over and over, the car wouldn't behave the exact same each time. Lighting, traffic, weather, pedestrians, traffic signal timing, etc. all create a different experience every time you drive that same street. The NN will react slightly differently each time depending on the visual input.

Of course you'll be generally close, but not exactly the same. The car might stay in the same lane every time, and will stay the same speed limit depending on the signs, but you'll see variations in how it accelerates, when it decelerates for a stop sign, how to reacts to cut-ins and cut-outs, etc.

An example, referencing U-Turns: How many humans come to a left turn and *think* they can make a U-Turn in one maneuver, but upon executing the U-Turn find they didn't have enough room and have to stop and make it a 3-point turn? It happens pretty frequently. That same human, given the same turn again may barely make it the 2nd time because they started the U-Turn sooner, or they turned the wheel faster. Then the 3rd time they try they end up with a 3-point turn again.

 

[aronth5]

Just another example of of the B-pillar camera and Telsa's inability to locate cameras correctly. What part of first principles did they screw up when deciding how to address obstructions?
I had to disengage quickly to avoid a car since FSD obviously didn't see the car that I did.

Picture 1- B-Pillar Camera obstructed view

Picture 2- Driver view leaning forward

 

[old pilot]

I drove my MY using FSD to Nashville Tn. on Sunday. The trip took about 40 minutes, the traffic was light because it was Sunday evening and snow was forecast for later that evening. I had a passenger with me and ask for his opinion after the drive. He had no complaints and felt very comfortable with the drive as did I. Now I want to emphasize this was in light traffic but still a very good drive. On the way home the snow had come early and heavy so I drove. I was very pleased to find that on slippery roads the regen. braking works just as normal anti-lock braking, that is, when a wheel starts to slide that wheel releases (or stops regenerating). When I learned to drive in Pennsylvania in the nineteen sixties in the winter in the snow on steep hills it was in an old Plymouth with three on the tree and non-anti-lock drum brakes, things have come a long way since then.

 

[mgs333]

Couple of recent failures:
1. On a freeway, after recently merging, the car wanted to change lanes to the left to go around a slow moving truck, there was a faster car in the left lane coming from behind, the car continued to go even though we would have collided (and the red blind spot warning was on), I disengaged as the other car honked.
2. On the same area of the freeway, while trying to merge onto the freeway, there was a truck to my left, the car just kept matching the pace of the truck for a few seconds (going about 50), had to disengage and punch the throttle to merge in front of the truck..

On a positive note, the autowipers are still working well in the rain today.

 

[LowlyOilBurner]

I drove my MY using FSD to Nashville Tn. on Sunday. The trip took about 40 minutes, the traffic was light because it was Sunday evening and snow was forecast for later that evening. I had a passenger with me and ask for his opinion after the drive. He had no complaints and felt very comfortable with the drive as did I. Now I want to emphasize this was in light traffic but still a very good drive. On the way home the snow had come early and heavy so I drove. I was very pleased to find that on slippery roads the regen. braking works just as normal anti-lock braking, that is, when a wheel starts to slide that wheel releases (or stops regenerating). When I learned to drive in Pennsylvania in the nineteen sixties in the winter in the snow on steep hills it was in an old Plymouth with three on the tree and non-anti-lock drum brakes, things have come a long way since then.

A good friend of mine is a used car salesman. He had a 2011? Jeep Wrangler with three pedals and a five speed over the summer for a couple weeks. I drove it for a bit, brought back memories of cool cars I drove as a young buck (MK3 & 4 Supra’s, E24 M6, E28 M5, Ur-Quattro (numerous old manual Audi’s, here’s looking at you Audi 5000 CS Turbo Quattro, V8 Quattro w/5speed, 200 Turbo Quattro (10 and 20valve), Ur-S4, I owned a lot of those old five cylinder Audi’s). Now a MK4 Supra costs as much as a new Model S. Sigh.

 

[Todd Burch]

Last night while driving through my neighborhood a cat ran across the street about 100 feet in front of my car.

11.4.9 recognized it and slowed down. It drew some sort of 3D model on the display, but it may have just been a general “small animal” model. It was definitely shown smaller than the deer model.

 

[Todd Burch]

Just another example of of the B-pillar camera and Telsa's inability to locate cameras correctly. What part of first principles did they screw up when deciding how to address obstructions?
I had to disengage quickly to avoid a car since FSD obviously didn't see the car that I did.

Picture 1- B-Pillar Camera obstructed view

View attachment 1010243
Picture 2- Driver view leaning forward

View attachment 1010240

I don’t think an obstructed view is the problem. My HW4 Model X repeatedly attempts to pull out into a street with cross traffic even when there are no obstructions at all. I think it’s a v11/software behavior problem, not a camera placement problem.

If the car properly and slowly creeps forward at that intersection, the maneuver is possible. The problem now is that after stopping, it either creeps forward aggressively (giving me and other drivers the appearance that it’s going to go), or actually pulls out without even checking.

Both software issues. That’s my take on it.

 

[kabin]

I don’t think an obstructed view is the problem. My HW4 Model X repeatedly attempts to pull out into a street with cross traffic even when there are no obstructions at all. I think it’s a v11/software behavior problem, not a camera placement problem.

If the car properly and slowly creeps forward at that intersection, the maneuver is possible. The problem now is that after stopping, it either creeps forward aggressively (giving me and other drivers the appearance that it’s going to go), or actually pulls out without even checking.

Both software issues. That’s my take on it.

No doubt there's more than one intersection failure mode and oodles of software/NN design snafus. Plus, if it can't see oncoming traffic it's doomed from the start. So much to fix. It's looking more and more like HW3/HW4 FSD will remain an overpriced novelty.

 

[jabloomf1230]

I don’t think an obstructed view is the problem. My HW4 Model X repeatedly attempts to pull out into a street with cross traffic even when there are no obstructions at all. I think it’s a v11/software behavior problem, not a camera placement problem.

If the car properly and slowly creeps forward at that intersection, the maneuver is possible. The problem now is that after stopping, it either creeps forward aggressively (giving me and other drivers the appearance that it’s going to go), or actually pulls out without even checking.

Both software issues. That’s my take on it.

This unsafe behavior is reported so often that it makes me wonder if it is either car-specific or model-specific. Some people report this behavior constantly and others deny ever experiencing it.

 

[hybridbear]

This unsafe behavior is reported so often that it makes me wonder if it is either car-specific or model-specific. Some people report this behavior constantly and others deny ever experiencing it.

I’ve wondered that too.