FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[uscbucsfan]

I wonder whether S/X need more clearance to move i.e. they will stop when a car comes from the other side more so than 3/Y which are smaller.

In your experience - are there specific ways / scenarios where S is better?

More confident in turns, stays centered more, less braking incidents.
There's a red light with a crooked stop sign beside it for a turning lane and the MY always thinks that's the stop sign to follow and tries to go after a clearing, but the MS has never once done this. This was resolved on my MY recently and I assume it's the map updates we get with every trip.

 

[fasteddie7]

I live in a rural area with tons of cris-cross roads and it‘s having quite the time trying to figure it out. I send reports several times a trip. It also does this weird stop sign thing where it jets forward and brakes hard then stutters forward while making a left or right at a four or five way intersection before committing to the turn. I think I might look into one of those magnetic self driving signs for my trunk to maybe infuriate people behind me less. does anyone use anything like that so others are more tolerant or just get away from us?

 

[kabin]

I live in a rural area with tons of cris-cross roads and it‘s having quite the time trying to figure it out. I send reports several times a trip. It also does this weird stop sign thing where it jets forward and brakes hard then stutters forward while making a left or right at a four or five way intersection before committing to the turn. I think I might look into one of those magnetic self driving signs for my trunk to maybe infuriate people behind me less. does anyone use anything like that so others are more tolerant or just get away from us?

There's vendors on Amazon that will build small magnetic signs to your specs. Reasonable price too. You could say something like "CAUTION: AI Junk SW Test" or "FSD Faking it until it Makes it." The possibilities are endless.

 

[arnolddeleon]

I wonder if others are finding this in their highway drives. Usually when driving FSDj my car stays very true to the map and follows the highway. Yesterday, it followed the car in front of me going off the exit ramp where I did not want to go. Anyone else?

There are times where it seems to be following a completely different route that what is being shown on the navigation screen. I've had confidently do a multi-lane change to a ramp to a different freeway than what was on the screen so it clearly wasn't following another car.

 

[aronth5]

It's not the fender camera that looks left/right, it's the pillar camera behind your door.

My understanding was the fender camera was also used since we know the pillar is pretty much useless on extremely obstructed views. Even if the fender camera isn't used for turns the front camera's range of view is still affected by how the car is facing to the side so the question still applies. HW4 also has a wider fender camera range and is now pretty close to 90 degrees to the side. I thought I saw somewhere that the added view would also help with turns but I could certainly be mistaken.

 

[JulienW]

My understanding was the fender camera was also used since we know the pillar is pretty much useless on extremely obstructed views. Even if the fender camera isn't used for turns the front camera's range of view is still affected by how the car is facing to the side so the question still applies. HW4 also has a wider fender camera range and is now pretty close to 90 degrees to the side. I thought I saw somewhere that the added view would also help with turns but I could certainly be mistaken.

First they are called Repeater cameras and they are primarily for identifying cars in blind spots and lane markings and mimic human looking in side mirrors. The B-Pillar cameras mimic human turning head to look down road. While the HW4 Repeater's have a wider view (likely the B-Pillars also have wider view but we can't see them) they are not really useful in looking forward or to the front sides. In turns Repeater's would be used to mainly to identify the rear wheels in relation to the curb or road edges/line markings and look for VRUs. Here is a view of my car (HW3) pulled up to a 5 lane road at a shopping center. As you can see the Repeaters offer NO help whatsoever in seeing the road, even if the view was MUCH wider. This would all be handled by the Forward cameras and the B-Pillars.

 

[GlmnAlyAirCar]

First they are called Repeater cameras and they are primarily for identifying cars in blind spots and lane markings and mimic human looking in side mirrors. The B-Pillar cameras mimic human turning head to look down road. While the HW4 Repeater's have a wider view (likely the B-Pillars also have wider view but we can't see them) they are not really useful in looking forward or to the front sides. In turns Repeater's would be used to mainly to identify the rear wheels in relation to the curb or road edges/line markings and look for VRUs. Here is a view of my car (HW3) pulled up to a 5 lane road at a shopping center. As you can see the Repeaters offer NO help whatsoever in seeing the road, even if the view was MUCH wider. This would all be handled by the Forward cameras and the B-Pillars.

View attachment 939295

View attachment 939291
View attachment 939290
View attachment 939292

These images mean nothing. These are cropped shots of the full, non-flattened, non-rectangular image. You can’t extrapolate as to what any of the cameras can see based on dashcam video.

 

[JulienW]

Just for kicks here is what Tesla shows on the Model S site. Red is Repeater angle and Green is B- Pillar.

 

[GlmnAlyAirCar]

Just for kicks here is what Tesla shows on the Model S site. Red is Repeater angle and Green is B- Pillar.

View attachment 939300

Of course there is no reason to assume this is technically accurate. It is marketing material

 

[Knightshade]

Of course there is no reason to assume this is technically accurate. It is marketing material

Marketed specs that are not accurate are fraud.

See the various car companies being heavily fined for inaccurate mileage or performance claims in their marketing material.

Also, do you have any evidence the dashcam footage is cropped versus what the driving computer gets? Green had a whole long thread about the differences and that is never cited.

The dashcam footage is more digiitally compressed, but same frame size far as I can tell...so . Several specific A/B pics are shown both ways and there's difference in color and detail, but not field of view.

https://twitter.com/x/status/1199881090917453829

 

[uscbucsfan]

My understanding was the fender camera was also used since we know the pillar is pretty much useless on extremely obstructed views. Even if the fender camera isn't used for turns the front camera's range of view is still affected by how the car is facing to the side so the question still applies. HW4 also has a wider fender camera range and is now pretty close to 90 degrees to the side. I thought I saw somewhere that the added view would also help with turns but I could certainly be mistaken.

The fender cam sees straight back. You can see what it sees by going in reverse.

There are side by side comparisons with HW4 and it's barely larger.

 

[scottf200]

Just for kicks here is what Tesla shows on the Model S site. Red is Repeater angle and Green is B- Pillar.

View attachment 939300

Of course there is no reason to assume this is technically accurate. It is marketing material

Mobileyes SuperVision/Zeekr cameras compares to Undocumented – TeslaTap

Camera POV - Mobileyes SuperVision requirements on their partner Zeekr

Via: Mobileye CES talk (see bottom of this post) A pretty interesting graphic of the camera placement and degrees of visibility (FOV) of Mobileyes SuperVision requirements on their partner Zeekr. - Of note: front camera (2 vs 3), Side Front Camera on mirror vs B-pillar. - "7 long range cameras...

teslamotorsclub.com

 

[n.one.one]

Some hope for this weekend (for a lucky few).

https://twitter.com/x/status/1659721963944964096

 

[JHCCAZ]

I've noticed a few dedicated right turn lanes, like below, 11.3.6 is starting the turn and then coming to a near stop mid turn (red mark shows where the car starts slowing down), even with a green light, and given the lane is dedicated and not merging for some 100+ feet after the turn. The last two versions handled these intersections/turns properly, without slowing down.

View attachment 938178

I have a slightly different situation that I pass through daily near my house. There is a dedicated right turn entry lane that becomes the rightmost of three lanes, so the major road I'm entering is widening there from 2 to 3 lanes. In this case, the dedicated right-turn entry lane is protected and marked by a series of thin breakaway poles, maybe 3 to 4 ft high, that are set into the asphalt along the painted gore. The only reason there are yield signs is because of a pedestrian/bike path; otherwise it's a protected right turn with no yields needed for cars in the high speed lanes

(This seems to be a pretty good approach, visible but safe in case someone hits the barrier. In the 6 months or so since those poles were placed, I've seen at least twice when a few of them were missing, obviously having been knocked away. Both times, the county replaced them within a few days.)​

In any case, FSD acts like it simply doesn't see those thin poles, or doesn't recognize them as a legitimate dedicated turn lane marker. If no traffic is coming in the two high-speed lanes to the left, the car will make a continuous smooth turn and proceed along into the newly-formed third lane. But if any traffic is coming from the left, it wants to stop and wait for a clear gap in those through lanes. In that case I have to goose the pedal to make it proceed through the protected right turn. Exactly the way I would expect it to behave in the absence of the poles and painted gore barrier.

I have no idea what Tesla's detailed map database might contain for this spot, nor what descriptors are available in the system to get it right. It's another example where I would love some individualized editing or at least crowdsourced contributions, possibly curated at the Mothership to filter out mistakes and pranks.

So I guess I would put Dedicated/Protected Right Turn Lanes on the sizable list of features that aren't yet broadly implemenrlted, even if perhaps some common and simple cases are working.

 

[powertoold]

Marketed specs that are not accurate are fraud.

That's not true, the marketed specs are a minimum.

 

[Ramphex]

Just watched Fast X, no one was using FSDj.

 

[WilliamG]

Some hope for this weekend (for a lucky few).

https://twitter.com/x/status/1659721963944964096

I absolutely love that 11.4.2 will supposedly fix my problem with FSD currently.

I’m absolutely aghast it’s even an issue to begin with. How is the most basic of driving scenario broken?

 

[EVNow]

I absolutely love that 11.4.2 will supposedly fix my problem with FSD currently.

I’m absolutely aghast it’s even an issue to begin with. How is the most basic of driving scenario broken?

Narrow unmarked roads is not a basic scenario. It's an edge case.

For eg. XPeng and other competitors won't even let you use their FSD on unmarked roads.

 

[SidetrackedSue]

In any case, FSD acts like it simply doesn't see those thin poles, or doesn't recognize them as a legitimate dedicated turn lane marker.

Our city uses these poles on regular side streets to slow traffic (by narrowing a stroad) and occasionally to reinforce the lines painted for a bike lane.

When using TACC (not FSDb, which I have on my car but don't use), it panics and asks me to take over, or over-dramatically drops my speed below the 25mph (actually 40kph) the car is set at.

 

[MP3Mike]

Narrow unmarked roads is not a basic scenario. It's an edge case.

But it isn't just an issue on narrow unmarked roads. Yesterday I was driving in a different area than normal, and things were going fine, then the road I was on, which had very clear lanes lines, narrowed by a foot, or two, and suddenly the car was a scaredy cat, and slammed on the brakes as every car from the opposing direction got near, even though they were clearly fully in their own lane.

So it is interesting how just a foot or two totally changes how FSDb works.