FSD v12.x (end to end AI)

[Mardak]

There's actually a lot of these gray area decisions, especially in a parking lot. Say there 2 possible paths to a pin in the parking lot but there's a small island in the way. You can turn before the island or after the island, 12.2.1 seems to get tripped up with these decisions.

I have a screenshot of this island issue, but some reason the video has been removed. For this same parking lot intersection, it kept wanting to turn left before the island going the wrong way consistently across a few attempts. But in this case there happened to be a vehicle coming, and the driver pressed the accelerator to force the car forward, and 12.2.1 did decide to go around the island after realizing turning into the cross traffic is the wrong move.

I would think end-to-end control is making decisions on every frame, so that's why the blue path can suddenly change. In this case, it usually wants to go the wrong way, so the decision wobble was actually a benefit allowing it to quickly switch to the correct path based on newer information. Presumably Tesla shadow mode data collection can detect wobble to send back what a driver would normally do to improve 12.3+, but so far 12.x has been "just" trying to drive safe enough.

I think the instability is more of early training cycle with the network unsure about which similar (weak?) strength signal to commit to especially for non-city-street scenarios like parking lots. Back to chess example, initially neural networks play each opening pawn equally, but it still needs to pick one to actually move, so it would seem like decision wobble, and later in training the same size network, it has much higher confidence and consistency in moving certain pawn for its opening.

 

[powertoold]

Back to chess example, initially neural networks play each opening pawn equally, but it still needs to pick one to actually move, so it would seem like decision wobble, and later in training the same size network, it has much higher confidence and consistency in moving certain pawn for its opening.

The chess example is easier because the decision can collapse into whichever has the higher probability since it's essentially a "perfect" environment where all the pieces are known and futures can be placed, but in real life driving situations, there may not be a clear higher probability, but as humans, we just commit to a decision and reap the consequences / benefits. You're right though, the wobble can be reduced with better training, but I think it's also a limitation with the parameters. That's why gpt4 performs better.

This is also related to video curation. It's a challenge to curate videos based on consistent heuristics that account for all the situations in day to day driving. The video dataset needs to be very very homogenous in the driving style and decisions made depending on the environmental, so that's a big task as well.

 

[aronth5]

the more my initial fears about this approach are realized. V11.4.9 was just overall more consistent, reliable, and dependable.

We'll just have to wait and see

Not my experience. V11.4.9 is inconsistent, very unreliable and the only dependability is the lack thereof. HW4 may be the cause of this but regardless my FSD is certainly behaving differently than your experience.

 

[swedge]

A few observations of 12.2.1 driving.

On a twisty road, it stays in the lane better than V11, but it slows down quite a bit to do it. So the basic skill of turning through curves is still weak. The speed limit is 35. V12 drive slows down to below 30, but I can drive it smooth and comfortable at 40. If I use the accelerator to keep V12 speed steady at 35, it veers across the centerline.

12 stills stops in the middle of a left turn in a large stoplight controlled intersection when it sees the red light for the cross traffic. It does not do this if following another car, but alone it stops in the middle of the intersection.

Smart Summon does back out of a parking spot if needed. If v12 does not reverse, this version of SS does not use V12 code. Oh, and for the first time, I actually used SS to come get me in the rain. Still a slow parking lot party trick.

It did swerve safely and nicely around some debris, a rag it appeared. (If the debris looked solid I would have intervened...). I had not seen this in prior releases. I'll have to go out and see if it avoids potholes now. V11 just banged through those.

 

[swedge]

A question for any knowledgeable AI folks here. Elon once described the older version as a whole bunch of NNs as well as 300,000 lines of C code just for lane choosing. (Apply grain of salt.)

If all that is now a single end-to-end neural net, might that take significantly longer to train? How does training time vary with NN size? Is it linear? Does the entire net need to be retrained from scratch for every iteration? Intuition suggests that vastly more iterations might be needed to train a larger and deeper net.

I heard a talk by a Cal Tech AI researcher who is working on medical applications, such as CAT scan reading. He bemoaned the vast cost of computer time to train the systems, costs which researchers find hard to cover. So far only huge tech companies have the budgets to invest, while life saving applications are lagging, even when we know they can do the job. FSD is an example of massive expenditure on an uncertain outcome.

 

[JB47394]

FSD is an example of massive expenditure on an uncertain outcome.

If it saves 40,000 lives a year by eliminating driving accidents, then it qualifies as a pretty big medical breakthrough.

 

[zoomer0056]

phantom drain

Vampire drain.
Phantom braking.

 

[JulienW]

If it saves 40,000 lives a year by eliminating driving accidents, then it qualifies as a pretty big medical breakthrough.

Not in the US. Every car on the road would need to be a Tesla AND using FSD (in the future) and still there would be some fatalities since driving is inherently dangerous. So an impossible but lofty goal.

Unfortunately humans killing humans in cars is just excepted as "the way it is". We can even empathize. However a computer in control when someone dies and that is unacceptable and we can't understand or accept this. Sucks but a HUGE hurdle to get over and will be VERY hard to do.

 

[primedive]

I wish there was another way to let it know you are there because there’s a fine line between apply force to the wheel for nag and disengaging FSD

I would get the blue flash sometimes, apply force to the wheel and it keeps flashing then I would apply more force and disengage it on accident

It would be useful to have some more control over fsd and autopilot without having it disengage immediately. For example, to avoid potholes/debris or keep the car positioned properly.

 

[primedive]

On v11 you can still change music volume or use speed up or down to satisfy the nag. Word is on v12 that's gone.

@AlanSubie4Life has mastered the art of satisfying nags without disengagements, read some of his tips.

Even so, autopilot has complained to me when I just turn it on and I use the scroll wheel to adjust the speed. It still wants me to turn the steering wheel to acknowledge my presence. It’s somewhat heavy-handed with the nags…

 

[Mardak]

in real life driving situations, there may not be a clear higher probability, but as humans, we just commit to a decision and reap the consequences / benefits

I think this is two different aspects. One is related to what is end-to-end predicting for outputs and actions, and it seems to be a path for the next ~2 seconds that reflects acceleration and steering over that time. This is as opposed to an instantaneous prediction of acceleration and steering because the blue path visualization we see with 12.x is able to reflect steering right and then straighter for parking on the side of the road. It seems like the highest probability action wants to be picked and not averaged, which could be dangerous if deciding between going straight and making a 90º right and ending up 45º right. So like you say, end-to-end needs to pick something that is the higher probability, and sometimes perhaps due to insufficient training, it switches between say 45% straight and 45% right turn and each frame one jumps up to say 46% becoming the highest resulting in decision wobble.

The other aspect is that you're suggesting humans tend to not change their mind after picking something, e.g., started a lane change and will complete the lane change. 12.x should be able to do that with more training when appropriate resulting in a prediction with say 90% probability that presumably is high because additional training shows that's basically the one action it should be considering, and any subsequent frames that are similar enough should predict the same way. However, training should also still allow for new information such as a seeing cross traffic to change the prediction. We also see 12.x take into account previous actions when making future actions such as engaging a turn signal allows a future prediction to complete the turn or lane change.

 

[Mardak]

Speaking of committing to a decision or not… Did 12.2.1 take this flashing yellow left because the closer oncoming car decided to stop at a green light making it seem like oncoming traffic had a red?

FSD Beta was about to take the left initially when the white car stopped (not sure why as repeater cam shows it's clear…) but then noticed the adjacent Model Y approaching quickly to decide to stay still. Then as the Y was passing, it committed to completing the left ahead of the white SUV perhaps noticing it was slowing down or at least not going as fast as the Y? The visualization shows the SUV not even entering the intersection by the time FSD Beta completed the turn, but unclear if the SUV had to hit the brakes.

"That was perfect… That was a little scary, I'm not gonna lie. "

 

[lzolman]

It would be useful to have some more control over fsd and autopilot without having it disengage immediately. For example, to avoid potholes/debris or keep the car positioned properly.

At first this sounds great, but then I'm thinking about how the interface might work, and wouldn't it basically be the same as if we just disengaged? Except it wouldn't make a lot of sounds and ask why we disengaged. Maybe just a short beep to acknowledge the press, and another short beep to acknowledge the re-engagement. But aside from the sounds and prompts, would it really be any different than it is already?

 

[Mardak]

At first this sounds great, but then I'm thinking about how the interface might work, and wouldn't it basically be the same as if we just disengaged?

With end-to-end, I could see collaborative driving being more useful where drivers can make adjustments and Autosteer continues. Previously Autopilot overall was very strict on wanting to stay centered in lane, so potentially collaborative would be more of competitive with driver wanting to steer from center and Autopilot wanting to steer back to center.

12.x should be able to realize the current steering (initiated or maintained by the driver) is a reasonable action to do something else, e.g., make a lane change or not crossing double yellow. This saves on fully disengaging then needing to reengage and should be much smoother than current jerky/accidental disengagements.

This would also probably be even better training data as it precisely knows when human vs FSD Beta was controlling steering as the human override effectively ends once it matches what FSD Beta would now want to do to complete the maneuver. Basically this should allow for better planning decision training data.

 

[powertoold]

It seems every 12.x release uses a new model, which takes about 5 weeks to train?

12.1 Dec 24
12.1.1 Jan 12
12.1.2 Jan 21
12.2 Feb 9
12.3 --> ~Mar 20?

 

[Supcom]

It seems every 12.x release uses a new model, which takes about 5 weeks to train?

12.1 Dec 24
12.1.1 Jan 12
12.1.2 Jan 21
12.2 Feb 9
12.3 --> ~Mar 20?

Teslascope was floating the rumor that an update (12.3?) would go out this weekend.

I think that my April prediction for wide release is looking pretty safe.

 

[kabin]

Not in the US. Every car on the road would need to be a Tesla AND using FSD (in the future) and still there would be some fatalities since driving is inherently dangerous. So an impossible but lofty goal.

Unfortunately humans killing humans in cars is just excepted as "the way it is". We can even empathize. However a computer in control when someone dies and that is unacceptable and we can't understand or accept this. Sucks but a HUGE hurdle to get over and will be VERY hard to do.

Agreed. And, on the other hand, with ~400k FSD owners on the road today, FSD might require 4M interventions annually just to avoid accidents. The team has some wood to chop.

 

[Yelobird]

Teslascope was floating the rumor that an update (12.3?) would go out this weekend.

I think that my April prediction for wide release is looking pretty safe.

Many seem to be in a rush to “me to” get the new version only to complain about what it can’t do the next day. There are 10,000 plus pages of hurry to complain posts. I personally say let them monkey around with it as long as they need to give us a moderately stable foundation to start with. Then we can complain about the non chauffeur driving style etc from that point.

 

[Mardak]

It seems every 12.x release uses a new model, which takes about 5 weeks to train?

Yeah, that cadence so far for initial 12.x(.0) matches up even including the original 12(.0.0) released in November. Hopefully after 12.x reaches a stable enough safety baseline for wide release, continuous training and releases will be at a more frequent rate to incrementally fix up issues like wobbles as they appear in the wild.

 

[rlsd]

I drove my car with thru the following neighborhoods in the last two days: Pacific Beach, La Jolla, Poway, Kearny Mesa, Solana Beach, and Leucadia in San Diego.

v12.2.1 performed very well in areas having narrow streets and many pedestrians.
Auto wipers worked OK in both light and heavy rain. Not perfect but safe for driving.

I had 3 disengagements in Leucadia (tourist destination) Saturday evening.

1. FSD tried to squeeze the car to make right turn on a narrow street when there was another car waiting at the intersection. The gap between my car and that car was less than 1 ft (according to my eye. It could be wider if it's actually measured). It's possible FSD could make turn but it could be a disaster. Disengaged.

2. FSD attempted to make left turn while there was another car moving thru intersection from the opposite direction on green light (my car was already in the middle of the intersection). Disengaged.

3. FSD crossed the railroad and stopped about 6ft from the railroad after crossing because the leading car was stopping. It should have waited behind the railroad.

And 1 disengagement in Kearny Mesa. The car was going very slow for right turn at the intersection corner when yellow light just turned to red and 5, 6 people were standing close to the curb at the intersection. I was not sure whether the car would stop or go. I pressed the accelerator to make the turn. I was not sure I did the right thing or not.