FSD v12.x (end to end AI)

[RNHurt]

Ok, so who has given v12 the real test - navigating a Chick-fil-A parking lot at lunchtime? I've always said that FSD will be complete once it can take me to get a chicken sandwich and some waffle fries at noon on a Tuesday.

 

[Buckminster]

With a bit more driving on the way home today:

1) Really quite good fundamentals. Some cool, smooth maneuvers around bicycles in the bicycle lane on a right turn, competent maneuvering to get off of high-speed streets to small driveways, etc.
2) Response time seems to be better, though that requires more examples and testing.
3) The fact that it signals fairly reliably is good. Can't believe it took NNs to solve this vexing problem. But definitely an improvement.
4) Some cool, casual crossing of double yellow line on turns, to cut off the corner - just like a human would. I wouldn't do it, but it seemed natural without any traffic around, and not really an issue.
5) Just feels more natural - drifts around the lanes, isn't rigidly locked to the center, and generally drives where a human would in the lane (not always correct of course - it definitely makes mistakes here - I'm just saying it feels better most of the time).

Plenty of warts, though, as mentioned:
1) Stopping is a disaster. It does stop, and most of the time it doesn't use the brakes (sometimes it does!), but it's premature to a fault. And then it slowly rolls up to the stop line. And then stops five feet before it. This is for traffic lights, stop signs, whatever. It doesn't matter. And dealing with stop signs adds an extra layer of excrement on the whole problem since it hesitates, waits, and generally is completely confused about what to do. (This is with absolutely zero traffic - maybe it does better with traffic though I doubt it.)
2) Speed control is a disaster. Probably the number one problem, since it is basically a constant problem that has to always be addressed, unless there is lead traffic that is going at a pace where it can keep up.

Fix these two problems robustly and it'll be quite good as a driver's aid. I'm not sure what utility that will have, but it's better than having a driver's aid that sucks. The main utility of a driver's aid is one which can go on a straight street with traffic lights and no turns for miles, without intervention. Anything more complex is not required, really. And once these problems are solved it'll be able to do that.

What's unclear is whether they are solvable.

I do think that I'll lose the next bet on Chuck's turn. I'm not sure about the follow-on 12.3 test with traffic though. I think that will be tough for this version. But sooner or later @Daniel in SD will extract a beer. He's pretty far in the hole, so it's about time.

It can't deal with flashing red disabled traffic lights in construction zones. It just stops and stays stopped. Nice smooth movement to merge away from closed lanes (with cones and construction trailers with the flashing arrows) though. (All on Miramar Road which is putting in the poop pipes still.)

Wow - this coming from you is great to hear. Have you done this?

https://twitter.com/x/status/1769945217761960392

 

[Mardak]

Minnesota's still at zero!

The earlier wave today to TeslaFi ~750 seems to have been HW4 mostly Arizona, California, Texas. The current rollout increasing to ~860 seems to be focused on HW4 as well but newly adding Minnesota, Colorado, Ohio so far. Interesting rollout behavior now that it seems like both HW3 and HW4 are validated to start rolling out to more regions.

 

[sleepydoc]

True, but she loved the 8 mile per gallon SUV I had before the MS.
Funny, all my life I was taught women are smarter than men....

Was it a woman who thought you that?

 

[AlanSubie4Life]

Have you done this?

Whole Mars, of course, is spewing nonsense (I guess if he means it will solve excessive speed he is completely right though, lol). Autospeed is what I'm using. I've tried Chill, Assertive, whatever. Doesn't matter. It's thoroughly broken. It just doesn't work. Constant accelerator required if you don't want to be traveling under the speed limit.

Press the accelerator to add extra speed, it's kind of sticky, stays briefly where you left it, and then it just decays down over the course of 10s of seconds. And then you have to press the accelerator again. It's completely stupid and very, very broken indeed.

My use case, where this is primarily observed, is nearly exclusively multi-lane California surface streets with 45-55mph limits. I also noted it turning into a neighborhood. It picked 15mph. Just broken.

If you don't mind driving like a typical Tesla driver (so infuriating - you have 300-500 ft-lbs of torque at the motor at your disposal!), then it's fine, I guess.

The auto-speed stuff definitely seems fixable. The stopping, I'm not sure. Improving their reaction time (it might well have improved from glacially slow v11 levels, as I said!) would help a lot with this, of course.

 

[FSDtester#1]

Wow - this coming from you is great to hear. Have you done this?

https://twitter.com/x/status/1769945217761960392

Has anyone noticed the up and down arrow near the speed when you use the right scroll?
It only happens once when up or down.

 

[FSDtester#1]

Was it a woman who thought you that?

Yes, and women spell that word: "Taught" ...

 

[fasteddie7]

Saw an Ohio pop in there, cmon Pennsylvania!

 

[AlanSubie4Life]

Has anyone noticed the up and down arrow near the speed when you use the right scroll?
It only happens once when up or down.

Yeah, it's adjusting the set speed in the other mode that you're not using (the old v11 speed offset mode, which is also now broken, so there's no point in using it). It doesn't seem to have any obvious impact on the mode in use.

If you use the up scroll a lot and then switch modes, you'll find you're at 60mph or 85 mph (depending on the type of street). If you scroll down a lot then switch modes you'll find you're at the minimum or some low number for whatever.

 

[DrChaos]

Yeah, "Highways" are ONLY Limited Access highways.

In my case, the freeways I use frequently are true limited access US Interstate (I-5, I-8, I-15) or California state highways, and that was what I was talking about in my experience. There is no "Auto" speed option on the freeway and the feel there is like the previous 11.4.9 version, whereas the feel in city streets is tremendously different.

Mostly improved but it has borked up some lane choices which the previous version got right. It drives like a human now, but like a human who is new to the area and doesn't know their way around, and makes mistakes in lanes and missing turns by failing to enter the turn lane sufficiently early.

 

[FSDtester#1]

Yeah, it's adjusting the set speed in the other mode that you're not using (the old v11 speed offset mode, which is also now broken, so there's no point in using it). It doesn't seem to have any obvious impact on the mode in use.

If you use the up scroll a lot and then switch modes, you'll find you're at 60mph or 85 mph (depending on the type of street). If you scroll down a lot then switch modes you'll find you're at the minimum or some low number for whatever.

At least the scroll speed is still working on the highway stack.
If it wasn't, highways would be manual driving.

 

[DrChaos]

It’s real curious why PA and other states aren’t getting it. It can’t be regulatory since it’s still beta

Likely good weather/better road states vs worse weather, ugly roads with poor markers.

 

[AlanSubie4Life]

It drives like a human now, but like a human who is new to the area and doesn't know their way around, and makes mistakes in lanes and missing turns by failing to enter the turn lane sufficiently early.

Yes. To me this problem seems unfixable. Since the NN acts like it's the first time it's seen a street, every time it sees it. There are a lot of tricky situations out there where you kind of have to know in advance how the lanes work. For example, when going straight, how would it know which lane to be in here? Normally the left lane should work (there's even two lanes on the far side of the intersection)! The only way humans know is that they've driven it before, or if they're really paying attention, they'll notice the obvious arrows on the road (the vast majority of drivers won't notice).

So that's the value in having a driver assist that only does roads with no turns, just traffic lights. That's where it has to excel. Adding complicated things like turning is just going to be too difficult to get correct.

Ironically, my example above is one where it can't handle going straight. It stays in the wrong lane. So there's even difficulties to just going straight!

Make the primary use case straight roads with intersections, with no lane merges or forks. That should be possible to solve.

 

[DrChaos]

Yes. To me this problem seems unfixable. Since the NN acts like it's the first time it's seen a street, every time it sees it. There are a lot of tricky situations out there where you kind of have to know in advance how the lanes work. For example, when going straight, how would it know which lane to be in here?

This isnt so hard--in theory. The maps have metadata about the lanes in each intersection and where they can turn. Car would have to recognize this sufficiently ahead of time and pre-emptively match the lanes now to the needed lane at the intersection, but it can do some of this already. It just makes mistakes.

Eventually with really large compute & feedback the developers could identify all those places where the autopilot consistently made mistakes compared to human training and put in annotated map hints and/or download videos with proper behavior and add those to the neural network training set.

But the fact that the early 12.3 is already so good is a promising sign to me, and I've previously been rather pessimistic. In the end, data and backprop has beaten the snot out of all sorts of difficult diverse CS problems over the last decade, upending all sorts of research careers and specialized solutions and decades of scientific work.

I really have no idea where the ceiling in performance will be.

Normally the right lane should work! The only way humans know is that they've driven it before, or if they're really paying attention, they'll notice the obvious arrows on the road (the vast majority of drivers won't notice).

Eventually I hope they will be able to do fleet learning and navigation---there are still more cars not using FSD and driving manually, so find their path information and choose the 'median path' that human driven cars take. This would take a big mapping and data ingestion infrastructure. Upon making a nav path, it would call back to the mothership which would supply additional fine pathfinding hints for the route based on history which had similar route goals (at least segments thereof) in navigation and yet were driving manually.

So that's the value in having a driver assist that only does roads with no turns, just traffic lights. That's where it has to excel. Adding complicated things like turning is just going to be too difficult to get correct.

Ironically, my example above is one where it can't handle going straight. It stays in the wrong lane. So there's even difficulties to just going straight!

Make the primary use case straight roads with intersections, with no lane merges or forks. That should be possible to solve.

 

[AlanSubie4Life]

This isnt so hard--in theory. The maps have metadata about the lanes in each intersection and where they can turn. Car would have to recognize this sufficiently ahead of time and pre-emptively match the lanes now to the needed lane at the intersection, but it can do some of this now already.



Eventually I hope they will be able to do fleet learning and navigation---there are still more cars not using FSD and driving manually, so find their path information and choose the 'median path' that human driven cars take. This would take a big mapping and data ingestion infrastructure. Upon making a nav path, it would call back to the mothership which would supply additional fine pathfinding hints for the route based on history which had similar route goals (at least segments thereof) in navigation and yet were driving manually.

Yep, they seem to have some elements of that now. Just seems really hard to get right, reliably, with frequent lane reconfiguration, construction, etc.

Just seems a lot harder than stopping, and maintaining speed.

 

[DrChaos]

Yep, they seem to have some elements of that now. Just seems really hard to get right, reliably, with frequent lane reconfiguration, construction, etc.

I think they'll have to go to having fleet-wide mapping and routing from human driven data, but then not use that too literally in the navigation path (don't force it as the actual path) but only as another input to the neural network planner and router, and train all of that with realistic circumstances. Like most of the time the fleet nav hints will be right but the train set has to include some where they're misleading or there's construction or sudden changes and the right path is something else.

 

[Ben W]

What's unclear is whether they are solvable.

The common low-hanging-fruit problems we're seeing, like weird stopping behavior at typical stop signs, and driving speeds on typical streets, I fully expect to be solved this year. And agreed that solving these issues alone may be enough to make it a useful L2 system for straight-line city-street driving. (By "useful", I mean one that is less mentally taxing to have on than off, without compromising safety. It's already there on the highway.)

But of course, there is an enormous gulf between "useful L2" and "L4". I am not yet convinced that city-streets L4 is solvable with current hardware (HW4), even with a decade of software refinement. For highway L3, my guess is 2026; i is emphatically not there now. I just did a 200mi highway drive yesterday, with a handful of critical interventions, most involving getting into the wrong lane or ramp. I also expect this to improve dramatically in the next year or two.

 

[lascavarian]

Likely good weather/better road states vs worse weather, ugly roads with poor markers.

Control group?

 

[Pdubs]

Is Tesla sending V12 only to "experienced" testers? Whatever that means.

11.4.9 has been sent to everybody, it seems, including those who have not yet paid forFSD and/or were not part of the early testers who had passed the safety score threshold.

But 12 appears to be going only to folks who had FSD V11 prior to 11.4.9. All of the updates to 12.x so far are from 2023.44.30 or before, of course, because 12.3 is 2023.44.25, and there is no "backdating".

But that means that none of these cars had been updated past 2023.44.16 in spite of the fact that there have been 9 updates since then. None of the 12.x cars received any of them. I'm not complaining, but why is this so?

I, for one, did not get invited to download any of those updates before the notice came that 12.2.1 was available. Many of the 10,000 TelsaFi folks who have updated to 2024.2.7 (11.4.9) were previously on 2023.44.30.14 or earlier, but almost all already had 11.4.9.

This all seems to suggest that Tesla is keeping a list of some set of early testers. Also it suggests that many of the 5,000 or so folks still on 2023.44.30.14 or earlier may well be on the list and may be updated to V12 soon, assuming it continues to roll out soon.

Again, I am not complaining. This all makes complete sense, to send it to experienced testers first. But when the production branch and FSD beta branch were merged with 11.4.9, there was some thought that the Beta program was over.

I think you're reading too much into it. Lots of guys who've been doing this for 2+ years haven't gotten it yet like @Ramphex . Some of them are my neighbors.

 

[Pdubs]

So.. Just to throw more cats into the henhouse..

At one time, pretty regular, there were the people with FSD and, unless one actually subscribed to FSD, there was no FSD, at all, in any way, on a Tesla. At that time, it was all pretty obvious: Those with the FSD loads had overall load numbers that were further back in time than Everybody Else. Everybody Else was getting nifty new features involving charging, games, screen layouts, and whatever the wiz kids back at Tesla could dream up. Those of us on FSD missed out on the whiz kid stuff, but we had our reward: Something that with a fair amount of hazard, could theoretically drive around on city streets.

Under this regime, the major load numbers and the FSD load numbers would line up roughly two or three times a year. At which point the FSD gang would get the whiz-kid stuff. And, to make a point about it, even when the load lines lined up, the FSD-enabled load was still different than the "main" load line. I dunno, making up an incorrect example, the main load line during a so-called line-up would be 2021.6.10.5, but the FSD load would be (at the line-up) 2021.6.10.6. Or 0.4.

As I said, the load lines lined up two or three times a year, but, very definitely, that line-up occurred at Christmas, at which point everybody got the holiday update. Even so, the FSD-containing load was still an .(n+1) or something different from the main load line.

Main point of all this: Suppose that, in March, some enterprising soul antes up $10k for FSD and pays up. They're not going to get FSD until the load line with FSD in it catches up with what they've got: They might wait months until that happened, grousing all the way.

This all changed in 2023. First, all the regular load lines acquired an FSD load. So, no matter what the main load number was, one would have access to 10.80.30 or whatever FSD was if one suddenly decided one wanted to either pay outright for FSD or for a subscription. No more waiting around for three or six months for the FSD load line to catch up: FSD would get enabled right off.

And then the miracle occurred last year in December: Not only did the FSD main load catch up to the load numbers that Everybody Else had, it became equal to what they had. Hah. THAT never happened before. So, we were all on 2023.44.30.whatever, both those of us who had paid for FSD and those who had not.

Interesting. So, the Everything Else gambit appears to be working. I think. People who haven't actually paid for FSD are getting all sorts of nifty updates: Better charging controls, better wi-fi and bluetooth controls, stuff like that, and it looks like the crowd is getting updated to 2024.8.4. But that load has 11.4.9 embedded in it. Meaning that if a 2024.8.4 user pays up for FSD, they're going to get 11.4.9. Which isn't too bad, I suppose.

In the meantime here's the rest of us with enabled FSD and we're messing around hoping to get 2023.44.30.25 with 12.3 baked in. Those who are moving to that are coming from a load from last year that had 11.4.9, or something earlier, or 12.2.1.

So, what's all this about, then?

1. Most of us waiting around don't even have the NHTSA icon font change Emergency Recall stuff that Everybody Else has got.
2. Methinks that 12.whatever isn't going General Release until it gets embedded in an Everybody Else load, so that people who want FSD immediately get 12.whatever.
3. The implication is that those of us who've opted into the whole FSD-b-testing regime are getting FSD loads that, I think, are more experimental than usual. Of, if not exactly experimental, are Tesla's serious attempts to come up with an FSD load that.. really.. is non-beta.

And that all might explain why things are moving more slowly than usual. Yeah, there's a lot more Dojo-style training going on, and Elon thinks that what they did between 12.2.1 and 12.3 may go faster. But, on the other hand, it looks like Tesla is trying to deploy just enough FSD-b of a release to get just enough valid data to train that release for the next step.

My conclusion: We just got through a wave of loads that worked across a dozen states or so. Betcha we end up waiting another week or so, then a different batch of states get a similar number of downloads. If nothing else, Tesla can do a contrast-and-compare. And this could end up repeating. Over and over with different batches of states until Tesla Is Happy. And, when they're happy - then it gets released to Everybody, even for those who haven't paid for it, embedded for the day that they do.

Fun.

You should write for a living. This would have normally been a tldr for me, but I was hanging on every word