Brad Templeton's F rating of Tesla's FSD

[TresLA]

Other companies aren’t selling their solution for 10k (soon 12k).

To be fair, other companies aren't selling their solution.

 

[diplomat33]

Not exactly. The idea is that if you need vision to work when the map is wrong, then that means the car can do it without maps. So if you trust that a map-using AV can safely drive when the maps are wrong, then you should trust the car when driving anywhere without maps (since maps can be wrong anywhere anytime because of construction, a parade, an accident, a cone, etc.)

But that is the fallacy. Vision is not always reliable as we all know. You may need vision to work when the map is wrong but that does not necessarily mean that the car can do it without maps all the time since your vision may not be reliable in all situations. Look at Mobileye. They are doing vision-only too with SuperVision but they still couple the vision-only with HD maps. They understand that vision-only can benefit from maps. Just because you are doing vision-only, does not mean you don't need maps.

That's the assumption I'm not convinced of. Waymo's working robotaxi is too restricted to be convincing. Case in point is that Waymo taxi rider who got stuck because of a few cones:

As I mentioned before, that was one incident back in August of last year that was caused by a bad remote operator. It had nothing to do with bad HD maps. It is an outlier. Keep in mind that Waymo has done millions of driverless miles.

But the reason I wrote that maps can make FSD more reliable in 99.9% of cases is because most of the interventions that I have with FSD Beta would be solved with maps. Yes, there will still be edge cases but HD maps can help with a lot of those common cases where FSD Beta still struggles.

Again, maybe it's regional, but FSD beta does great on right turns for me IF the cross traffic is controlled (e.g. - traffic coming from the left has a stop sigh, red light, etc.) or IF turning right at an intersection where we have a green light (right green or straight green). But yes, it's jerky/super hesitant/not good enough when turning right at a stop sign onto a large/fast street.

I will give you a few examples where FSD beta struggles for me, that I believe would be much better with HD maps:

- Doing an unprotected left from my residential lane unto a 55 mph main road. Now granted, it can be tricky with traffic. But even when there is no traffic at all, FSD Beta still struggles to even make the turn in a smooth way. It basically moves out and then jerks the wheel hard to the left, and swerves a bit as it centers itself in the left lane.

- Making a right turn from a 55 mph main road unto a 25 mph residential street. FSD Beta does not slow down enough and tries to take the turn too fast. It does not turn tight enough so it goes wide into the incoming lane.

- Making a protected right turn from a 55 mph main road unto a 25 mph campus road. Again, it does not slow down and takes the turn too wide going into the incoming lane.

- Making a protected right turn at an intersection. It slows down a bit but takes the turn too wide, once it almost hit the car in the street it was turning into.

- On my campus roads, even when I manually reduce the TACC speed, the car is still very hesitant and jerky in making the tight turns. it will also stop too long at an intersection and creep forward too much even when there is no traffic. FSD Beta takes too long to make the turn after the stop sign. And there are parked cars on the side so it is important to be precise. It's super easy for me as a human but FSD Beta struggles. I only try when the campus is empty. If the campus is busy, I don't even try because of the risk with students crossing the streets all the time.

Ex: tight turns, speed limit is 15 mph.

Ex: turning left. FSD Beta creeps forward too much, takes the turn way too slow and hesitant.

These two scenarios would be much better with HD maps as the car would know exactly how to steer to smoothly navigate the streets.

Tesla Vision is pretty good with detecting other road users so if it had HD maps to help it make the turns better, I believe FSD Beta would be able to handle these scenarios a lot better. I definitely cannot trust FSD Beta now to make those turns. Even when it does do the turn correctly, I can't be sure that it won't screw up in the future. Hopefully, Tesla will eventually solve these scenarios with vision some day.

 

[EVNow]

Other companies aren’t selling their solution for 10k (soon 12k).

Nobody ever said Tesla is like other legacy automakers.

 

[Dunsel]

A human can see and adjust quickly when a lane that used to be able to go straight is now a turn only lane. A human can adjust if the road they need to turn onto is blocked off because of an accident. But if a FSD car requires that kind of exact predetermined map to operate it would not be able handle these kind of everyday situations. And if the FSD car doesn't require such maps and can still drive competently and safely like a human could even when the predetermined map is wrong then you don't need such exact predetermined maps at all.

Yes, if something is not required then it is not needed.

No human can “drive competently and safely like a human” on their first drive.

We don’t want our cars to drive like humans.

We can’t download maps into human brains — they need to generate those through experience. We want our cars to drive safely immediately after we have received the update! Why not give them every advantage (like the best maps) to make up for the lack of experience they have?

To save processing time, humans are constantly comparing the present with the past - if something appears exactly as it was in the past why re-invent the wheel? Computing resources and time are limited. Instead of generating their own memories, cars can have the up-to-date collective memory of the fleet (and cartographers).

Would some students perform better on their practical driving tests if they studied detailed maps of the route and which maneuvers would be required BEFORE the test? Clearly not all students need such maps, but for some it might be the difference between passing or failing. If such maps aren’t always available to humans, we want them to fail. But don’t we want the cars to always succeed, even on the very first drive?

 

[gangzoom]

That was one instance out of millions of miles and it was caused by a remote operator giving the car bad advice. Waymo handles cones just fine all the time.

Really, its a computer algorithm, if it happens once, it'll happen again, and again, and again.

 

[diplomat33]

Really, its a computer algorithm, if it happens once, it'll happen again, and again, and again…

You need to learn about what actually happened in the incident. And I am not ignorant or dismissing the incident. It was a serious failure. But the incident happened several months ago. Waymo probably would have fixed the issue by now. They are not going to just let the problem continue to happen again and again after the first time.

 

[nvx1977]

Other companies aren’t selling their solution for 10k (soon 12k).

This is a non sequitur. Can we stop conflating the capability of FSD with the business practices of selling FSD? There are plenty of dedicated threads for the latter.

 

[nvx1977]

@bradtem

I appreciated your article. And also cool to see your presence here! TBH when I started to read the article, I was preparing myself to become defensive against the typical shallow FUD-like articles. But I read through all of it and agreed with your assessments. I think if I had not been part of FSD beta since 10.2, I may have had more optimism for FSD and likely would have disagreed more with your piece.

I do see incremental improvements since 10.2, but there are also lots of regressions. Overall, while I'm happy that FSD is improving, it's improving at quite a slow rate. Especially when things that most users expect to be easy (right turns at residential T intersections, for example) continue to be a struggle.

Musk would have us believe that the AI will be improving at an exponential rate, and early on in the curve, the improvements seem linear, and we has humans have a tendency to think linearly. So any day now, we might hit that inflection point where progress becomes noticeably faster. But from 10.2 to 10.8, I've yet to feel any acceleration in improvement, so for now, I'm going to extrapolate linearly and assume FSD won't approach L4 for at least 2-5 years.

AI Day did mention the idea of crowdsourcing mapping. It might not be HD mapping, but the idea that a particular intersection's rules can be refined and remembered through multiple cars' vision data seemed promising. Here in New England / northeastern USA, there are so many wacky intersections that don't have an analog anywhere else in the country. Drivers new to the area are regularly honked at for not knowing how to navigate these areas. I have no expectation that FSD could ever do better without some sort of memory of how the intersection works.

 

[bradtem]

@bradtem

I appreciated your article. And also cool to see your presence here! TBH when I started to read the article, I was preparing myself to become defensive against the typical shallow FUD-like articles. But I read through all of it and agreed with your assessments. I think if I had not been part of FSD beta since 10.2, I may have had more optimism for FSD and likely would have disagreed more with your piece.

I do see incremental improvements since 10.2, but there are also lots of regressions. Overall, while I'm happy that FSD is improving, it's improving at quite a slow rate. Especially when things that most users expect to be easy (right turns at residential T intersections, for example) continue to be a struggle.

Musk would have us believe that the AI will be improving at an exponential rate, and early on in the curve, the improvements seem linear, and we has humans have a tendency to think linearly. So any day now, we might hit that inflection point where progress becomes noticeably faster. But from 10.2 to 10.8, I've yet to feel any acceleration in improvement, so for now, I'm going to extrapolate linearly and assume FSD won't approach L4 for at least 2-5 years.

AI Day did mention the idea of crowdsourcing mapping. It might not be HD mapping, but the idea that a particular intersection's rules can be refined and remembered through multiple cars' vision data seemed promising. Here in New England / northeastern USA, there are so many wacky intersections that don't have an analog anywhere else in the country. Drivers new to the area are regularly honked at for not knowing how to navigate these areas. I have no expectation that FSD could ever do better without some sort of memory of how the intersection works.

Regressions are disappointing. All software teams I know of build giant regression test suites and every new build is run on all the logs or simulations of prior situations to minimize regressions. This is harder if you are using neural nets, and also if you are using a fleet of customer cars which don't upload a complete log of everything from a previous failure event. Tesla claims their simulator is great, and perhaps it is, but it doesn't seem to find these regressions you are reporting. That will be a bigger issue if it goes into production.

Crowdsourced mapping is what MobilEye (which has a fleet of 100M cars though not all can participate) and DeepMap (NVidia) are now doing, I will have an article about that soon.

Elon seems to like to put bold stakes in the ground, like not using LIDAR (and radar) and not using detailed maps. That makes it hard to back off from them. The error is that even if he's eventually right, and you can pull it off without those things, everybody else feels you want to use them to get there faster, and then later you can remove them if it works. His bet is the long bet, theirs is the more assured. Or rather, he feels his bet is that using these things is distracting people from the real winning choice. It's not impossible, but I don't agree with him on this.

Not using maps is crazy. A map is just remembering what you saw (up close and from all angles) when you or a cousin car drove here before. Why would you throw that away? It's also nice that a map can be built not in real time, and that humans can improve the QA on it, but even if you don't have those things, a car that can drive without a map is a car that can make a map. You drive better on the streets you know than the ones you see for the first time.

 

[Battpower]

like not using LIDAR (and radar) and not using detailed maps. That makes it hard to back off from them.

This is one thing that gets me. What engineering approach shuts the doors so firmly on any possible solution? If you are going to back one horse like vision, then surely step one would be to have a perfectly stitched and processed 360 image that you can treat as a single integrated data source? As for other sensors, just don't mention them until there is a compelling reason to give them consideration.

Trucks jumping from lane to lane in the rear quarter zones apparently as the data feed priority switches from one view to another would seem to negate the single sensor objective and rule out the commensurate benefits.

 

[CarlThompson]

My opinion is that Tesla (and Musk) already know they've lost the FSD race and any actions they take now are to lower costs while at the same time balancing the need to minimize the risk of legal action from those who made past buying decisions based on seemingly false or grossly misleading prior statements.

In other words Tesla still needs to appear to be working on FSD but wants to do so in a way that minimizes costs while still being plausibly believable.

 

[S4WRXTTCS]

HD maps are a crutch to deal with the current limited ability to perceive and process what the car perceives. Clearly it makes more since to improve the cars ability to perceive. How can a car that relies on networked HD maps truely be level 5?

Crutches serve a purpose though.

One purpose HD maps serve is a way to trigger additional learning when the perception is wrong. For example snow plowing removed lane divider dots on a multiple lane road near me. FSD Beta doesn't recognize it as two lanes, and centers itself in a way where its a bit in the other lane.

HD Maps would tell it that there are actually 2 lanes there.

The other purpose HD Maps serve is to give the car information when perception is itself limited due to poor signage.

When I drive in areas I'm unfamiliar with there are times where I screw up because I didn't perceive things correctly.

Like one time was a really dark multi lane intersection over a hill where there were multiple directions you could go, and no lane markings in the intersection. So it was easy to go "wtf" where you have to look on the maps to get a better indication.

I was hoping to test FSD beta there, but when I went back there I saw that they ruined it by adding lines and improving the lighting.

Lots of things REAL autonomous cars have on them can be considered crutches, but they serve a purpose.

Lidar
Rear Corner Radar
HD Maps

Other things have small, but important use cases like down facing cameras.

I'd rather overengineer a product, and under promise than do the Tesla thing of under engineering and over promise.

 

[CarlThompson]

Crutches serve a purpose though.

One purpose HD maps serve is a way to trigger additional learning when the perception is wrong. For example snow plowing removed lane divider dots on a multiple lane road near me. FSD Beta doesn't recognize it as two lanes, and centers itself in a way where its a bit in the other lane.

HD Maps would tell it that there are actually 2 lanes there.

This is actually a really good argument for why you DON'T want to rely on HD maps.

I used to live in a state where snow plowing would routinely turn 2 lanes into 1 and in such situations there's not a lot of room between the snow banks and it would be unsafe to try to squeeze 2 lanes into that space especially in a winter snowy situation where you should leave MORE space between cars.

 

[Battpower]

appear to be working on FSD but wants to do so in a way that minimizes costs while still being plausibly believable.

How many ground up rewrites will be tolerated before it becomes a hollow message? How slow does progress need to get before the only credible solution needs to be a(nother) ground up rewrite?

And while you lucky beta testers at least get something for your cash and efforts, other markets may never see FSD permitted based on current hardware capabilities, overall FSD performance and 'local regulations'.

 

[S4WRXTTCS]

This is actually a really good argument for why you DON'T want to rely on HD maps.

I used to live in a state where snow plowing would routinely turn 2 lanes into 1 and in such situations there's not a lot of room between the snow banks and it would be unsafe to try to squeeze 2 lanes into that space especially in a winter snowy situation where you should leave MORE space between cars.

The situation I was describing involved no snow as all the snow had melted.

It does present an interesting case where the human mind easily handles the task.

When there is no snow we use internal maps to know there is supposed to be two lanes there, and its not just a single lane. In my observation about 95% of the drivers knew there was two lanes and kept to their sides, and there was only one or two people who seem confused. This was over about a week of going through that area.

When there is snow banks we ignore the internal maps, and we normally form a single lane with maybe one or two idiots that try to squeeze by occasionally.

Essentially what I'm getting at is the equivalent of HD maps exist for humans. We're often driving over the same road over, and over so we know what changes. If someone steals a stop sign we know we're still supposed to stop.

 

[Battpower]

Our UK road markings present similar challenges pretty much on all roads. Bus stops (a boxed area right in the middle of the pavement / road surface - UK readers remember the pavement is where cars drive and not the pedestrian sidewalk) confuses lane position, as does the convention of lanes widening and a central line suddenly being added to make two traffic lanes. Several major cities have surface tram networks with rails set into the road surface for some sections then splitting off on more railroad like sections. After heavy rain road markings often become very hard to see when the road surface is evenly wet with surface water even though visibility is otherwise perfect.

There are so many city road layout conventions in UK that are pretty much all edge cases based on FSD beta performance so far.

 

[boonedocks]

Yes, if something is not required then it is not needed.

No human can “drive competently and safely like a human” on their first drive.

We don’t want our cars to drive like humans.

We can’t download maps into human brains — they need to generate those through experience. We want our cars to drive safely immediately after we have received the update! Why not give them every advantage (like the best maps) to make up for the lack of experience they have?

To save processing time, humans are constantly comparing the present with the past - if something appears exactly as it was in the past why re-invent the wheel? Computing resources and time are limited. Instead of generating their own memories, cars can have the up-to-date collective memory of the fleet (and cartographers).

Would some students perform better on their practical driving tests if they studied detailed maps of the route and which maneuvers would be required BEFORE the test? Clearly not all students need such maps, but for some it might be the difference between passing or failing. If such maps aren’t always available to humans, we want them to fail. But don’t we want the cars to always succeed, even on the very first drive?

Without maps and more importantly WITHOUT a memory, which us humans actually have, FSD continues to make the same mistakes ad nauseam.

There are many places on my daily commute that FSD has to be disengaged and if it had a map for these areas or a memory of it, both of which humans DO have in addition to our two cameras, the same errors wouldn’t be repeated.

It is truly over simplified to say “humans have two cameras and our cars have 8 and that’s all that is needed”. Just not a truly viable statement at all!!

 

[Bladerskb]

Waymo doesn't have true FSD. It only works at low speeds and only in specific carefully curated areas. Sure it's better than what Tesla has but that's not true FSD in my book.

You call routinely going 45+ mph low speeds?
You call 50sq mile, the size of an entire city a specific carefully curated area?

And what I'm telling you is that that's nonsense. Streets are constantly changing. You can't just map them once and expect them to remain the same forever!

Streets actually rarely change. Nevertheless crowd-sourcing solves this problem. Its already happening with tech like REM map that Mobileye has already deployed.

You do understand that what they wrote in a design document and what they allow the car to do in real life are two entirely different things, don't you?

There are hundreds of videos of Waymo driving with no driver that are freely available. Have you even bothered to watch any and check out the speeds for yourself?

That doesn't say that it can operate without the maps when they're wrong.

All self driving cars operate like this unless they would crash immediately if all they relied on was their map.

 

[Sporty]

My opinion is that Tesla (and Musk) already know they've lost the FSD race and any actions they take now are to lower costs while at the same time balancing the need to minimize the risk of legal action from those who made past buying decisions based on seemingly false or grossly misleading prior statements.

In other words Tesla still needs to appear to be working on FSD but wants to do so in a way that minimizes costs while still being plausibly believable.

My opinion is they’re on the ropes but I’m not sure about “lost”.

I do wonder if the price increase is a move to get fewer FSDs sold. And the current progress, intentionally or not, helps run the clock out. As more time goes on more people sell their cars. If they don’t deliver? Well if I sold my car I don’t care anymore and probably took a loss when I sold it.

I can’t imagine what the software eoy meetings are like. They seem to be nearly totally focused on fsd without progress for most?/(some?) customers for years. Then add in a few misteps in other sw areas along the way.

I think for my $10k I got a ‘ding’ for green lights over 4 years. And, of course, smart-summon (joke) and some fsd ui improvements but the best added feature was the ding. (I also got a regression on AP but that’s because I signed up for beta and went to vision only)

 

[TresLA]

The situation I was describing involved no snow as all the snow had melted.

It does present an interesting case where the human mind easily handles the task.

When there is no snow we use internal maps to know there is supposed to be two lanes there, and its not just a single lane. In my observation about 95% of the drivers knew there was two lanes and kept to their sides, and there was only one or two people who seem confused. This was over about a week of going through that area.

When there is snow banks we ignore the internal maps, and we normally form a single lane with maybe one or two idiots that try to squeeze by occasionally.

Essentially what I'm getting at is the equivalent of HD maps exist for humans. We're often driving over the same road over, and over so we know what changes. If someone steals a stop sign we know we're still supposed to stop.

Tesla is using maps, just not HD maps. Your Tesla knows a stop sign is coming up before the car can even see it (if the map is up to date). Perhaps not addressing your wider point, but at least in the case of someone stealing a stop sign, that’s pretty dangerous even in a world of just human drivers, since some will remember/know there’s supposed to be one there, and some will not (newcomers, infrequent drivers of the area, etc.)