Brad Templeton's F rating of Tesla's FSD

[voyager]

A Robocar Specialist Gives Tesla ‘Full Self-Drive’ An ‘F’

A look at Tesla's badly named "full" "self" driving product (FSD) from a specialist in self-driving cars finds it badly wanting. Full video included with sample ride.

t.co

Who is Brad Templeton? He covers robocar technology in media such as Forbes, and previously worked
on Google's car team. He chairs the program on computing and networking for Singularity University.

https://twitter.com/x/status/1481764387253747714

 

[Battpower]

I surprised this kind of review is not more common.

His point about visualisations showing items appearing and disappearing or jumping around when such behaviour is impossible is something that hasn't changed much throughout the FSD journey.

I'm not 'spitting with rage’, 'foaming at the mouth' or sticking pins into an Elon doll. Neither to I regret paying a fee £££ in the UK where FSD City Streets really has minimal chance of seeing the light of day during my car's life.

But I kinda regret being associated with a product and company that should be (needs to be) a flagship as the pinnacle of integrity, sustainability, longevity and technology, when in fact the only thing that has performed incredibly well once the show got moving is TSLA - but even that is guaranteed.

As a human driver I am very aware of what I would call my own confidence and ability level (like are my glasses clean, am I wide awake and in good shape to drive, are the mirrors and seat adjusted correctly and is the car checked out and in good shape?) While actually driving, I have an idea about situational confidence such as is it foggy, am I driving into bright sun, is visibility likely to change.

These factors have a big effect on how I negotiate any given stretch of road. I use sound and feal as well as sight. If I hear an emergency vehicle siren, I am extra vigilant.

Being responsive to changing situational dynamics obviously goes far beyond keeping a safe distance from the car in front and completing a journey without hitting anything, but it feels FSD City Streets is still devoid of any notion of 'how confident am I' and 'how am I doing'. 'Driving Good Citizenship' is an important part of driving that is completely missing from the FSD model it would appear.

I sincerely hope we start to see some solid and consistent evidence that the points Brad makes are being addressed with solid solutions.

 

[boonedocks]

That was actually a very good, honest and real review of FSD Beta in its current state. I can't use FSD Beta when any/all of my family is in the car and a lot of it has to do with exactly as Brad described as "jerky" from both steering and braking and accelerating.

I have pre-paid for FSD on three of my previous 4 Teslas and did NOT on my current Refresh S. I am currently a monthly subscriber and in the FSD Beta program but will probably bail on the subscription as well once / if Tesla raises it ~20% as they are on the purchase price.

 

[K1Woods]

That was actually a very good, honest and real review of FSD Beta in its current state. I can't use FSD Beta when any/all of my family is in the car and a lot of it has to do with exactly as Brad described as "jerky" from both steering and braking and accelerating.

I have pre-paid for FSD on three of my previous 4 Teslas and did NOT on my current Refresh S. I am currently a monthly subscriber and in the FSD Beta program but will probably bail on the subscription as well once / if Tesla raises it ~20% as they are on the purchase price.

I agree with the points made - it's fair and balanced.

On my last Tesla, I purchased FSD at the $8K level and am currently a subscriber to FSD Beta with my 2022. I'm debating whether or not to renew my subscription. After completing the safety process and receiving beta, it's difficult to decide whether or not to cancel the subscription if you plan to acquire it later.

As a driver, I discovered that you become accustomed to the FSD driving style to the point where it becomes rather predictable. The jerky stuff will get corrected with time. What concerns me is that while it can excel at doing intricate maneuvers, it struggles with simple tasks such as taking corners too quickly. In tight intersection turns with nearby stationary vehicles stopped at junctions, my car will sometimes generate FCWs. That really irritates me and is extremely embarrassing if others are in the car - the beta generating its own FCW.

I have another stop signal intersection where it frequently fails to make a left turn at a light. FSD slows and sometimes stops in the middle of the road when there are no cars in sight. When I'm demoing FSD to others, I avoid these areas. My guess is that others do the same thing - they disengage where there are issues thus lowering the bar. I know it'll get better but geeze.....c'mon man. Raising the price at this point is rich, I'm already getting the "how much did you pay for this??"

 

[bradtem]

Thanks for calling it fair and balanced. And yes, my sweetie doesn't want to ride with FSD or use it, either. I wanted it to be better, in spite of what the haters think about my giving it a poor grade. I look at it with a different lens than they do. If you've never seen a self-driving car, then FSD makes you think you've got KITT with some flaws. But it is far from that.

Now I only paid $2,000 for it (one week special back in 2019.) When it jumped up to much more, I told people, "Don't buy it. If you believe in it, buy TSLA shares instead." Boy was that great advice, far better than I imagined it would be. After all, if Tesla does deliver on FSD, your TSLA shares will return more than enough to pay the higher future price. Today, at $12K and with the stock so highly valued, this prediction is a big tougher to make.
Sadly, you just don't get a lot. You get smart summon (used it once, gave up) and nav on autopilot (only modestly useful) and maybe FSD. Now, that's over EAP, and I had EAP with my car which is why FSD was only $2K. Over basic autopilot you get a bit more, autopark and lane change, but $12K?

 

[CarlThompson]

Yeah, FSD sucks and as it is it's a waste of money. I made that mistake in my first Tesla but I won't buy it for my second.

I do understand why Tesla refuses to use maps, though. It's very simple: all maps, no matter how up to date, are outdated in some places. If you use maps you still need the car to understand how to ignore the map and operate safely without it for those spots where the map is wrong. And since it needs to know how to override the map and operate safely without it there's no point in using the map at all; it just adds complexity.

 

[Dunsel]

Great Review.

I’m not a robocar consultant, but it is obvious that the lack of maps is a design flaw. This flaw handicaps the Tesla — so much of the driving information would be ready a priori, but this forces the car to figure everything out ex nihilo. Tesla will eventually have maps.

I paid $6,000 for FSD and I love autosteer on city streets and use it all the time every day. I concur with the reviewer that just like autosteer on the highway, one has to learn HOW to use autosteer on city streets. I use the most aggressive settings and I constantly use the go pedal and turn signal to help the car. The result is a much more relaxing and safer commute. In addition to knowing that I’m staying in my lane and not turning into anyone, I can also detach from the other crazy drivers — I don’t react to their driving because I’m not driving, the car is driving and I’m just assisting.

 

[EVNow]

Who is Brad Templeton? He covers robocar technology in media such as Forbes, and previously worked
on Google's car team. He chairs the program on computing and networking for Singularity University.

A guy from “singularity“ university calls FSD badly named

Anyway, ex-Google guy, so dogma is strong. I remember how all the industry experts from GM to Mercedes laughed at Tesla for panel gap. Who is laughing now ?

 

[Battpower]

Sure, with solid evidence of improvement, you could grade based on hopefully eventually converging with your targets, but there seems to be quite a lot of retrograde steps like increase in braking issues with VO.

FSD (as specified by any means used to date) is clearly a long way off still and not much in the way of consistent improvement to clutch at based on reports like Brad's.

 

[diplomat33]

And since it needs to know how to override the map and operate safely without it there's no point in using the map at all; it just adds complexity.

Yes, you need to be able to operate safely when the map is wrong but the map will be wrong like .1% of the time and your fleet can automatically update the map when it is wrong. The other 99.9% of the time, the map will be up to date and will improve your FSD a lot. So you only need to be able to operate without the map .1% of the time and the other 99.9% the map will greatly improve your FSD. A little extra complexity in exchange for better FSD 99.9% of the time is a good trade IMO. So maps are far from useless. The benefits outweigh the costs.

Think of it this way: if HD maps allowed FSD Beta to be true L4 for like 99.9% of driving, meaning you did not need to supervise the car at all for 99.9% of driving, would it be worth it? I think so. The alternative is that Tesla refuses to do HD maps and FSD Beta struggles and requires driver supervision. That's dumb.

 

[qdeathstar]

Does real life grade on a curve? Would you get a passing grade because your car almost saw that pedestrian in time?

It is irrelevant. We are judging a new technology and all that really matters is how it stacks up against its competition.

FSD doesn’t have a problem seeing pedestrians… just mainly cross traffic.

 

[JulienW]

It is irrelevant. We are judging a new technology and all that really matters is how it stacks up against its competition.

FSD doesn’t have a problem seeing pedestrians… just mainly cross traffic.

Yes, in most cases it is MUCH more about working on predictions and reactions to VRUs. I also see the cross traffic/B-pillar as possibly the biggest weakness in the cameras ability to see.

 

[TresLA]

Yes, you need to be able to operate safely when the map is wrong but the map will be wrong like .1% of the time and your fleet can automatically update the map when it is wrong. The other 99.9% of the time, the map will be up to date and will improve your FSD a lot. So you only need to be able to operate without the map .1% of the time and the other 99.9% the map will greatly improve your FSD. A little extra complexity in exchange for better FSD 99.9% of the time is a good trade IMO. So maps are far from useless. The benefits outweigh the costs.

Think of it this way: if HD maps allowed FSD Beta to be true L4 for like 99.9% of driving, meaning you did not need to supervise the car at all for 99.9% of driving, would it be worth it? I think so. The alternative is that Tesla refuses to do HD maps and FSD Beta struggles and requires driver supervision. That's dumb.

Maybe it’s regional, but here in california there is so much construction popping up and then gone that I’d be scared to rely on maps. Another way I think about it is: as often as I see cones is the minimum of when I can’t rely on maps (i.e. - there are more situations than just “cones present” where the map is no longer reliable). And look how happy we were to see cones appear in the visualization (I had to wait 6 months longer because I needed HW3 upgrade first). There are a lot of cones out there.

 

[diplomat33]

Maybe it’s regional, but here in california there is so much construction popping up and then gone that I’d be scared to rely on maps. Another way I think about it is: as often as I see cones is the minimum of when I can’t rely on maps (i.e. - there are more situations than just “cones present” where the map is no longer reliable). And look how happy we were to see cones appear in the visualization (I had to wait 6 months longer because I needed HW3 upgrade first). There are a lot of cones out there.

But what about when there is no construction? Wouldn't it be great if FSD Beta could just be reliable in those normal cases when there is no construction zone?

I know the idea is that if you need vision to work when the map is wrong then it is better to just use vision all the time. The problem is that vision-only doesn't work all the time. So not using maps does not make the system any better, you are still left with a system that won't work reliably. And like I said, if maps can make your system more reliable in 99.9% of cases, that is still a net positive IMO.

In my area, FSD Beta can't even do simple right turns without being very jerky and hesitant. I would be very happy if FSD Beta could just do simple turns reliably and if that takes a little mapping, so be it. I guess I am just frustrated because all the problems I am having with FSD Beta could easily be solved with a little mapping.

 

[CarlThompson]

Yes, you need to be able to operate safely when the map is wrong but the map will be wrong like .1% of the time and your fleet can automatically update the map when it is wrong. The other 99.9% of the time, the map will be up to date and will improve your FSD a lot. So you only need to be able to operate without the map .1% of the time and the other 99.9% the map will greatly improve your FSD. A little extra complexity in exchange for better FSD 99.9% of the time is a good trade IMO. So maps are far from useless. The benefits outweigh the costs.

Think of it this way: if HD maps allowed FSD Beta to be true L4 for like 99.9% of driving, meaning you did not need to supervise the car at all for 99.9% of driving, would it be worth it? I think so. The alternative is that Tesla refuses to do HD maps and FSD Beta struggles and requires driver supervision. That's dumb.

I think you're way off with your belief that maps are 99.9% accurate. In my opinion at best with a herculean effort (i.e., incredibly expensive) you could maybe keep your maps 95% accurate if you're lucky.

I do absolutely agree that maps would make it easier to have passable FSD features for that 95% of the time when the map is accurate. And if you were simply making a driver assist feature that would be good enough. But the problem is true FSD needs to work in 100% of the situations that an average human can handle. In order to do that you need to have a system which can handle that remaining 5% safely and effectively just as a human could. And the exact same skills needed to handle that 5% would allow the car to handle all 100%. And remember, here in the U.S. we just passed a huge infrastructure bill last year. Do you think all the changes that will be made to our streets will be captured by maps instantaneously?

A long time ago when I was a software engineering manager I had a saying that I think is apt: "Show me something that works 99% of the time and I'll show you something that doesn't work."

 

[diplomat33]

I do absolutely agree that maps would make it easier to have passable FSD features for that 95% of the time when the map is accurate. And if you were simply making a driver assist feature that would be good enough. But the problem is true FSD needs to work in 100% of the situations that an average human can handle. In order to do that you need to have a system which can handle that remaining 5% safely and effectively just as a human could.

You have it backwards. You can do passable driver assist that works 95% without HD maps. See FSD Beta. But to do true FSD, you need HD maps. AV companies like Waymo which have true FSD, use HD maps.

Yes, maps can sometimes be wrong. That is why AVs like Waymo don't just rely on HD maps. They combine HD maps with cameras, lidar and radar to have a complete system that can handle what HD maps can't or handle what vision-only can't.

Yes, true FSD must work 100% of what an average driver can handle. That is why you need HD maps to do true FSD. Vision-only might get you to 95% but can't get you to 100%!

The fact is that if we look at the current state of autonomous driving, it's the AV companies that use HD maps that have true FSD and Tesla that is still struggling. So clearly HD maps can't be that bad!

And the exact same skills needed to handle that 5% would allow the car to handle all 100%.

That sounds nice in theory. Except that does not work because we see from FSD Beta that vision-only cannot handle all 100%.

Do you think all the changes that will be made to our streets will be captured by maps instantaneously?

Not instantaneously but very quickly. Updating HD maps is very easy now. You just need to drive the new area once to capture the data and then process it and upload it to the entire fleet. Waymo says that a lot of their map changes are actually done automatically.

 

[CarlThompson]

AV companies like Waymo which have true FSD, use HD maps.

Waymo doesn't have true FSD. It only works at low speeds and only in specific carefully curated areas. Sure it's better than what Tesla has but that's not true FSD in my book.

For example, Waymo can drive even when the maps are wrong.

I'm not sure that's true. What actual evidence do you have to back that statement?

Vision-only might get you to 95% but can't get you to 100%!

At the risk of sounding like Elon I'll just point out that your statement is obviously false because humans (and other animals) do exactly that.

Except that does not work because we see from FSD Beta that vision-only cannot handle all 100%.

You won't get any argument from me there. Tesla's current implementation is garbage. But you are creating a straw man argument. I didn't say vision-only is the way to go. I just said that true FSD can't rely on maps (outside of for basic navigation).

Not instantaneously but very quickly. Updating HD maps is very easy now. You just need to drive the new area once to capture the data and then process it and upload it to the entire fleet.

And what I'm telling you is that that's nonsense. Streets are constantly changing. You can't just map them once and expect them to remain the same forever!

 

[qdeathstar]

The fact is that if we look at the current state of autonomous driving, it's the AV companies that use HD maps that have true FSD and Tesla that is still struggling. So clearly HD maps can't be that bad!

Those companies only operate in a very small geographical region. If you limit where you operate then perhaps hd maps are more tenable. What what do you do when you need to cover an entire country with HD maps and the server goes down or is hacked?

HD maps are a crutch to deal with the current limited ability to perceive and process what the car perceives. Clearly it makes more since to improve the cars ability to perceive. How can a car that relies on networked HD maps truely be level 5?

That sounds nice in theory. Except that does not work because we see from FSD Beta that vision-only cannot handle all 100%.

But you don’t know if that is a perception issue or an issue with what to decide to do with the perception. Saying it is impossible for it to work because it isn’t working currently is foolish.

 

[diplomat33]

Waymo doesn't have true FSD. It only works at low speeds and only in specific carefully curated areas. Sure it's better than what Tesla has but that's not true FSD in my book.

Waymo works up to 65 mph according to the ODD that Waymo published. That is not "low speeds" in my view. You might be thinking of Cruise which limits their AVs to 35 mph.

And Waymo is L4. I consider L4 to be "true FSD" since the car is driving with no human supervision. I guess you only consider L5 to be "true FSD"?

I'm not sure that's true. What actual evidence do you have to back that statement?

Waymo has said so repeatedly. Here is one example:

You won't get any argument from me there. Tesla's current implementation is garbage. But you are creating a straw man argument. I didn't say vision-only is the way to go. I just said that true FSD can't rely on maps (outside of for basic navigation).

You are using a strawman now as well. Nobody relies exclusively on HD maps. AVs like Waymo also use cameras, radar and lidar to drive. HD maps are just one part of their autonomous driving stack.

And what I'm telling you is that that's nonsense. Streets are constantly changing. You can't just map it once and expect them to be the same forever!

No, you don't just map it once and forget it. Roads will change at some point. My point was that if a car detects a change to the map, it can update the map for the fleet. Waymo and others have processes to update the map when needed. But it is also why they have a system that does not rely on maps alone.

 

[Sporty]

Those companies only operate in a very small geographical region. If you limit where you operate then perhaps hd maps are more tenable. What what do you do when you need to cover an entire country with HD maps and the server goes down or is hacked?

HD maps are a crutch to deal with the current limited ability to perceive and process what the car perceives. Clearly it makes more since to improve the cars ability to perceive. How can a car that relies on networked HD maps truely be level 5?



But you don’t know if that is a perception issue or an issue with what to decide to do with the perception. Saying it is impossible for it to work because it isn’t working currently is foolish.

Maps are a crutch in the sense that humans drive just fine without (necessarily) having a map of what the area they are driving in. I know, for me, I sometimes slow way down in areas when navigating in an unfamiliar place.

In any case there have been a couple of posts that have indicated that fsd is doing the wrong thing because the map is wrong. (Where stop signs are. Lights. Posted & appropriate driving speeds). Using the map is a pragmatic approach to try solving some of the problems right now. It be could argued ‘shouldn’t be needed’ but it could also be argued ‘it’s just another input’. I’d say a human actually does normally drive with a map in their head. E.g., During the vast majority of my driving, I ‘know’ to a high degree what’s coming around the next corner. And if I’m surprised because there’s something outside of my expectations I slow down and it grabs more of my attention. If it stays that way for many days, I update what ‘know’ about that area.

I think map usage by cars is fine and even makes sense. But yes the sw will need to know how to cope when it perceives something that doesn’t align well with its side band knowledge of the area.