Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

FSD Tracker + a couple of other Qs

This site may earn commission on affiliate links.
Hi, all --

Full disclosure, I'm short Tesla because I think the stock overvalued. I'm more than willing to discuss, but this isn't the right forum, nor relevant to my questions.

1: I'm really surprised that


isn't better known within the Tesla community -- including the Tesla short community! The sample size is really small, but longitudinally ... Is there something wrong with it? Is there a better source? Why isn't the response to, "FSD Beta is getting better over time," just, "I'm not going to disagree with your experience, but it's not universally shared" followed up with a link to the FSD Beta tracker? Tesla has released FSD miles-driven numbers, but as far as I know, other than that, this is the only data we have.

2: I might have missed something -- Has Tesla released any kind of white paper on V12? Is there any formal or informal documentation on how this is supposed to work?

3: Tesla continues to release, and people continue to test, versions from the V11 branch. Why?

4: There's a very long thread here devoted to V12. I'm up to date on it, I've read it as time has passed. But I'd like to poll community's sense as to what V12 actually is. (A white paper might help here!) Do you think it's actually photons-in, actions out?

Yours,
RP
 
Hi, all --

Full disclosure, I'm short Tesla because I think the stock overvalued. I'm more than willing to discuss, but this isn't the right forum, nor relevant to my questions.

1: I'm really surprised that


isn't better known within the Tesla community -- including the Tesla short community! The sample size is really small, but longitudinally ... Is there something wrong with it? Is there a better source? Why isn't the response to, "FSD Beta is getting better over time," just, "I'm not going to disagree with your experience, but it's not universally shared" followed up with a link to the FSD Beta tracker? Tesla has released FSD miles-driven numbers, but as far as I know, other than that, this is the only data we have.

2: I might have missed something -- Has Tesla released any kind of white paper on V12? Is there any formal or informal documentation on how this is supposed to work?

3: Tesla continues to release, and people continue to test, versions from the V11 branch. Why?

4: There's a very long thread here devoted to V12. I'm up to date on it, I've read it as time has passed. But I'd like to poll community's sense as to what V12 actually is. (A white paper might help here!) Do you think it's actually photons-in, actions out?

Yours,
RP
V12 has not been released to the public.

It was released to a few Tesla staff such as the CEO.

The theory is instead of coding by hands, the entire program is written by AI.

Like every single update since 2016, expect it to be mind boggling. Like every year, expect this year or next year to be L4 or 5.

Shorting Tesla based on FSD is a bad idea because most customers don't use it.
 
  • Like
Reactions: Kimmi
Hi, all --

Thanks for the replies! Let me narrow this down a bit: Is there any reason to think that


isn't a good source for, well, tracking FSD Beta? If it is, I'm surprised it isn't more widely referenced, If it isn't, (and I agree that more miles would be nice), why not?

Yours,
RP
The only true data is from Tesla.

Sources from others are just guesses. The "community" data is biased because it's not mandatory. Those who want to shape the answer would volunteer to contribute the data.

92% no disengagement (last 30 days) sounds good, but Uber had only 1 fatal accident, and its Autonomous Vehicle division was wiped out immediately.

GM Cruise had no fatality, but one non-fatal accident has shut down most of its activities, including executives.

On the other hand, Tesla has had quite a few fatalities, but the company is still making money after 7 years of government investigation.

Thus, betting on or against FSD is a bad idea no matter which source you got the data from.
 
Hi, Tam --

> The only true data is from Tesla.

Good point, but the only data *we* have is from tslafsdtracker. Tesla, for whatever reason, doesn't think it in shareholders' interests to release any details.

> betting on or against FSD is a bad idea

I probably shouldn't have disclosed my position, since it's really not relevant, I just didn't want to be accused of false-flagging later. I'm not focused on the stock price , although obviously the conversation has implications.

Eyeballing the tracker makes it look like there's been zero progress over the last 2 years. Highly unscientific social media monitoring (including of course the awesome TMC!) supports that; I still see roughly the same split between, "Haven't touched the wheel in weeks" (well, pre-recall) and "It tried to kill me within minutes."

Let's bin the social media monitoring, which honestly is worthless. Is there any reason to doubt the #s teslafsdtracker is throwing out, other than small sample size from self-selected audience? As far as I can tell, it's a fan-run site sourcing data from fans, so I don't think they're being funded by Big Oil or anything like that.

One of the mysteries to me is -- It can take a while for knowledge to percolate through a community. Like, the shorts were saying, "Unprofitable without ZEV credits" for a couple of quarters after that was no longer true; there are currently longs saying, "Industry leading margins" a couple of quarters after that stopped being true. The FSD tracker would seem to settle some questions that are still hotly debated, but is mostly ignored despite having been around for a long time. I'm wondering if there's a reason for that, like their data is known to be no good for some reason.

Yours,
RP
 
...The FSD tracker would seem to settle some questions that are still hotly debated, but is mostly ignored despite having been around for a long time. I'm wondering if there's a reason for that, like their data is known to be no good for some reason.

Please clarify what the FSD tracker answered. Does it answer that a Tesla can drive without a human driver (L4 or L5) by the end of this year, 2023, or 9 more days?

 
Please clarify what the FSD tracker answered. Does it answer that a Tesla can drive without a human driver (L4 or L5) by the end of this year, 2023, or 9 more days?

Hi, Tam --

The tracker updates a variety of metrics for each release. There is no predictive or forward looking content. For robotaxi purposes, I take the key stat as being "Mile to critical disengagement -- city", or whatever exactly they call it. My eyeballing shows roughly zero improvement anywhere over the last 2 years. A statistician (I can barely spell statistics!) might be able to refine that into "Less than 1% chance of >20% improvement" or something like that.

Your question makes me think you're not familiar with the site; one of the mysteries to me is why it's not more widely known.

Yours,
RP
 
I’ve been convinced for a few years that the camera-only autonomy is 10-20 years away and existing cars never will let the driver take the eyes off the road in a meaningful context and ofc never be a robotaxi. I still won’t short TSLA since Elon can move the stock with his cabal any way he likes. It’s an un-investable instrument in my view.
 
Hi, Tam --

The tracker updates a variety of metrics for each release. There is no predictive or forward looking content. For robotaxi purposes, I take the key stat as being "Mile to critical disengagement -- city", or whatever exactly they call it. My eyeballing shows roughly zero improvement anywhere over the last 2 years. A statistician (I can barely spell statistics!) might be able to refine that into "Less than 1% chance of >20% improvement" or something like that.

Your question makes me think you're not familiar with the site; one of the mysteries to me is why it's not more widely known.

Yours,
RP
I am not interested in that site because I only want to know can Tesla system safely provide hands-off and eyes-off experience.

Otherwise, it's just noise.
 
  • Like
Reactions: APotatoGod
Hi, Tam --



Well, I don't think any can know that right now, but the site's historical data can at last illuminate prior progress.

Yours,
RP

Thus, it is noise.

Mercedes has sold its L3, hands-off eyes-off in Germany for over a year with zero accidents. To achieve that it uses all the common sensors that Autonomous Sensors using and the driving restrictions that promote collision-free possible (daylight, not nighttime / dry road, not rain / highways, not city streets / 40 MPH, not 80 MPH...)

Thus, hands-off eyes-off is possible right now if we are willing to pay for resources. We need infrastructure for V2X. Maybe specific roads for the car so it can get all the info from the V2X sensors/transmitters.

Waymo can do L4 because it uses driving restrictions that promote collision-free possible: No strange roads. They are all familiar roads being geofenced. They then can expand to make strange places into familiar places: First was Phoenix, AZ, then San Francisco, then now Los Angeles...

We need to treat Autonomous Vehicles as a dumb machine, and we have to create conditions from V2X to algorithms to help those dumb machines from hitting objects.

Tesla and Cruise go the wrong way thinking their systems are smart. If it works in one scenario it will adapt and work in all scenarios on its own.
 
Hi, all --

Thanks for the replies! Let me narrow this down a bit: Is there any reason to think that


isn't a good source for, well, tracking FSD Beta? If it is, I'm surprised it isn't more widely referenced, If it isn't, (and I agree that more miles would be nice), why not?

Yours,
RP
Hello, I've been maintaining the community tracker for nearly 2 years.
Hi, all --

Thanks for the replies! Let me narrow this down a bit: Is there any reason to think that


isn't a good source for, well, tracking FSD Beta? If it is, I'm surprised it isn't more widely referenced, If it isn't, (and I agree that more miles would be nice), why not?

Yours,
RP
I've been maintaining the community tracker for nearly 2 years. People complain about sample size but not too many people are willing to put in the work to collect data. Tesla could just share the data, but at least there's a small group of us willing to share. I think the best thing about the tracker is that it's the only objective data we have on fsd. I've been tracking my data since Oct 2021 and alot of why the data looks like no progress is due to a bunch of little recurring issues that are still happening regularly.
 
Hello, I've been maintaining the community tracker for nearly 2 years.

I've been maintaining the community tracker for nearly 2 years. People complain about sample size but not too many people are willing to put in the work to collect data. Tesla could just share the data, but at least there's a small group of us willing to share. I think the best thing about the tracker is that it's the only objective data we have on fsd. I've been tracking my data since Oct 2021 and alot of why the data looks like no progress is due to a bunch of little recurring issues that are still happening regularly.
Hi, Elias --

Thanks for the reply! I appreciate it, as I do all of your great work with the tracker.

> I think the best thing about the tracker is that it's the only objective data we have on fsd.

Yeah! That's why I think it insane that it isn't more widely referenced.

If you've got the time, I'd really love to discuss the tracker. You know way more about it than I do. I'm interested in, what does it all mean? There's a lot of stuff there; to keep things tractable, let's focus on one #, city miles to disengagement, which I take to be the relevant metric for robotaxi scenarios. Here are some questions or observations. None of them should be construed as criticism of you or the tracker. I'm putting these out there because you would know the best. Also, there may be some suggestions for tracker enhancements mixed in.

1: This a question/maybe enhancement. Could you describe your feelings about the universe of testers/reporters? Is it pretty much the same group over time or do people drop in and out, like that kind of thing. In the unnamed table to the left of, "Tester locations by data volume", you could maybe add "#testers column" next to "Miles" or wherever it might fit. This would help, especially early in a release, in calibrating data quality (not the right word), like "Oh, 220 miles driven, but it's just that one guy in Virginia." Just, in general, anything you could share about the participants would be helpful.

2: What is the meaning of/your feelings about "City Miles 2DE Critical"? Over on the "fsd-v12-end-to-end-ai" thread (I think), someone was saying things like, "Well, that's all subjective, a user disengagement doesn't mean the vehicle would actually *crash*", which is I guess fair enough, but what else have you got? We can't measure crashes! On the other hand, there are factors that bias the value of that down, e.g., people not using FSD where they know it won't work. My tendency is just to take the # at face value and say, "As best as we can tell, an FSD robotaxi could maybe go a day without crashing." Is that wrong? What are your feelings about that?

3: Leaving the magnitude of CM2DEC aside, I'm struck by the longitudinal results. I haven't subjected them to statistical analysis, nor am I qualified to do so, but I eyeball these as, "Zero progress over the last 2 years." Actually, I eyeball everything you report as "Zero progress over the last 2 years," but for simplicities sake let's just stick with CM2DEC. Based on your intimate experience with the tracker, are there reasons to discard that eyeball impression?

(As a sidenote, I've been around software projects that were never going to ship, like, "You hired this four-person consultancy from Cleveland, they don't have the horsepower, they're in way over their heads, the next bug fix release will introduce new bugs, the next feature release will break existing features, you need to get rid of those guys and start over from scratch." Or, alternatively, "Unbeknownst to you, your 'simple' application requires you to solve a Noble Prize-winning mathematical problem, I suggest you abandon the project." From the very, very outside, that's what FSD development looks like to me. This isn't a criticism of the developers, who were perhaps tasked with doing the impossible.)

I've droned on long enough. My general question is, "Look! Here's the only source of data on FSD! What are its limitations? Should I be using this to inform my decision making? Colloquially, is it any good?" I'd just love hear your thoughts on this. Again, no criticism is intended at any point, I think you've been providing a great service.

Yours,
RP
 
...That's why I think it insane that it isn't more widely referenced...

I think a worthy site is the one that conducts various tests that Tesla needs to pass. If Tesla fails, it will fix their codes to pass those tests. Then, after that the site would invent new tests that Tesla will fail the tests. Then the cycle happens again, and Tesla would fix their codes to not only pass the old tests but new tests as well.

Those tests could include what owners have reported:

Unwanted slowdowns (phantom brakes)

Running red lights and stop signs

Unwanted veering with Autosteer (5 times in 1 month in Yosemite in 2021)

Dan O'Dowd tests on dummies

Lidar Tests, which frequently happened in CES
 
Hi, Elias --

Thanks for the reply! I appreciate it, as I do all of your great work with the tracker.

> I think the best thing about the tracker is that it's the only objective data we have on fsd.

Yeah! That's why I think it insane that it isn't more widely referenced.

If you've got the time, I'd really love to discuss the tracker. You know way more about it than I do. I'm interested in, what does it all mean? There's a lot of stuff there; to keep things tractable, let's focus on one #, city miles to disengagement, which I take to be the relevant metric for robotaxi scenarios. Here are some questions or observations. None of them should be construed as criticism of you or the tracker. I'm putting these out there because you would know the best. Also, there may be some suggestions for tracker enhancements mixed in.

1: This a question/maybe enhancement. Could you describe your feelings about the universe of testers/reporters? Is it pretty much the same group over time or do people drop in and out, like that kind of thing. In the unnamed table to the left of, "Tester locations by data volume", you could maybe add "#testers column" next to "Miles" or wherever it might fit. This would help, especially early in a release, in calibrating data quality (not the right word), like "Oh, 220 miles driven, but it's just that one guy in Virginia." Just, in general, anything you could share about the participants would be helpful.

2: What is the meaning of/your feelings about "City Miles 2DE Critical"? Over on the "fsd-v12-end-to-end-ai" thread (I think), someone was saying things like, "Well, that's all subjective, a user disengagement doesn't mean the vehicle would actually *crash*", which is I guess fair enough, but what else have you got? We can't measure crashes! On the other hand, there are factors that bias the value of that down, e.g., people not using FSD where they know it won't work. My tendency is just to take the # at face value and say, "As best as we can tell, an FSD robotaxi could maybe go a day without crashing." Is that wrong? What are your feelings about that?

3: Leaving the magnitude of CM2DEC aside, I'm struck by the longitudinal results. I haven't subjected them to statistical analysis, nor am I qualified to do so, but I eyeball these as, "Zero progress over the last 2 years." Actually, I eyeball everything you report as "Zero progress over the last 2 years," but for simplicities sake let's just stick with CM2DEC. Based on your intimate experience with the tracker, are there reasons to discard that eyeball impression?

(As a sidenote, I've been around software projects that were never going to ship, like, "You hired this four-person consultancy from Cleveland, they don't have the horsepower, they're in way over their heads, the next bug fix release will introduce new bugs, the next feature release will break existing features, you need to get rid of those guys and start over from scratch." Or, alternatively, "Unbeknownst to you, your 'simple' application requires you to solve a Noble Prize-winning mathematical problem, I suggest you abandon the project." From the very, very outside, that's what FSD development looks like to me. This isn't a criticism of the developers, who were perhaps tasked with doing the impossible.)

I've droned on long enough. My general question is, "Look! Here's the only source of data on FSD! What are its limitations? Should I be using this to inform my decision making? Colloquially, is it any good?" I'd just love hear your thoughts on this. Again, no criticism is intended at any point, I think you've been providing a great service.

Yours,
RP
Hi, Elias --

Thanks for the reply! I appreciate it, as I do all of your great work with the tracker.

> I think the best thing about the tracker is that it's the only objective data we have on fsd.

Yeah! That's why I think it insane that it isn't more widely referenced.

If you've got the time, I'd really love to discuss the tracker. You know way more about it than I do. I'm interested in, what does it all mean? There's a lot of stuff there; to keep things tractable, let's focus on one #, city miles to disengagement, which I take to be the relevant metric for robotaxi scenarios. Here are some questions or observations. None of them should be construed as criticism of you or the tracker. I'm putting these out there because you would know the best. Also, there may be some suggestions for tracker enhancements mixed in.

1: This a question/maybe enhancement. Could you describe your feelings about the universe of testers/reporters? Is it pretty much the same group over time or do people drop in and out, like that kind of thing. In the unnamed table to the left of, "Tester locations by data volume", you could maybe add "#testers column" next to "Miles" or wherever it might fit. This would help, especially early in a release, in calibrating data quality (not the right word), like "Oh, 220 miles driven, but it's just that one guy in Virginia." Just, in general, anything you could share about the participants would be helpful.

2: What is the meaning of/your feelings about "City Miles 2DE Critical"? Over on the "fsd-v12-end-to-end-ai" thread (I think), someone was saying things like, "Well, that's all subjective, a user disengagement doesn't mean the vehicle would actually *crash*", which is I guess fair enough, but what else have you got? We can't measure crashes! On the other hand, there are factors that bias the value of that down, e.g., people not using FSD where they know it won't work. My tendency is just to take the # at face value and say, "As best as we can tell, an FSD robotaxi could maybe go a day without crashing." Is that wrong? What are your feelings about that?

3: Leaving the magnitude of CM2DEC aside, I'm struck by the longitudinal results. I haven't subjected them to statistical analysis, nor am I qualified to do so, but I eyeball these as, "Zero progress over the last 2 years." Actually, I eyeball everything you report as "Zero progress over the last 2 years," but for simplicities sake let's just stick with CM2DEC. Based on your intimate experience with the tracker, are there reasons to discard that eyeball impression?

(As a sidenote, I've been around software projects that were never going to ship, like, "You hired this four-person consultancy from Cleveland, they don't have the horsepower, they're in way over their heads, the next bug fix release will introduce new bugs, the next feature release will break existing features, you need to get rid of those guys and start over from scratch." Or, alternatively, "Unbeknownst to you, your 'simple' application requires you to solve a Noble Prize-winning mathematical problem, I suggest you abandon the project." From the very, very outside, that's what FSD development looks like to me. This isn't a criticism of the developers, who were perhaps tasked with doing the impossible.)

I've droned on long enough. My general question is, "Look! Here's the only source of data on FSD! What are its limitations? Should I be using this to inform my decision making? Colloquially, is it any good?" I'd just love hear your thoughts on this. Again, no criticism is intended at any point, I think you've been providing a great service.

Yours,
RP
Sorry for the delayed reply. I didn't have email notifications turned on, and I didn't check back until now.
There's definitely a ton of different metrics and views on the dashboard. A lot of that was driven by community feedback, and also my craze with data. I love exploring data and uncovering new insights, so this has been a fun project for me.

Here are responses to your #'d comments, including some links for references:

1. Haha - I'm in VA! Most of the data is from only 5-6 testers.
Some people drop in and out, but most people dropped out. I get it, it's hard to be consistent and collect data on a daily basis. I'm sure it's also frustrating to keep reporting the same issue for months with no positive effect. It's frustrating sometimes when people complain about sample size or say the data is insufficent, but people don't realize how much time goes into data collection & reporting. It also takes a lot of patience to be consistent and keep tracking drives no matter how many times you know exactly how many disengagements you will get on that drive.
If you want to drill into data for specific testers, I would recommend the following tabs:
Tester View: You can filter by testers who have driven at least 500 miles on a version, so basically view data for testers with more consistency/miles
Tester % Drives: You can filter by Tester ID which is sorted by # of drives descending (high volume on top).

2. On the top of the page above the dashboards, there is a "Definitions (Tap to Expand)" option that opens up a section that has descriptions for everything including critical disengagement. The question "would the car have crashed" is very subjective. Critical disengagements are something that is safety related or illegal. I've seen my car not stop for a red light or try to move forward at a red light. If no one is around, that is still an unsafe action and thus a critical disengagement. I've had situations where both someone was there and not there and the car should not be having those critical disengagements at this point years into development IMO. FSD may function better in some areas than others, but as we've seen from videos posted for San Francisco, we still see FSD blowing through a stop sign as recently as v11.4.7.3. I've gone on 2 separate occasions over 1k consecutive miles without a critical disengagement, but there are too many little (non-critical) disengagements). However, some of these little disengagements like lane issues could lead to accidents. FSD could confuse other drivers or not get in the proper lane until the last second which could be dangerous. FSD as a robotaxi is definitely not ready. Until we get to a point where critical disengagements are rare, we're not close IMO. BTW - we saw a critical disengagement in the live stream on X from Elon with v12 where it tried to move forward at a red light. I'm hopeful for v12 but skepticism is still there due to all the hype put on v11 and it didn't bring us forward like we thought it would.

3. You can't focus on just 1 or 2 metrics. There are plenty of other metrics or views that show some form of progress. My favorite one is the "Issues Tracking" tab on the dashboard. We can see thousands of thing being fixed in this small tester pool. However, there are still a lot of persistent issues that have yet to be fixed. These issues that haven't been fixed is what is holding back progress. My commute analysis is probably the best example of this.
I have driven the same route every day (12 miles) for the past 2+ years, and except for 10.69.25.1, I always have 1 or more disengagements. That is why the metric for miles to disengagement stays low if you always have 1 or more disengagements on a short drive.

I really appreciate the thorough reply and I don't mind taking the time to explain / answer questions. Transparency and objectivity is important to me, and I try to let the data do the talking.
I've also been on software projects where it never got finished and only got worse over time, so I can definitely relate. I saw a lot of "we need to add all these features" instead of fixing existing bugs and it only ended up introducing more bugs.
In some ways, Tesla's FSD could be viewed as similar, as there are a bunch of bugs that have gone unfixed for a long time and new bugs like lane drift were introduced. Tesla has made improvements, but they are clouded by all of these other issues that have not been fixed or new issues being introduced. In the end, Tesla is collecting way more data than we are on this tracker, but I know that they know that it's not ready based on this small dataset that we have. If they were ready, robotaxi would already be here, and it isn't. I talked to someone who used to work for Tesla via DM on X, and I suspect Tesla compares their data to the community tracker. We were tracking 'safety critical disengagements' before Elon ever talked about it, so I wonder if they started tracking it because we did :)
 
Hi, all --

Full disclosure, I'm short Tesla because I think the stock overvalued. I'm more than willing to discuss, but this isn't the right forum, nor relevant to my questions.

1: I'm really surprised that


isn't better known within the Tesla community -- including the Tesla short community! The sample size is really small, but longitudinally ... Is there something wrong with it? Is there a better source? Why isn't the response to, "FSD Beta is getting better over time," just, "I'm not going to disagree with your experience, but it's not universally shared" followed up with a link to the FSD Beta tracker? Tesla has released FSD miles-driven numbers, but as far as I know, other than that, this is the only data we have.

2: I might have missed something -- Has Tesla released any kind of white paper on V12? Is there any formal or informal documentation on how this is supposed to work?

3: Tesla continues to release, and people continue to test, versions from the V11 branch. Why?

4: There's a very long thread here devoted to V12. I'm up to date on it, I've read it as time has passed. But I'd like to poll community's sense as to what V12 actually is. (A white paper might help here!) Do you think it's actually photons-in, actions out?

Yours,
RP
the tracker is a waste of time. people dont come online to tell everyone how their car had zero issues and was a good, boring driver all day. but if it makes one mistake, you'll hear ALL about it.

Negative voices are louder than positive ones. Thats a place for complainers to feel validated.
 
Hi, all --

Negative voices are louder than positive ones. Thats a place for complainers to feel validated.
the tracker is a waste of time. people dont come online to tell everyone how their car had zero issues and was a good, boring driver all day. but if it makes one mistake, you'll hear ALL about it.

Negative voices are louder than positive ones. Thats a place for complainers to feel validated.


Raurele, I'm not sure you understand what the tracker is or how it works. Fortunately, we've got Elias right here!

Elias, sorry for the slow response, I kind of wondered away. I again appreciate all the good stuff you've done creating (and explaining!) the tracker. My general feeling is that:

1: Due to small size and a whole bunch of other things, the absolute value of the #s is .... ehh, who knows? If somebody threw $1OOMM at this they'd doubtless come up something different, like, I don't know, 60 miles per CM2DEC instead of 50.
2: But the longitudinal part -- the "tracker", not the "measurer"! -- should be pretty good.

Yours,
RP
 
  • Like
Reactions: elias.d.martin
Hi, all --

Thought I'd put this here, as it seems to be the only teslafsdtracker discussion; I'll also be posting this on the V12 thread.

If you look at the the Teslafsdtracker --


, you'll see an enormous increase in miles per city critical disengagement, while all other stats are roughly in line. I'm looking for anecdotage on this; does that match your experience? Do you think 12.3 is much safer?

Yours,
RP