Safety Score

[Daniel in SD]

It's really interesting that they:
1) Came up with an "safety score" using previous data
2) That score is highly non-linear and biases everyone in the 80-100 range
3) That score weights things in a non-intuitive way
4) A score of 100 represents a collision rate lower than the average Tesla rate, yet no scores above 100 are possible
5) After a week of data, magically there are almost exactly 1000 people in the group.

Hmm, I wonder if they actually wanted something that gave them 1K "safest people" and actually created an algorithm that would to pick ~1K people if run for a week on a the data set, instead of being only focused on relative safety of drivers. For all we know it puts 1K at 100, 2K at 99, 3K at 98, etc, so it's actually just an engineered roll out under the PR of "safety."

Yep. Maybe they’re smarter than people give them credit for.

I think the weights in the score are very intuitive. By far the biggest factor is hard braking which intuitively would be correlated with hitting things. Think about braking events as having a long tail where at the extreme you’re braking at 100% and hit something. People with that distribution shifted to the left are safer drivers.

 

[dduffey]

Was driving on AP today when someone entering the highway didn't see me (blind spot) and tried to change lanes across two lanes to pass someone. AP was on and car took evasive action. Hands were on the wheel and was instantly kicked out of AP. Was worried that my score was dinged, but it didn't.

 

[jebinc]

Guess who's in the club

View attachment 718889

Thanks to whoever wrote Safety Score Calculator and the explanation of daily weighted scores, which I eventually figured out just in time for a 99->100 correction.

 

[powertoold]

Was driving on AP today when someone entering the highway didn't see me (blind spot) and tried to change lanes across two lanes to pass someone. AP was on and car took evasive action. Hands were on the wheel and was instantly kicked out of AP. Was worried that my score was dinged, but it didn't.

Are you saying AP did that swerve maneuver?

 

[Joels1017]

 

[nvx1977]

lots more entries on the spreadsheet today. Cory is looking at me, so please hit that subscribe button, I mean, please log your data here if you're so inclined. thanks -Sandy Munro.

FSD Beta - Participants and Rollout Status

Safety Score This spreadsheet is no longer actively managed, and most of the version data has gone stale. NOTE: If you have more than one Tesla vehicle, create a row for each one. Add new entries to the bottom. We will sort you into place. TMC Screenname (optional),Location,Safety Score - Mile...

docs.google.com

 

[dduffey]

Was driving on AP today when someone entering the highway didn't see me (blind spot) and tried to change lanes across two lanes to pass someone. AP was on and car took evasive action. Hands were on the wheel and was instantly kicked out of AP. Was worried that my score was dinged, but it didn't.

Are you saying AP did that swerve maneuver?

I can't be 100% for certain. I did not see him approaching my lane as he was to my right ... I'm a bit surprised the front camera doesn't catch when he enters the lane.

I did honk after the swerve / AP disengaged.

But the events of "AP disengaged, swerve, and OMG he is in my lane about to hit me" all happened (as memory can best recall) "at the same time".

And I didn't get dinged on it, no hard turns / etc.

 

[StealthLSU]

Well, started with a 96, had a 100 every day since and was at a 99 for at least a week, and finally today got up to a 100 score. Took about 100 miles today to get it, but now time to not drive for a couple of days and hope Elon stays to his word(haha) about everyone with a 100 getting it.

 

[gearchruncher]

You think they have some future prediction algorithm that would find exactly 1k people that will get 100/100? For someone who thinks Elon is just flailing around with no plan and no real path to L5 FSD (not saying you're wrong), I think you're giving him too much credit

If you have 6B miles of data, it's trivial to mess around with the coefficients until you kick out ~1K people with 100 scores. It's much easier than making a real safety score.

 

[gearchruncher]

I think the weights in the score are very intuitive. By far the biggest factor is hard braking which intuitively would be correlated with hitting things.

Not unsafe following distance, which is the least important? It's intuitive to you that hard braking is 100X more predictive of hitting something than unsafe following distance, even though we know hard braking can be caused by a hill or yellow light, but unsafe following distance by definition means you are very close to a car at >50 MPH? And acceleration or speeding don't matter AT ALL?

What I find most interesting is how deeply they claim a Forced AP disengagement means you are an unsafe driver. Is that really super correlated to collisions, or is that maybe just a filter for getting AP? I mean, you're on AP at the time- a collision there would actually be caused by the failure of AP to avoid the collision.

Everything about this reads as if it was engineered to find 1000 people that would be good AP users, without ticking people off too much by including acceleration, and that they didn't really care which 1K users they got out of the 15K that would do just fine, as long as it was a good game that distracted for a few weeks from FSD being delayed yet again.

 

[EVNow]

What I find most interesting is how deeply they claim a Forced AP disengagement means you are an unsafe driver. Is that really super correlated to collisions, or is that maybe just a filter for getting AP? I mean, you're on AP at the time- a collision there would actually be caused by the failure of AP to avoid the collision.

I think this means the driver didn't respond to AP nags - so not paying attention. In the context of FSD Beta testing, this would be important.

 

[AlanSubie4Life]

What I find most interesting is how deeply they claim a Forced AP disengagement means you are an unsafe driver. Is that really super correlated to collisions, or is that maybe just a filter for getting AP? I mean, you're on AP at the time- a collision there would actually be caused by the failure of AP to avoid the collision

I think you are confused by what this means. It means you ignore all the AP nags.

Everything about this reads as if it was engineered to find 1000 people that would be good AP users,

I don’t think there is really that much about the score that incentivizes AP use. It’s really not necessary to use it at all (except for selective brief masking) and on average I think will result in worse scores than manual driving, mostly due to the limitations of AP capabilities. There are situations where AP is useful for scoring optimization but generally I don’t use it.

Certainly if you want to be sure of getting 100 for the day even if someone cuts you off, you need to spend significant time off of AP. (This will instantly drop you to a 99 if you preferentially use AP but happen to be in manual mode when it happens.)

 

[StealthLSU]

agreed, if you don't respond to nags from AP, it means you likely won't for FSD beta either, and since it is just a beta and will likely cause a crash if you aren't paying attention, they need people who are fully attentive.

 

[FalconFour]

I think you are confused by what this means. It means you ignore all the AP nags.

I don’t think there is really that much about the score that incentivizes AP use. It’s really not necessary to use it at all (except for selective brief masking) and on average I think will result in worse scores than manual driving, mostly due to the limitations of AP capabilities. There are situations where AP is useful for scoring optimization but generally I don’t use it.

Certainly if you want to be sure of getting 100 for the day even if someone cuts you off, you need to spend significant time off of AP. (This will instantly drop you to a 99 if you preferentially use AP but happen to be in manual mode when it happens.)

AP is most useful for "I literally need to use my car to drive, and I want to keep this perfect score". If you're a good AP user and know how to set it/where to be at what times, AP is awesome. I'd say it was the most useful part of keeping me at 99 and now being 100. If you can manage AP properly (predict its F-ups, leverage its advantages, prevent its unmasked faults on disengagement), it can be a real benefit.

Especially now that (we/I realize more clearly that, rather) daily scores are mileage-weighted. That means AP usage will roll in the miles for your daily score, amplifying it, whatever it may be - while keeping it easier to correct any mistakes you made (since it doesn't count positive nor negative ratios, you can easily add a quick positive to fix a negative).

It's just another tool in this game

 

[ek8990]

Was driving on AP today when someone entering the highway didn't see me (blind spot) and tried to change lanes across two lanes to pass someone. AP was on and car took evasive action. Hands were on the wheel and was instantly kicked out of AP. Was worried that my score was dinged, but it didn't.

Keep in mind that it doesn't count anything within 3 seconds of AP being disengaged.

 

[AlanSubie4Life]

If you can manage AP properly (predict its F-ups, leverage its advantages, prevent its unmasked faults on disengagement), it can be a real benefit.

Yes, it has its benefits for particular types of driving. I'm just saying it's not really strongly incentivized, and that it's essential that you spend some time off of AP.

 

[gearchruncher]

I think this means the driver didn't respond to AP nags - so not paying attention. In the context of FSD Beta testing, this would be important.

Of course it does.
And in context of FSD beta testing, it is important.
However, is it really intuitive that ignoring these nags for sure makes you an unsafe driver?
I mean my whole discussion here was about the intuitiveness of the weights of the scores for SAFETY as a NORMAL driver, not how likely Tesla wants you in the "FSD" "Beta".
Because it's a safety score, right? Not a FSD acceptance score? Not a game? It's a highly engineered, data driven score that is clearly right. Anyone that has ever once got kicked out of AP is a tragic driver and should be taken off the road immediately. The score has spoken.

 

[EVNow]

However, is it really intuitive that ignoring these nags for sure makes you an unsafe driver?

Yes - it indicates not paying attention.

 

[AlanSubie4Life]

It's a highly engineered, data driven score that is clearly right.

Has anyone claimed that? As I understand it, even Tesla has not claimed that.

 

[gearchruncher]

and since it is just a beta and will likely cause a crash if you aren't paying attention, they need people who are fully attentive.

Auto wipers are a beta.
Every single AP feature to this point is a beta.
It means nothing. Tesla has ruined the word "beta" on purpose.

Do we really expect City Streets Autosteer beta to be so bad, that the expected outcome of ignoring nags is a crash?
If that's true, it means you're expecting FSD beta to be attempting to crash about every 90 seconds.
Meanwhile others are saying they fully expect it to handle a 20 mile drive to work with no intervention.