A bit old news at this point, (and sorry if this was already explored, I
really can't keep up with this thread) but I just finished listening to the Tesla Daily podcast featuring the interview with Jake Fisher from Consumer Reports on the Model 3 reliability score, and found it quite enlightening, so I thought it might be beneficial to some of those who simply don't have the time.
TL
R -
The average reliability score for the Model 3 is not calculated as an average of the 17 different reliability categories, but rather it shows how this car compares to the reliability of all other 2018 models released by all other manufacturers.
Key takeaways:
- The vast majority of newly-released cars (2018 models) will not experience any major issues, particularly in their first year on the roads, so the comparison becomes much more granular and things like faulty door handles or strange cabin sounds may result in a different classification
- Jake was very careful to specify that a below-average reliability score does not mean the car is unreliable (meaning, it will break down and leave you stranded), but simply that on the scale of all other 2018 models it has on average more issues to be addressed than the median car; he agreed that this was not clear in their report and they will probably make changes in how they report these numbers in the future
- the reliability score is calculated by using different weights for different items, for instance faulty trim is weighed much lower than drivetrain issues; he also mentioned that it showed them that the Model 3 has a lot of what one may call cosmetic issues (trim, misaligned body panels, paint, etc.) compared to the average, in order to get that 2/5 rating
- As they only have data for a relatively short time span on new models, they use the reported issues to extrapolate on what is likely (and how likely it is) to go wrong in the next several years with those cars
- Jake reported that Tesla has been in touch and not only acknowledged the reported issues but, refreshingly, already addressed most of them; CR plans to issue a refreshed report by the end of the year; Jake expects this new report will again add the Model 3 to their "recommended" list, which in turn will attract them a lot of hate mail from the shorts
- Why the average of the 17 individual reliability categories is so different to the overall reliability score: the score for each category is determined by the percentage of reported occurrences of that fault compared to the number of cars for which they get feedback; a 4% fault rate (not sure if that's the actual value they employ) would attract a bad rating, but a 1% wouldn't, the car would get a green circle in that category; however, if most 2018 models only get around 0.2% faults for that issue, the car would stack up poorly in the overall rating; moreover, they only received feedback on about 500 cars, which they admit is a fairly small sample size
- Many of the suggestions the podcast interviewer made on improving the way these numbers are reported made Jake Fisher think and admit they could improve or change how they do things, which tells me that, first, their models are somewhat brittle, and second, CR are really not used to so much granular analysis of their reports and the feedback they got on this particular report was a bit of a shock to them; they agreed to make some serious changes across the board on how they do the numbers
- On the question of why the results of the report were disseminated ahead of time, allowing some parties to effectively trade on insider information, Jake was again surprised by how things played out, saying that they regularly send these reports to news organisations under an embargo to allow them to publish the news at the same time; they will talk to their legal team to see if that needs to change as well in the future.
The overall feeling I got is that the way the report was received was a shock to them, and they feel they were criticised unfairly. The press did run with the "below average" rating to claim that CR says the car is unreliable, which is not what they claimed, but it still attracted them a lot of hate from Model 3 owners and supporters of the company. They say they're only reporting the numbers they receive, but they will now try to provide more context in their reports and emphasise the individual issues they find rather than just giving an overall nondescript number rating.
The podcast is definitely worth a listen if you have the time, it runs at about 42 minutes.