I was troubled as many of us by the CR rating of Model S. So I dug deep into the methodology of CR. This is what I found:
1. They collect the data from some 4 million subscribers which result in a huge 'n' which is essential for any statistical analysis. Out of the ~ 750000 responses 1400 were from Tesla owners. It is not a great but good enough.
2. CR employ qualified statisticians to crunch the numbers and so I assume their methods are scientific.
3. They divide the questions into 17 categories like fuel system, engine major (engine rebuild or replacement), engine minor (oil leak), climate control etc.
4. Engine-major, cooling system, transmission-major, and driveline are weighted more heavily because they are more expensive and the car is more likely to be out of service. This is one of the main reason why model S performed badly in CR rating.
Changing the engine in a ICE car is a different ball game than changing it in Model S which Tesla does seamlessly and painlessly with a loaner car. This is not taken into account by CR in their rating but explains the high customer satisfaction. It will be interesting to see if Tesla can maintain that level of service with increasing fleet.
4. The most puzzling of the report was the predicted reliability. Model S reliability was rated good in previous 3 years but suddenly crashed to below average in 2016 - which grabbed all the headlines.
Predicted reliability is based on the previous 3 years data. To quote from CR website - "The Predicted Reliability, also called New Car Prediction, forecasts how well a new model likely to hold up based on its recent history" "For this Rating, we average a model's Used Car Verdict for the newest three years" "Over the years, we have found that several years of data are a better predictor than the most recent model year alone". "One or two years of data may be used if the model was redesigned within that three-year time frame"
I am not sure this even applies to Model S. CR doesn't take it to account continuous changes Tesla makes like the dual motor and constant updates. The "below average" rating was based on earlier years data. I am guessing we have to wait couple of years to see the effects of the current model S reliability in the CR rating. I am not sure Tesla will even redesign model S in few years as they are constantly updating it - unlike other ICE car manufacturers do - which I think is nothing but a sales gimmick. May be this is another new paradigm change Elon wants to make to auto industry.
5. I haven't seen the data on engine replacement for current dual motor model S compared to earlier cars and it will be interesting to see those numbers even in a informal setting like this forum in a separate thread.
6. Many of the CR's 17 categories like exhaust, fuel systems and emissions doesn't even apply to cars like Model S. I am wondering if CR have to design a separate questionnaire for electric cars.
7. CR claims "In order to earn a
CR Recommendation, a model needs to meet three criteria:
- The model needs to do well in our road tests.
- The model must have at least average Predicted Reliability.
- If the model was crash-tested by the National Highway Traffic Safety Administration (NHTSA) or the Insurance Institute for Highway Safety (IIHS), it must perform at least adequately. In addition, pickups and SUVs must not have tipped up in the government's rollover test."
Model S was outstanding (out of the world, really) in 2 of the 3.
To another question "
How can a car be recommended one year but not the next?
Our subscribers tell us about their experiences each year. In each Annual Questionnaire, we ask subscribers to
ConsumerReports.org and
Consumer Reports magazine about problems that they had experienced in the preceding 12 months. As their reports change from one year to the next, our Predicted Reliability scores sometimes change as well."
This is in contradiction to their earlier claim about predicted reliability that it is based on data from newest 3years.
8. Tesla should have more accurate data on the improvement in reliability especially of the engine. They should publish those and take CR head on. That's of course if the problems were fixed.
In conclusion I wouldn't put too much weight on CR rating. But Tesla needs to turn this CR rating around before the model 3 release as many of the target customers will be CR believers.