Consumer Reports is wrong about future reliability

[AmpedRealtor]

CR itself said that if Tesla were rated as a brand - which they are not because they only have one product currently - then it would rank between Acura and GMC. That puts it above Infiniti, Cadillac, and Mercedes Benz.

 

[jesselivenomore]

Elon Musk on Twitter: echoing exactly what I have been talking about.

 

[DrManhattan]

Consumer Reports is not a very good ratings website. They are able to document some past problems with things, but they cannot predict future reliability. Their ratings are not balanced enough to differentiate between real problems (drive unit failures) and minor issues (creaking noises). I'm not sure why anyone, let alone media outlets give them any attention. When is the last time anyone has ever used consumer reports? Are you also under 65?

 

[cwerdna]

Consumer Reports is not a very good ratings website.

What's a better one for car reliability?

When is the last time anyone has ever used consumer reports? Are you also under 65?

I do. I subscribe to their print magazine and web site. I read some of it. I most definitely look at their car reliability ratings as a screener for cars I'm considering. I'm under 65.

I've found their reliability results to be reasonably accurate for cars that my parents and I have owned. I also have found them to often correlate w/known problem areas on other cars that I've owned vs. popular web sites for them (e.g. my350z.com, priuschat.com, maxima.org, mynissanleaf.com). Are their results perfect? No.

Sometimes, we can't really tell what's behind the rating. For example, when I last looked, I'd seen a trend of Priuses once hitting a certain age (usually when their 12 volt goes bad and requires replacement) getting dinged w/electrical system bad ratings. Should be it electrical? Unfortunately, the symptoms on a Prius of a dying/dead 12 volt are NOT the same as a regular ICEV. On a Prius, you will NEVER hear a starter cranking too slowly or not being able to turn the ICE because the ICE isn't started by the 12 volt and it has no dedicated starter motor.

Instead, weird stuff happens usually accompanied by intermittent no start conditions (unable to put car in to READY mode). So, if the Prius owner doesn't know this (they would if they're on Priuschat paying attention), they might think they have an electrical prob and take it to a repair shop/dealer.

IIRC, once the wave of dying 12 volts passes (usually ~4 to 6 year mark), electrical system reliability looks fine again.

I have no issues with CR's testing methodology since they have experts testing the vehicles on a broad spectrum of metrics that are beneficial to the consumer. But their annual reliability survey is NOT a statistically accurate sampling. When they say they surveyed 1400 Model S owners, those owners are CR subscribers who chose to participate in the survey. For it to be statistically valid, it must be a random sampling; therefore, the reliability survey is really most accurate to the type of individuals who would subscribe to CR and like to take the survey.

Ok, so what other reliability survey of cars is statistically valid in your mind and has much more than 1400 responses?

And that is my point, it is poor analysis. It is barely analysis at all. They do not break down reliability by model year, instead lumping them all together.

They do give reliability ratings for each model year.

These guys did another reliability test and found exactly what I have been saying: While Teslas are less reliable than typical new cars

Full stop! You've just stated it right there.

So instead of like CR is extrapolating, reliability is getting better as Tesla matures as a manufacturer, and as Teslas technology matures.

You're extrapolating as well. How do you know this? Reliability does NOT always improve for a given model or automaker. CR has shown this time and time again.

Common sense for most people, but something the people at CR can't seem to grasp.

No, you're trying to interject something else into the results that were given.

All 3 model years are early generation products. Even 2016 models are early generation, Tesla has JUST started making cars.

Yes, Model S was the first car they built from the ground up. But, they did have some help from the Mercedes parts catalog.

There was the Roadster which first shipped in early 08.

The question is has the reliability and build quality been improving?

I'm not sure I can answer that. Their survey is to readers of what happened in the past 12 months for THEIR car. The averages (already pointed to earlier) get worse as cars get older. They're comparing the problem rates to the averages for those years.

Unless this survey can break down its data by model years, it does nothing to answer that question.

They do.

CR's brain dead prediction of the same 40% below average forever.

They don't do that. They had a different prediction last year. Whatever that number was (don't have it handy), it worked out to average.

And, if there were reliability ratings the year before, they had another one. In a year from now, you'll see a different one.

 

[jesselivenomore]

Full stop! You've just stated it right there.

I never disputed CR's data or their survey results, Sherlock. How many times do I have to repeat, CR is welcome to take Tesla off their recommend list based on their criteria. I never said that Teslas did not have "below average reliability" going by CR's definition. Read the title of the thread, it is their FUTURE reliability prediction that I have a problem with. Their prediction makes no sense because it is based on how every other car manufacturer operates, manufacturers that have been in business for decades if not 100 years.

You're extrapolating as well. How do you know this? Reliability does NOT always improve for a given model or automaker. CR has shown this time and time again.

You are right, I am extrapolating. I don't have a problem with CR extrapolating, just that their extrapolations don't make sense.

My extrapolation is based on common sense: a young manufacturer inexperienced with the process of making a brand new technology will most likely get better at manufacturing in time with more experience. Now, that may not turn out to be true, and I may be wrong. Any prediction is susceptible to being wrong, but at least mine are based on logic. And so far, what I have said has been verified by an independent source:

Contradicting Consumer Reports, survey says Tesla reliability is improving - San Jose Mercury News

And by Elon Musk himself: Elon Musk on Twitter:

Meanwhile, CR's prediction is based on ill-logic: SINCE when most 100 year old manufacturers making a product that has been refined for 100 years makes an unreliable product, they are likely to make the same unreliable product the next year, THEREFORE, Tesla a 3 year old manufacturer making a product that is 3 years new will follow the same pattern.

That simply does not make sense. And in reality, already proven false, as Elon stated: new models have improved reliability compared to early ones.

Full stop.

 

[artsci]

I have a major problem with the reliability of CR's data. Unless it's based on a true random sample of owners (highly unlikely) and weighted back to represent key characteristics of owners (age, income, gender, etc) the data can't be trusted.

In my professional life I supervise I lot of very sophisticated survey research. From what I know about CRs sample and study design if a member of our research staff brought this research to me I'd throw it in the trash. Not to say that a standard questionnaire designed to measure the reliability of ICE cars needs to be totally revised for an electric car that has new technologies and standards.

 

[cwerdna]

My extrapolation is based on common sense: a young manufacturer inexperienced with the process of making a brand new technology will most likely get better at manufacturing in time with more experience.

Yes, but other automakers aren't standing still either. They have a lot more experience and some of them actually really seem to care. Look up kaizen.

Now, that may not turn out to be true, and I may be wrong. Any prediction is susceptible to being wrong, but at least mine are based on logic.

So, therefore CR is "wrong"?

And so far, what I have said has been verified by an independent source:

Contradicting Consumer Reports, survey says Tesla reliability is improving - San Jose Mercury News

From the above "But the car is still three to four times less reliable than the typical new vehicle, said Michael Karesh, the statistician who runs the TrueDelta website." That's definitely well below average.

BTW, from this in Oct 2013 Tesla Model S Recommended | Car Reliability - Consumer Reports News :

Based on data collected from more than 600 owners in our 2013 Annual Auto Survey, the Tesla Model S earned an average predicted reliability score. Owners of the 2012 model reported very few problems, although 2013 owners cited quite a few more. The combined score allows us to recommend it.
...

Reliability was the final piece necessary to earn our recommendation.

The reported problems were not on the sophisticated electronic systems, powerful motors, or giant battery pack, as one might expect. Instead, it was the smaller details that caused some minor issues, such as wind noise, squeaks and rattles, and body hardware (including the sunroof, doors, and locks). These mechanical items expose one of the myriad challenges in competing with massive, long-running corporations. For Tesla, everything is new and almost every part is developed and fit together for the first time in modest volume.

It seems that Tesla got what could be considered the tough stuff right initially; no owner in our survey has reported problems with the electric drivetrain.

From Oct 2014 Tesla Model S Has Average Reliability - Consumer Reports News :

Now, with the data from our latest annual reliability survey, we found that other owners have had service experiences similar to ours. That survey netted a sample size of 1,353 Model Ss. Comparing the Model S with all other cars the same age, our analysis pegged its predicted reliability as “average.” That’s good enough for Consumer Reports to continue to recommend it.

Most of the reported problems related to slow-responding retractable door handles and some body-component issues. Our car, for example, had a creak emanating from the base of the windshield, which Tesla fixed for us under warranty. Early 2012 examples also suffered some driveline issues, which Tesla proactively rectified.

In the grand scheme of things, an average score, while not impressive, isn’t terrible either. Lots of high-end cars have an average score, including, for instance, the Acura RLX. A few, largely from Lexus and Audi, are outstanding, but many, such as the Mercedes-Benz S-Class and Cadillac XTS are far worse.

And now to 2015 Tesla Reliability Doesn’t Match Its High Performance - Consumer Reports :

As part of our Annual Auto Reliability Survey, we received about 1,400 survey responses from Model S owners who chronicled an array of detailed and complicated maladies. From that data we forecast that owning that Tesla is likely to involve a worse-than-average overall problem rate. That’s a step down from last year’s “average” prediction for the Model S. It also means the Model S does not receive Consumer Reports’ recommended designation. (To be recommended, a vehicle has to meet stringent testing, reliability, and safety standards, including having average or better predicted reliability.)

The main problem areas involved the drivetrain, power equipment, charging equipment, giant iPad-like center console, and body and sunroof squeaks, rattles, and leaks.

Specific areas that scored worse on the 2015 model, compared with the 2014 model in last year’s survey, were the climate control, steering, and suspension systems. Complaints about the drive system have also increased as the cars have aged—specifically for the 2013 model, which was the car’s first full model year. (See more details on the Tesla Model S model page.)

But those problems mostly still fall under Tesla’s four-year/50,000-mile bumper-to-bumper warranty (and eight-year/unlimited mileage battery and drivetrain warranty), so they are generally being corrected at no cost to owners.
...
The list of issues also includes more significant problems, which could be pricey to fix once out of warranty. Based on survey responses, Tesla has made a habit of replacing the car’s electric motors. The brake rotors tend to warp. And the door handles often fail to “present” themselves as drivers approach their cars.

Those automatically retractable door handles are a signature element of Tesla vehicles, but when they don’t work they effectively lock drivers out.
(... and so on about the 97% owner satisfaction, good customer service, etc.)

From just looking at the quotes, can you see why it might be difficult to infer improvement? As I mentioned in the 350Z example at Consumer Reports reliability of Model S - worse than average - Page 14, you can have the phenomenon in CR ratings of problem areas trickling thru the model years until the part or whole car is redesigned.

 

[Freudianly]

Only a few people in Fremont know the true reliability statistics of the whole fleet of Model S and how they evolve in real time. The real problem is that today more than ever before, capitalism is based on secrecy (not that communism was any more transparent). So however much you love or worship Elon, he will not share those stats with you. His tweet answering CR is incredibly short compared to their yearly stats. Does he have a lot to hide, or is he right to be that confident? We cannot tell for sure.
The only way to find out it to have better stats than CR by setting up a new website entirely dedicated to Tesla reliability stats, and to attract many more than 1,400 Model S owners.

 

[Al Sherman]

Interesting info from Consumer Reports. Tesla was below average for reliability but scored a 97 out of 100 for owner satisfaction. The next closest sedan was an Audi A7 which scored an 84 for owner satisfaction and was actual 30+ above average for reliability. So... Tesla owners are still more satisfied with their below average reliability cars compared to Audio owners.

This. ^^^
I still love my car and the company. It still amazes me. The entire process from ordering to now has been one of the most satisfying experiences of my life. Was everything perfect? Of course not. When something has been wrong Tesla has come to the house, picked up my car and dropped off a loaner, and fixed my car. Who else does that?
There's still nothing even remotely comparable to my "classic" and Tesla is still moving forward.
I'm actually thankful to have been able to drive this car on a daily basis.

 

[CHGolferJim]

I know that the stock price dropped and people generally seem anxious about CR's report. However, I can say as a prospective buyer, it doesn't make me any less interested in the car. In fact, watching the CR video gives me a pretty good feeling about it. If there are some issues, but the car still has a 97% satisfaction rate, then it's probably not a decision I'll regret. That said, I agree that reliability will have to improve over time if Tesla is going to have success selling a mass-market car and it sounds like that is happening. The only thing I'm torn over is whether to get the pano roof. I'd be really irritated by a leak, but it looks so cool...

+1........placed my order on Wednesday......with pano

 

[caps04]

It's not that big of a deal. They removed iPhone from the recommendation list too due to Antennagate but it hardly had any impact in its sales. And that iPhone still had over 90% customer satisfaction rating.

 

[SR22pilot]

Wow. I just skimmed the thread. Some posters remind me of the strategy of doubt as seen in "Merchants of Doubt." It's easy to throw doubt on something. It is difficult to come up with an alternative methodology. Employees would come into my office and say "This plan is lousy." I would say "Ok, so what should we do instead." I would get "I don't know but this is lousy." Consumer Reports does their reliability survey the way they do because it is the best way they have come up with that isn't prohibitively expensive. It is a reader survey with all the selection bias that drags along with it.

Based on what I read on this forum I would suspect that there are a number of reliability issues. It's a great car but from a young company. Musk says there are design changes made weekly to the Model S. If Tesla is really working the reliability Pareto chart then all this will work out in the end. Model S sales are increasing and will swamp out the early cars. If the car is indeed getting more reliable this will show up. Hey, people can always get a subscription to CR and take the survey.

 

[artsci]

Wow. I just skimmed the thread. Some posters remind me of the strategy of doubt as seen in "Merchants of Doubt." It's easy to throw doubt on something. It is difficult to come up with an alternative methodology. Employees would come into my office and say "This plan is lousy." I would say "Ok, so what should we do instead." I would get "I don't know but this is lousy." Consumer Reports does their reliability survey the way they do because it is the best way they have come up with that isn't prohibitively expensive. It is a reader survey with all the selection bias that drags along with it.

Based on what I read on this forum I would suspect that there are a number of reliability issues. It's a great car but from a young company. Musk says there are design changes made weekly to the Model S. If Tesla is really working the reliability Pareto chart then all this will work out in the end. Model S sales are increasing and will swamp out the early cars. If the car is indeed getting more reliable this will show up. Hey, people can always get a subscription to CR and take the survey.

With all due respect, this is just reductive thinking: CR can't come up with a better way to the research so that makes the findings acceptable. I could recommend a reliable way to the research but it would cost way more than CR would be willing to spend. So they've settled on the best methods they can envision and afford. And they may be better than other methods for this kind of relaibiablity survey. But that doesn't mean they're reilable or that we should trust them. In fact we shouldn't.

I've had very few problems with my car. Should I conclude that the car is problem free? No. And there's no reliable evidence one way or the other from CR or anybody else. Only Tesla knows and they aren't saying. I think the bigger question is how well and quickly does Tesla satisfy owner problems. On that score the anecdotal evidence (and it's only anecdotal) is very well.

 

[bob_p]

Tesla continues to have market leading owner loyalty - which is probably more important than the reliability ratings.

Even though my "classic" P85 has required several component replacements - I'm still taking the car into service less frequently than my previous cars. And even after factoring in the cost of the pre-paid service and extended warranty, I believe I'm coming out ahead on the cost of service.

Many of the issues with the early model cars like mine have likely been addressed by improvements in design and manufacturing. And unlike the other manufacturers, Tesla is continuously making changes - not waiting for "model years".

So how CR calculates reliability is probably not going to be a good predictor of reliability for new Tesla vehicles (Model S and X) - since Tesla is able to more rapidly make corrections to reliability issues - and get those corrections into production vehicles than any other manufacturer.

If I decide to purchase a new Tesla - I would not be concerned about hardware reliability. Though when we do decide to purchase our next Tesla - I hope they've addressed their software issues - catching up on their growing backlog of promised-but-not-delivered or requested features...

 

[BoerumHill]

With all due respect, this is just reductive thinking: CR can't come up with a better way to the research so that makes the findings acceptable. I could recommend a reliable way to the research but it would cost way more than CR would be willing to spend. So they've settled on the best methods they can envision and afford. And they may be better than other methods for this kind of relaibiablity survey. But that doesn't mean they're reilable or that we should trust them. In fact we shouldn't.

I've had very few problems with my car. Should I conclude that the car is problem free? No. And there's no reliable evidence one way or the other from CR or anybody else. Only Tesla knows and they aren't saying. I think the bigger question is how well and quickly does Tesla satisfy owner problems. On that score the anecdotal evidence (and it's only anecdotal) is very well.

Right - my understanding is the entire rating is based on surveys of 1400 Model S owners. It's an annual rating; last year, when the predicted reliability was average, it was based on 650 surveys.

I think the video explaining the rating is very even handed.

Tesla Reliability Lags Its High Performance | Consumer Reports - YouTube

They point an array of issues which resulted in the downgraded rating. They are VERY quick to point out several mitigating points:

- Issues are resolved quickly & effectively by Tesla, which is noted by owners in "virtually every survey response."
- Almost all of these issues are under warranty & result in no out of pocket expense to the owners.
- Tesla Model S has a higher overall satisfaction rating than ANY ORHER CAR tested.
- 97% of Model S owners would buy the same car again.

I doubt there is another vehicle rated unreliable that garners this much praise from its affected owners. But that doesn't make for a very good headline, does it?