Discussion of statistical analysis of vehicle fires as it relates to Model S

[kenliles]

ken, may i ask, are you really able to follow what i mean? even my head hurts now.

yep- that's the perfect detail of explanation I needed and thanks very much for putting up with my persistence; breaking into tailed binomial is what I needed for it to click (and for other readers, I did bother to check much of his work from this post- spot on);
I'm much more familiar with wavelets and similar predictive math/correlation and that probably places in my psyche a higher confidence level of what these statistics can mean in reality, and they're ability to accurately predict events.
Now it may be that conditions will change due to new higher clearance (or not), but I'm confident now of your analysis- it may not translate to a problem that even needs much fixing given the the overall superior safety profile- but it does characterize Tesla's EV differently. I think Tesla could address it with education largely, but this will take time.


For anyone interested, my long investment will remain en-tacked, but I will be moving some of my J15 LEAPS out to J16 (just playing the odds- I think most of this is rung out of the price, but inevitable follow on fires, for whatever reason, will revisit the issue and dampen the stock for a while- still see it mostly rising significantly from here over the next 6-12 months- just my 2c)

thanks ever so very much luvb2b

 

[SFOTurtle]

are you going to tell me you've never made an investment decision from a set of imperfect data? give me a break!

you have no other analysis to present. and you don't want to believe what i've posted, which fyi is now being affirmed directly by mit and at least indirectly by the nhtsa.

I don't see that your conclusion, which is that from a statistical "analysis" the Tesla MS is much more likely to catch fire in a "collision" than an ICE car, is affirmed by the MIT person. In fact, as Omarsultan pointed out, given the extremely limited number of Tesla fires (and he incorrectly includes the Mexico crash without including any other numbers of Mexican fires, let alone the other reasons for discounting or eliminating the Mexican event from the analysis), he all but concludes just the opposite:

"It’s also probably too early to make the reverse claim—that the Model S is more likely to catch on fire–based on the numbers I give above."

With respect to the NHTSA, what do you mean that the NHTSA has "at least indirectly" affirmed your "analysis"? What exactly has the NHTSA indirectly affirmed, and how have they indirectly affirmed it?

 

[TSLAopt]

just to be clear, the 5,020 figure is an estimate i derived for 2012 - not an nfpa number. i posted the derivation in the first post. the mit article that lola just posted has a similar discussion:
Early Data Suggests Collision-Caused Fires are More Frequent in the Tesla Model S than Conventional Cars | MIT Technology Review

regarding selling tesla stock, well that's a judgement of the stock's value, and whether or not it has already discounted the collision fire issue. i think judging by the response to today's bloomberg / mit articles you can see the smart investors already knew what i've been trying to explain.

had i gotten on this earlier there were probably a good 40-50 points to be made after that second fire was reported (because as i see now, even 2 fires was enough). and possibly the information would have saved a bunch of people on the board a bunch of dollars.

well good to see you again. i'll check back in a bit later to see if there's anything new here but i think this horse as been beaten pretty hard for now. time to start getting back to vacationing!

Luvb2b
Are you long OR short TSLA currently?
just curious.

 

[kenliles]

Luvb2b
Are you long OR short TSLA currently?
just curious.

Not to speak for him, but just to provide an answer,
he's posted in his thread- he's currently neither and doesn't see a substantial long or short opportunity right here

 

[SCW-Greg]

Personal opinion: this thread has gotten painfully tedious. FCOL people, you're dissecting 2 freak events (and throwing in a drunk driving incident in a foreign country for good measure) then extrapolating all that to compare it to a high number situation in totally different circumstances. BTW, the accident in Mexico would only count if you included Mexican ICE statistics in all your calculations and how many Model "car-years" have been driven in Mexico? My gosh, it's probably less than 1 car-year and 1 fire occurred, that must mean there's a 100% chance of Model S catching fire in Mexico!

The only true and certain findings are:

1. The Model S battery pack can be pierced if hit by a heavy steell object with a force of ~25 tons.
2. A battery pack compromised by said metal object could catch fire after the fact.

If we say that the battery pack is the potential weak point of Model S, then we also have to admit that gasoline and oil are the potential weak point of all ICEs. In fact we have a whole thread on how dangerous gas stations are. You do yourselves a disservice trying to compare those 2 freak accidents to the thousands of fires in ICE cars which undoubtedly all had different causes and many of which caused lasting injuries or even death to the vehicle occupants.

[/rant]

^THIS!!!

This whole debate is baseless!
We're trying to apply the probably of a coin flip, based on two accidents, which has already been shown to be a number too low to make any reliable trend assumptions. And all the while pontificate adnauseum over it in order to A) rationalize the math we try to apply and/or B) make ourselves look like some sort of credible professor on the subject and hope the Pied Piper effect will take hold.

Sorry... I've sooo moved on.

 

[sub]

I stopped reading after a few pages. First off, it's confirmed to me every day when I visit TMC that there are some brilliant people here with the same beliefs/passion/hopes for the future that I have. I think time is the only factor that will settle this debate. I've never encountered road debris that I feel would have damaged a Model S and I live in a rural area. Odds are that this will become a very rare occurrence that won't make the news 2 years from now just like ice fires don't make national news. These fires have made the news for one reason, and it isn't the safety of the Model S.

its almost a bonus to have your car burn up, get a new updated model for free!

 

[Johan]

How the people on the ground gather the data that goes into the study is what matters.
Even if the first responders were told to gather such data ( and I don't believe they are ) they have no method or inclination to do so.
They don't do it.

This! This is a better way of saying what I was trying to say above. I too think the quality behind the data in the big control group is just not good enough/precise enough to be making any good comparisons. We just don't know enough about the method of collection and the definitions used etc.

 

[lolachampcar]

sub,

Be careful for the bonus you wish for. As I interpret it, Tesla will fix your car under warranty should it catch fire, not replace it. That sounds like months out of service on top of the repair of significant damage.

 

[tray loader]

if model s true probability is on the low side of my confidence interval, we'll see have a 90% chance of seeing the next fire by 10,820+22,781 = 33,601 total car-years, at or before the end of 2014q3.
if model s true probability is at the sample mean of my confidence interval, we'll see have a 90% chance of seeing the next fire by 10,820+8,295 = 19,115 total car-years, at or before the end of 2014q1.
if model s true probability is on the high side of my confidence interval, we'll see have a 90% chance of seeing the next fire by 10,820+2,850 = 13,670 total car-years, before or near the start 2014q1.

if you put the gun to my head and said, "just shut up and tell me when already!", i'll say the models predict you'll see another collision-fire before the end of 2014q1.

ken, may i ask, are you really able to follow what i mean? even my head hurts now.

Man, I do hope Your statistics are wrong, because otherwise....
Thank You for breaking this down and sheding some light on this highly complex and yet subejctively interpretated topic.

 

[luvb2b]

I don't see that your conclusion, which is that from a statistical "analysis" the Tesla MS is much more likely to catch fire in a "collision" than an ICE car, is affirmed by the MIT person. In fact, as Omarsultan pointed out, given the extremely limited number of Tesla fires (and he incorrectly includes the Mexico crash without including any other numbers of Mexican fires, let alone the other reasons for discounting or eliminating the Mexican event from the analysis), he all but concludes just the opposite:

"It’s also probably too early to make the reverse claim—that the Model S is more likely to catch on fire–based on the numbers I give above."

With respect to the NHTSA, what do you mean that the NHTSA has "at least indirectly" affirmed your "analysis"? What exactly has the NHTSA indirectly affirmed, and how have they indirectly affirmed it?

the mit review guy didn't run the numbers, if he did he would have seen the same thing i did.

with respect to the nhtsa, i am referring to this quote:
http://www.technologyreview.com/vie...-are-more-frequent-in-the-tesla-model-s-than/
"Not if you’re only looking at fires caused by collisions. And that’s what the NHTSA wants to know–whether the Model S is particularly vulnerable to catching on fire when it collides with something."

they're seeing the same thing i am, that the number of collision fires are well outside predicted boundaries.

 

[ntam]

In the report from nfpa.org, it also states that (page 6):

http://www.nfpa.org/~/media/Files/Research/NFPA reports/Vehicles/osautomobilefires.pdf

"While collisions or overturns were factors in only 4% of the automobile fires, these incidents caused
three of every five (60%) automobile fire deaths."

So while it might be true that collision with object hitting the bottom of the Model S (not collision in general with large objects or other vehicles) might be more likely to result in a fire. Model S does not have any death resulted from these fires. Since most people should agree that death is worse than having your car catch fire after an accident. In my opinion, Model S is better than an ICE car in terms of safety.

It would be a more meaningful analysis if luvb2b can do an statistical analysis of how safe the Model S is by comparing the probability of collision that resulted in deaths rather than focusing on collision that resulted in car fires.

 

[NigelM]

they're seeing the same thing i am, that the number of collision fires are well outside predicted boundaries.

My wife and daughter both left the house wearing black and white this morning, every woman in the country must own black and white clothes so based on my sample I predict that every woman in the country will be wearing black and white today. Ooops, a freak occurrence will always be outside predicted boundaries.

 

[BillHamp]

Has anyone seen this: Musk Claim of Fewer Tesla Fires Questioned in MIT Report - Bloomberg

It says a few things:
1. MIT did an analysis and says Musk's stats are wrong. He is not justified in claiming the MS has a lower fire risk than other vehicles. Neither is it justified to say that the MS has a higher fire risk.
2. The NHTSA has said that they don't believe electric cars, in general, have a higher fire risk that ICE, but that certain models may have higher risk. In other words, just as with ICE cars, some EVs have issues that others do not.
3. The National Fire Protection Association says that "We should not be surprised to see additional vehicles fires with new technology.” They also said that we shouldn't forget about the risks that ICE cars pose.
4. Center for Auto Safety says Musk is making a mistake in emphasizing the lack of injury associated with these fires as you don't need injury to prove a defect and have a recall.

All in all, this might be the most balanced article I have seen.

I think this (Early Data Suggests Collision-Caused Fires are More Frequent in the Tesla Model S than Conventional Cars | MIT Technology Review) is the original article from MIT. Interesting read.

 

[Mario Kadastik]

I think point #1 is what I was trying to make at around page #41 or smth With two or three fires there just is no statistical way to claim either. The car could be far far safer or far far worse, but at this level of statistics both hypothesis are consistent with observation and therefore no conclusion can be drawn.

 

[JST]

Has anyone seen this: Musk Claim of Fewer Tesla Fires Questioned in MIT Report - Bloomberg

It says a few things:
1. MIT did an analysis and says Musk's stats are wrong. He is not justified in claiming the MS has a lower fire risk than other vehicles. Neither is it justified to say that the MS has a higher fire risk.
2. The NHTSA has said that they don't believe electric cars, in general, have a higher fire risk that ICE, but that certain models may have higher risk. In other words, just as with ICE cars, some EVs have issues that others do not.
3. The National Fire Protection Association says that "We should not be surprised to see additional vehicles fires with new technology.” They also said that we shouldn't forget about the risks that ICE cars pose.
4. Center for Auto Safety says Musk is making a mistake in emphasizing the lack of injury associated with these fires as you don't need injury to prove a defect and have a recall.

All in all, this might be the most balanced article I have seen.

I think this (Early Data Suggests Collision-Caused Fires are More Frequent in the Tesla Model S than Conventional Cars | MIT Technology Review) is the original article from MIT. Interesting read.

I read the MIT piece, and I take the point--given the data we have so far, the MS looks like a higher fire in collision risk than the total vehicle fleet. The first and biggest issue with all of these claims, whether made by Elon or MIT (as others who know more about math than I do have pointed out), is that the dataset that we have for MSs is so small that drawing conclusions is useless. It's like taking a presidential poll with a margin of error of 100 percent; literally ANY conclusion you draw is flawed. EDIT--I see Mario posted while I was typing this, making this same point, which I've seen him make elsewhere.

Moreover, I'm not sure that it's right to just compare risk of fire in collisions. A non-collision related fire is, IMHO, potentially a much bigger deal than a collision related fire, because it suggests a design defect that may be more troubling than a collision-related fire. Any object filled with flammable liquid (like gasoline or the Tesla electrolyte) subjected to enough kinetic force may catch fire; an object that catches fire WITHOUT kinetic force is more worrisome. And before you object and say that at least people aren't inside in a non-collision related fire, think about how many cars are parked in, around, and underneath occupied dwellings.

But even that doesn't really help, because we're in some sense comparing two totally distinct universes of things that have different constituent components. A "Fire" means one thing in a gas car. It may mean something else in a Tesla. If Tesla-fires are generally less catastrophic than ICE fires (which is a big if, but certainly possible), then comparing fire frequency is totally meaningless.

I realize that Musk and Tesla are the ones that started this whole "let's compare fire frequency" thing. It was dumb to do that, though I understand the instinct. I hope that everyone can get away from it as quickly as possible, because at this point it's just a logical and statistical mess from which no clear answers can be drawn.

 

[Gtoffo]

the mit review guy didn't run the numbers, if he did he would have seen the same thing i did.

with respect to the nhtsa, i am referring to this quote:
http://www.technologyreview.com/vie...-are-more-frequent-in-the-tesla-model-s-than/
"Not if you’re only looking at fires caused by collisions. And that’s what the NHTSA wants to know–whether the Model S is particularly vulnerable to catching on fire when it collides with something."

they're seeing the same thing i am, that the number of collision fires are well outside predicted boundaries.

Am I the only one who thinks all this is irrelevant?

The premise of this analysis is wrong IMHO.

The question is not if following a CERTAIN type of collision the car has a higher probability of catching fire. In the event of a collision (any kind) the NHTSA (or anyone else for that matter) doesn't care if the car is totaled. They only need to evaluate if the car has adequate security measures in place to guarantee a certain level of safety. So what if the car burns down 10 minutes after it has been parked safely at the side of the road? Was the fire under control? Did it enter the passenger cabin? Did it place the driver/passenger in danger?
The Model S could have 50 times more chances of catching fire following such an extreme/rare event (battery puncture, at high speed, by metal debris of the exact height and strength, which has only been encountered by cars higher than the model S) but if the security measures in place (firewalls, user warnings, etc) worked correctly then the car DOES NOT pose a security risk.

I also think it is wrong to discount car fires not caused by collisions. When Elon said that a Model S was LESS LIKELY to catch fire I'm sure he was talking about the risk of fire in general. Cars catch fire for many reasons and given its electric powertrain a Model S will naturally be affected more by certain kind of events, and less by other kind of problems. WHy should that be a problem? It is an obvious consequence of having a different architecture. If "70% of car fires are from mechanical or electrical malfunction" then should we assume that all non Model S vehicles should be recalled because the chances of a mechanical/electrical malfunction causing a fire are 100% higher than a Model S? Of course not, the two key questions are: were those fires caused by random/unpredictable events or by structural/design problems/defects? AND did those fires place the vehicle operators at unnecessary risk?

Was the cause of the Model S fire a collision? YES. Was the operator/passenger ever at risk? This is what the NHTSA will be determining, but from the sequence of events I have read so far (car warns driver, car stops gradually, a lot of time until the fire breaks out, fire breaks out gradually and only in the front of the car, fire never reaches the cabin, passenger exit is not obstructed), I would tend to say NO.

Nothing to see here.....move along....move along....

 

[Krugerrand]

Am I the only one who thinks all this is irrelevant?

You most definitely are not alone. I have an image in my head of a pack of dogs fighting over a bone that's already been chewed to death, making it (in case my point is missed) a useless bone. But hey, some dogs just can't help themselves and have to be carrying *something*, *anything* in their mouths to be happy.

 

[ModelS8794]

But even that doesn't really help, because we're in some sense comparing two totally distinct universes of things that have different constituent components. A "Fire" means one thing in a gas car. It may mean something else in a Tesla. If Tesla-fires are generally less catastrophic than ICE fires (which is a big if, but certainly possible), then comparing fire frequency is totally meaningless.

I realize that Musk and Tesla are the ones that started this whole "let's compare fire frequency" thing. It was dumb to do that, though I understand the instinct. I hope that everyone can get away from it as quickly as possible, because at this point it's just a logical and statistical mess from which no clear answers can be drawn.

+1, and something I have been thinking but unable to articulate well. If it turns out that Tesla'a battery placement makes the car more susceptible than an ICE in the category "undercarriage strikes by heavy metal objects with 25 tons of force, resulting in a fire," what are we to make of that conclusion? If it's 10x more likely to result in a fire, but the fire is contained due to Tesla's battery module and firewall engineering and poses no threat to passengers due to the Model S warnings systems, is that a better outcome or worse outcome than the behavior of an ICE car getting struck with the same object at the same force, resulting in no fire but the object penetrates into the passenger cabin instead? Is that a safety defect, or a safety feature?

(I am weak-minded and know nothing of first principles, so I rely on analogy more than Musk does. My apologies)

Imagine someone manufactured a car that was 100% impregnable from passenger cabin penetration due to a miraculous new material coating the cabin that repels all force regardless of how powerful. The only side effect: The crash-proofing material used caused the car to be positively charged and exactly 3x more likely to be struck by lightening than another car.

Car maker boldly asserts - FIRST CRASH-PROOF CAR EVER MADE, CHANCE OF INJURY IS ALMOST ZERO. (never say never, right?)

A year later, 2 freak storms result in 3 lightening strikes that hit 3 of the cars, frying the electrical systems and scaring the bejeezus out of the drivers, but otherwise causing no harm. The cars are totaled as a result.

Is the car more dangerous than other cars which are much much less likely to be struck by lightening?

Is the safety claim legitimate?

Should a buyer of the car worry about lightening strikes, or should they take comfort in the safety of the vehicle? Should the NHTSA worry about lightening strikes, or focus on the safety of the vehicle?

 

[luvb2b]

Man, I do hope Your statistics are wrong, because otherwise....
Thank You for breaking this down and sheding some light on this highly complex and yet subejctively interpretated topic.

lol. well you're welcome. i do feel like i did the work a bit late to be of much use from an investing standpoint, but such is life.

i've read through the rest of the thread, and i want to leave with a couple parting thoughts.

cars that hit small amounts of debris, and then *much* later have a mechanical fire, i think people are right those won't be classified as collisions - they will be classified as mechanical failures. but the teslas did not hit a small amount of debris. i would venture that most anyone who hit something of that size and had their car catch fire will go into the collision grouping. also remember that the nfpa data includes all vehicles, including very old ones which are many times more likely to catch fire than new ones. so while there may be a few misclassifications, it's unlikely that very many incidents like hitting large objects are going to get categorized improperly in the data set. the nfpa has been doing this work for 30 years, during which time i am sure they have improved the quality of collection. for those who say on the ground collection is poor, i say fine. make some adjustments then. triple the number of collision-fires, make an adjustment for old vehicles being included, and then you'll find even then that the tesla collision-fire per car-year probability confidence interval is above the ice.

everyone who has complained about data issues has made no effort to present anything alternative. this is an investing thread. the smart money doesn't make investment decisions by throwing up their hands, saying "this sucks and can't be done." so you're entitled to your opinion about all the flaws and defects in the data, and you're also entitled to be parted from your precious capital by not attempting to come to any investible answer.

to nigel - with all due respect as moderator - your first comment was that an analysis can't be done. i showed that it can be done, detailed two different generally accepted procedures on how to do it, and had results confirmed in posts by a phd, an md, and a couple other people on the board. the remainder of your comments added nothing to the discussion and seem kind of smart-alecky. if you don't know how to analyze this kind of data, i think you should just accept that and step aside.

to you and other people who have think you can't analyze 3 fires, i'll show you something interesting by analyzing zero fires. it will prove a different point.

we'll use my prior estimate of the number of total automobile fires from the first post: 125,500.
now note that 69% of automobile fires per year are due to mechanical and electrical failure: 69% x 125,500, or 86,595 fires per year due to mechanical & electrical failures.

dividing by 128.1 million cars we get .000676 for the probability of an ice having an electrical or mechanical fire in a car-year of operation (that's 1 per 1,479 car-years).

as before, i would argue that the .000676 could be treated as a point estimate because it will have very small standard deviation due to the millions of cars that went into the sample.

so far model s have had zero electrical or mechanical fires in 10,820 car-years of operation.

we can calculate a two tailed 95% confidence interval in this case for model s, using the same approach i used before in the post linked below.

but you can construct a two-tailed distribution with the binomial distribution. the upper and lower bounds of a two-tailed 95% confidence interval will be defined by the answers to these questions:

given a binomial distribution of collision-fires, what is the probability p such that i would have a 97.5% chance of observing 3 or fewer car fires in a sample of 10,820 car-years of experience?
given a binomial distribution of collision-fires, what is the probability p such that i would have a 2.5% chance of observing 3 or fewer car fires in a sample of 10,820 car-years of experience?

these questions can be answered quite easily, you can just use goal seek with excel and this formula:
=binomdist(3,10820,<< insert estimate here >>,true)

of course the numbers have to be modified, this time it's:
given a binomial distribution of mechanical/electrical-fires, what is the probability p such that i would have a 98.5% chance of observing 0 or fewer car fires in a sample of 10,820 car-years of experience?
given a binomial distribution of mechanical/electrical-fires, what is the probability p such that i would have a 0.5% chance of observing 0 or fewer car fires in a sample of 10,820 car-years of experience?

we do =binomdist(0,10820, << insert estimate here>>, true) and goal seek to 0.025 and 0.975.

this will give you a 95% confidence interval for the probability of model s having a mechanical or electrical fire in a car-year of [.0000023, .000341]

now this 95% confidence interval is cleanly below the .000676 estimate for ices. (note: even the 99% confidence interval will be cleanly below ices)

so from the data - which is **zero** fires - i can say with 95% (even 99%) confidence that model s is safer than an ice in terms of the probability of having an electrical or mechanical fire. i had said as much in the first post, and it seemed no one argued with that claim, which of course was based on ** zero ** (non-collision) fires.

so a tesla favorable claim made with zero observations is welcome, and a tesla-negative claim based on 2 or 3 fires is derided by many.

if you can't be objective, should you really be investing?

- - - Updated - - -

I think point #1 is what I was trying to make at around page #41 or smth With two or three fires there just is no statistical way to claim either. The car could be far far safer or far far worse, but at this level of statistics both hypothesis are consistent with observation and therefore no conclusion can be drawn.

mario, if you are going to say it can't be done, please go through the method i presented a few pages back and show me what's wrong with that analysis. everyone is saying "it can't be done" and yet no one can poke holes in how it's done.

 

[BillHamp]

I think point #1 is what I was trying to make at around page #41 or smth With two or three fires there just is no statistical way to claim either. The car could be far far safer or far far worse, but at this level of statistics both hypothesis are consistent with observation and therefore no conclusion can be drawn.

Yes and no. It seemed that the article was saying that the stats, while tentative and not conclusive, don't favor Elon's interpretation in the very limited case of collisions resulting in fire. I agree with you that we are bordering on statistical significance, but that isn't the only thing that matters. The NHTSA has found design flaws in similar circumstances and the idea that they would wait for the problem to reach statistical significance can be both good and bad. Good in the sense that they don't want to waste time and money on flukes. Bad in the sense that they shouldn't wait for more serious problems (potential injury) before investigating. It is a fine balance that the NHTSA is trying to strike.

The hypotheses are not equally consistent with observation. One carries more weight than the other. Could you publish a hard line in a journal and be taken seriously? No. Could you publish your tentative findings and be taken seriously? Yes. I come from medicine and epidemiology, which deal with difficult stats like this all of the time. Finding "cancer clusters" and dealing with rare adverse drug events requires analysis of very limited data. That does not mean, however, that an adverse event isn't there or that a cluster doesn't exists. It only means that finding very specific things in a broad field of issues can be difficult. In the case of the Tesla, it isn't that the car is unsafe overall, it is that it may have a very specific weakness that is a serious problem and won't come out until many more cars suffer similar damage. Depending on how you look at the stats, they start to become significant and I think it the difference in how you and I approach the issue that determines how we decide how serious it is. Neither one of us is wrong, we just see it from different perspectives. I hope you can see it from mine as I try to see it from yours.