Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Discussion of statistical analysis of vehicle fires as it relates to Model S

This site may earn commission on affiliate links.
I see 1 major flaw in your analysis, all the similar ones out there and even Elon's reasoning. (Note: not trying to be hostile here, this is just my 2 cents).

You are all talking about car fires, and how many happen per year, etc. as if all those scenarios were the same. But I do not think they are.

The statistics we have mainly captures fires that are a result of a major crash, a faulty fuel line, some mechanical faliure due to maintenance, build quality/materials or under-specced parts. I am not sure this is the right comparison as we haven't seen that with Tesla so far. Maybe the 2nd fire falls under that category, but the consensus seems to be, that pretty much any car would have burst into flames in that scenario.

The Model S seems to have 1 potential weakness that presents itself in the most unusal situation. You need an enormous force concentrated on a small section of the thick bottom plate that manages to drill through it. That is a very odd situation - if it weren't, you would see a lot more of these cases. We have seen dozens of reports of huge Model S crashes, even many photos on these forums with no fire at all. Comparing these 3 cases to the general fire statistics creates the impression the Model S bursts into flames when it crashes - when that has never happened so far.

The only scenario that seems to defeat that thick housing is road debris of the sepcial kind.

Consider this: you need to
- have a large enough debris on the road that the S cannot simply drive over
- a small enough debris that the front of the S would not throw out of the way
- a heavy, strong enough debris that can provide the force (will not crumble)
so this can get stuck under the car, with the full weight of the vehicle pressing against it at high speeds for a long enough time.

Now, granted this has happened 2 times, but so far it seems the biggest issue was that the car acted like a gigantic broom on the highways: it sat about 1 inch lower than the majority of cars. Now that they updated the firmware, you could also add to the list above that you would also need to be the 1st car on the road to hit that debris - you can probably run the statistics on that with the number of cars on the road and miles driven.

And than you have the low number of cases once more: drawing conclusions from 0,015% of the Model S catching fire (3 out of 20000) is just raping statistics - in my opinion. Doing this per miles driven and comparing that to other, similarly used cars could make more sense, but no one has those numbers.

(By similarly used cars I mean A6/7, S8, M3, M5, etc. When people start comapring this to Leafs never catching fire, I start wonder if they relaize we don't have 20k Leafs going 70-100 miles an hour on highways for an extended period as they are small city cars with a very limted range at that speed. So what are the chances of them hitting debris at highway speed?).
 
i'll refer you to table 2 on page 6 over here:
http://www.nfpa.org/~/media/Files/Research/NFPA reports/Vehicles/osautomobilefires.pdf

there were 125 fatalities per year in an annual average of 5,700 collision related automobile fires from 2006-2010. that means roughly 2.1% of collision related fires had fatalities.

with tesla having only 3 collision related fires, simple binomial probabilities would tell you there's a 93.5% chance of seeing zero fatalities in 3 collision related fires if a model s is as safe (from a collision fire fatality standpoint) as an ice.

in the case of fatalities, there isn't yet enough data to make a firm conclusion.

Thanks. That's actually the conclusion i was expecting to see regarding the overall fire statistics - 2 or 3 incidents (depending how you count the "drive through a concrete barrier" episode) just seems like such a small sample size. But my grounding in statistics is only good enough to know just how much i don't know - so i'm certainly not going to argue the numbers.
 
luvb2b, what I'm seeing here is that you've thrown out 96% of incidents in order to show that the Tesla has greater risk. I don't see how throwing out 24 of every 25 incidents makes for anything other than cherry-picking your data.

well this is why i'm saying we have to be very clear in the wording. i'm not saying the tesla has greater overall risk of fires, quite the opposite.

what i'm saying is that if you compare collisions only, the model s has a greater risk of collision related fires. so this means that despite being a great vehicle, there is something specific to the collisions (the impact?) that causes model s to have more fires in that case.

you may not think that it matters, but from what i've seen in the past it does.

the most direct analogy i could give you is the case of the honda cr-v from 2002-2004 or so. there was a problem with the cr-v in the way the oil seal gasket was positioned near the exhaust. this caused an unusually high number of fires shortly after oil changes - the technician wouldn't seat the oil filter seal properly and the oil would leak onto the exhaust system and ignite. there's a huge article on this in the new york times archive.
http://www.nytimes.com/2004/10/12/business/12honda.html?_r=0

now following your reasoning, you might say cherry picking fires after oil seals wouldn't reflect on the overall safety of the honda crv. however, the fact is that the number of fires after oil changes was statistically anomalous, and it pointed to a serious problem with the oil filter seals not being seated properly, the location of the oil filter, and the exhaust system. the nhtsa did an expansive investigation on this issue and eventually honda changed how the whole thing was configured.

another analogy i could give you is from biotech. when you do drug trials, especially earlier ones, you might find that overall a drug isn't working as well as you thought. but within the group of patients, there may be a cohort (a subgroup) with certain characteristics where the drug did really well. the drug companies then try to figure out why that cohort is behaving differently.

it's the same thing here in my view. something seems clearly anomalous about these collision related fires, but that doesn't mean model s is less safe "overall".
 
you are right. the definition of "collision" is the key question here. in going through nfpa data the fires were categorized by "direct causal factor" as follows:

http://www.nfpa.org/~/media/Files/Research/NFPA reports/Vehicles/osautomobilefires.pdf


Mechanical failure or malfunction 69,100 (45%) 22 (11%) 217 (28%) $201 (38%)
Electrical failure or malfunction 35,800 (24%) 2 (1%) 113 (15%) $105 (20%)
Intentional 15,900 (10%) 23 (11%) 44 (6%) $112 (21%)
Exposure fire 8,200 (5%) 3 (1%) 14 (2%) $47 (9%)
Collision or overturn 5,700 (4%) 125 (60%) 144 (19%) $38 (7%)
Smoking materials 2,700 (2%) 8 (4%) 31 (4%) $11 (2%)

as best as i could determine, mechanical and electrical failures are not those caused by collisions. this is things like an electrical wire melting and starting a fire, or an oil filter leaking oil which catches fire. none of the other categories fit the tesla fires.

collision includes things like hitting other cars, trees, barriers etc. as far as i could determine. if someone else has the exact definitions it would help as the definition of collision is central to this conclusion.

It may well be that road debris is categorized as mechanical failure, not a collision. After all isn't this the inference made by the investigation? That the car should not fail(catch fire) via road debris. Under that finding, a recall would be issued (the problem you site) to provide a fix to the design, as opposed to a 'collision' - Under your example of statistical coverage the 2nd fire would be conferred (under 'there is a problem' scenario) to a design defect related to collision causing fires. Knowing what we know about the 2nd fire (and that's statistically huge when 3 is the total number here), including it for Tesla is even more invalid that Excluding trucks (don't trucks also engage road debris?).

As for the characterization of the lightening example I think you actually proved his point that it is of PR vanity; You state you 'don't care about being hit by lightening' - regardless of the precision of the numbers, that's the PR point he's making- you don't care because the odds are too small and if you choose to be on the road with any vehicle (taking that life risk to be a baseline), you shouldn't care about being injured in a Tesla fire- so far statistics for that event say it's a ZERO chance- whatever the lightening strike chances are, they exceed zero and you injured or killed for every one of those events (ModelS fire chances are higher if lightening strikes the car :) - never mind)
 
in the case of fatalities, there isn't yet enough data to make a firm conclusion.

I have a problem with anyone using statistics based on very small numbers (don't take it personally luvb2b) as there are way too many random factors/influencers. Firstly, I don't think it's valid to make a comparison to of 3 accidents relative to 13k car-years versus much higher numbers of ICE car-years. I don't argue with your math but when numbers go down to the 4th decimal place (or more) I think we're lost in the minutiae. Secondly, using the overall number of 3 is ignoring the detail of the accidents:

Accident 1. Was the first fire, or even the intensity of it, caused by the improper cutting open of the battery compartment and the application of water?
Accident 2. Arguably there is close to 100% chance that any car driven into a wall and a tree at those speeds would have caught fire.
Accident 3. Maybe the only one truly comparable, but it's also arguably a black swan event as the debris was so huge.

In order to make any sort of valid statistical comparison, one really needs to use comparable factors such as car weight and speed. A Model S weighing in at 4,647lbs being driven at 80mph is going to strike anything with a force of ~25 tons; AFAIK there's no vehicular concept directly comparable so one can only start substituting lighter cars at higher speeds. Without broken down data and with such small Tesla numbers concrete conclusions are going to be tough to argue; larger numbers tend to smooth out the random factors and make for more acceptable arguments.
 
It may well be that road debris is categorized as mechanical failure, not a collision.

It may be relevant to note that the cars did not fail immediately upon encountering debris, but they told the driver to pull over and gave them several minutes of driving time before doing so. If this is the case, the car was still drivable after "colliding" with debris, and later "mechanically/electrically failed" as a result of the debris, then all of a sudden we have a different classification and a much lower risk, as expected. How many of these ICE "mechanical failures" happened because the car hit something and continued driving, only to have the damage catch up with them miles down the road? Is that covered in the data set, is there any place in the document which describes the precise definition of collision vs. mechanical failure?

Again, just as the data set itself states at the end, as I quoted above, trying to dissect these statistics when you do not have specific breakdowns is folly.
 
The only scenario that seems to defeat that thick housing is road debris of the sepcial kind.

Consider this: you need to
- have a large enough debris on the road that the S cannot simply drive over
- a small enough debris that the front of the S would not throw out of the way
- a heavy, strong enough debris that can provide the force (will not crumble)
so this can get stuck under the car, with the full weight of the vehicle pressing against it at high speeds for a long enough time.

[/I]

and additionally- you would have to distinguish the ICE situation to allocate a flaw of design- namely this unique situation can also catch ICE on fire (rule lines or tank); Less chance perhaps due to surface area exposed, but also much more deadly (no time to react- more explosive in nature)
 
I unfortunately can’t follow the math, but here is someone who appears to be able to. According to this post:

What do you do for work? Just passing time waiting for updates from Tesla :) - Page 36 (Post #355.)

…he is a senior researcher at CERN.

In another thread he posted the following. As I understand it, it’s an expansion (?) of what zeron, mrdoubleb, Tempus & Nigel has mentioned up-thread.

Have you taken this into consideration?

[Ugh. Another Model S fire - 2013-11-06 - Page 40 (Post #399) (My edit.)]

Please everyone stop [drawing erroneous conclusions from (My edit.)] the statistics. The ICE fires are in the thousands and their distribution is governed by the normal distribution that most of your statistics are based on. The Model S statistics are so low that it's governed by Poisson statistics and that has completely different characteristics. I deal with low probability events daily (Higgs search at LHC) and have had to handle the differences and you can't believe how much difference there is. Your math here has error bars that are so huge that you cannot draw any conclusions really. In Poisson statistics 0-2 events are statistically inseparable so even if you expect 0 events and observe 2 you cannot claim disparity between the two measurements. With three you start to get somewhere, but only if you really expected 0 in the first place. If you expect even one (or worse ca 3), then one to about six events are fully compatible (or one to ten). You can start using your normal statistics when the number of incidents expected is largish i.e. my statistics teacher used to say that 30 and infinity are about the same, it's not quite that simple, but around that region the Poisson starts to converge towards the normal distribution...

so overall I'd have to do some more complex math and not going to do this from my iPad in bed, but three or one fires make no statistical difference at this point. They do however make a world of difference to public perception especially due to nearness in time. Physics is full of freaky occurrences where unlikely events happen at start and are averaged out over time. We almost claimed discovery in 2011 of a new particle when events started to pop up at high mass with a subtantial gap to anything expected. We expected ca 0.1 events and saw 3 in a very short timeframe all together. Papers were written and taistics were debated as it was borderline close to discovery threshold. For safety it was conceded that a fourth event would lock this down hard so the papers etc were held ready and a special priority analysis was run almost live on new data daily, some people didn't sleep for a week as this was big (fundamental physics changing big). Int he end the event didn't come. After a couple of weeks we went from red alarm to orange to yellow to green as background expectations caught up and we went from 99.9% probability down to 95% to 68% and dropped further. Statistical fluctuations happen, but nature takes care of it over time...

[Ugh. Another Model S fire - 2013-11-06 - Page 87 (Post #868) (My edit.)]

My point on stop [drawing erroneous conclusions from (My edit.)] the statistics was that claiming that three fires is very huge news and significant in comparison to ICE fires is an absurd statement because so low event count systems require one to properly compute the confidence intervals and those are not necessarily always deducible "logically". If you talk about 3000 vs 6000 fires (i.e. population is 2000 times higher), then that's far more likely to be significant, but in this case we went from 2 fires to 3 fires and everyone and their uncle went bonkers that this is happening so much and it's more relevant than the ICE fires etc etc.

Yes you can compare the various distributions, but you have to take into account the error bars. That's what I wanted people to fathom especially as the error bars on the 3 events are pretty large and therefore we cannot claim that we have a significant deviation from mean expectation where the mean expectation is taken from high statistics of ICE miles driven and fires (and therefore I'd assume with relatively low error bars). Our assumptions and results are fully dominated by the observed number of events and its inherent low count at this point.

So I'm not arguing with you, just making sure everyone understands that in statistics most of the time you cannot ignore the errors that are involved if you want to make any claims. If you have huge huge statistics (billions of miles driven by millions of cars and thousands of fires), then you can maybe work with just mean expectation values and ignore the errors as they'll be relatively small, but in a 20k car 3 fire event situation you cannot ignore them.
 
Interesting read, but I keep wondering about this very small sample of only 3 events. What are the odds that we will see similar events (large metal objects puncturing the battery) over the coming months and years, and at a frequency that resembles what we've seen this fall. Even the vehicle sample of 19,000 could be too small to be statistically relevant.

My guess is that by increasing the clearance in the firmware, Tesla greatly reduced the probability of a fire.

Now you need to run over a larger, but similarly shaped object, to pierce the battery.

In the second fire, there was another car in front of the Model S. That car went over the trailer hitch just fine, it had enough clearance.

Now when MS has larger clearance, its chances on catching a trailer hitch will be similar to other cars. If there's another comparable car in front of a MS, it will catch the metal object first. If the object is small enough and the car in the front does not catch it, the MS will not catch it neither.

I still think that Tesla will make an improvement (reinforce the car's underbelly), but it needs time to execute this with the least expenses. I believe that the new firmware effectively buys them this time.
 
I mostly agree with your post in that Elon has significantly exaggerated his statistics, however, there is still too little data on Model S fires to do any apples to apples comparisons. Three fires in a short period of time out of a sample set of 19,000 with an average lifetime of 6 months can't be compared to a hundred thousand plus fires out of several hundred million vehicles. It just doesn't work. The number of occurrences isn't statistically significant. In fact, it makes more sense that the Model S has an even lower fire rate and this was just a fluke because there were no fires for over a year and then several in rapid succession. It will take a few more years, or many more fires to say anything definitively.
 
it may turn out to be wrong later, but the fact is right now the null hypothesis is rejected at a confidence level above 99%.
Not really. You're badly misusing confidence intervals.
i respect your view though, that the number of observations is small. but you're looking at the 3, the number of trials here is large 13,300 tesla model s years of road experience is a large number.
This isn't a normal distribution. Bluntly, your statistical analysis is no good. Neither is Elon's.

You can treat it as a Poisson distribution, and find that you have no confidence that this is different from the null hypothesis. Or you can do a Bayesian analysis, and find that your posteriors are determined almost entirely by your priors. In short, there's no meaningful statistical information from three incidents.

There was a serious problem with the Boeing 737 which led to a substantial number of crashes under wind shear conditions. It was only possible to identify the nature of the problem after there had been, IIRC, over a hundred such incidents. It was only possible to be sure that it was a real problem after, IIRC, 30 or so.

Anyway, staying away from the misuse of statistics, and instead looking at physics, it seems that the fires happen under very specific circumstances only, of undercarriage collision with a heavy metal object which acts as a lever. (Or if you're a crazy drunk who drives over a wall, in the Mexican incident.)

Under those circumstances, in ICE cars, from what research I've done, it seems that the collision often kills the driver, though I haven't managed to find a meaningful sample or population of such events to look at. It also seems that it often causes a gas tank rupture.

---
Unlike Mario I'm not professionally applying statistics at CERN. My mother, however, was a professor of statistics, and spent an awful lot of time teaching me about misuse of statistics. Statistics is a counterintuitive subject.
 
well this is why i'm saying we have to be very clear in the wording. i'm not saying the tesla has greater overall risk of fires, quite the opposite.

what i'm saying is that if you compare collisions only, the model s has a greater risk of collision related fires. so this means that despite being a great vehicle, there is something specific to the collisions (the impact?) that causes model s to have more fires in that case.

and I think that's the point your statistical conclusion fails. To conclude there is a design-related issue, you have to 'compare collisions only'. So now the type of collision must be in the denominator.
i.e. How many of the fire 2 type collisions cause a fire in ModS vs ICE; How many of the fire1,3 type collisions cause fire in ModS vs ICE
 
It may be relevant to note that the cars did not fail immediately upon encountering debris, but they told the driver to pull over and gave them several minutes of driving time before doing so. If this is the case, the car was still drivable after "colliding" with debris, and later "mechanically/electrically failed" as a result of the debris, then all of a sudden we have a different classification and a much lower risk, as expected.
What's particularly interesting about this to me, in physics terms, is the question of whether the cars (in Accidents 1 & 3, not the crazy Mexican collision) would have caught fire at all had the driver pulled over and stopped instantly. We don't actually know the cause of the fire. Was it mere exposure of battery internals to air... or was it due to continuing to run power through the battery pack *after* the collision?
 
(Or if you're a crazy drunk who drives over a wall, in the Mexican incident.)

Through a wall.

Anyway, given that everyone has shown how the statistical analysis in this post is not correct, luvb2b, I think that you should delete it, or edit it to correct that it was wrong, so that people don't get further misinformed reading it.

- - - Updated - - -

I have posted this in another forum (and for some reason it's pending moderation).
In your regression you have to account for age of cars, clearly newer cars tend to catch on fire a lot less vs. older cars.

And I have posted this in another thread, and for some reason you didn't respond to it (maybe it has to do with you being a new member and all your posts being fearmongering? hmm). In your assertion you have to show statistics, instead of continuing to make stuff up, clearly the things you are saying are made up. If you can cite something to back yourself up then please do so.

If it's so important to account for age of cars, then do it. You haven't. I won't expect you to.
 
Last edited:
it may turn out to be wrong later, but the fact is right now the null hypothesis is rejected at a confidence level above 99%.

I stopped believing you understood statistics at all after reading this. I realize that sounds mean, but honestly... I am still picking my jaw up off the floor.

As has been said over and over again, you can't compare such a small sample with such a large one. I think Nigel really hit the nail on the head, with all three events not being related! You could make a strong argument that the only fire caused by road debris to total the entire car, was in the third in accident. The first could have been totally fine until the fire department vented the top. The second was obviously not road debris, but an extraordinary event that should have killed the driver. This really only leaves us with an N of 1.
 
...
Now, granted this has happened 2 times, but so far it seems the biggest issue was that the car acted like a gigantic broom on the highways: it sat about 1 inch lower than the majority of cars. Now that they updated the firmware, you could also add to the list above that you would also need to be the 1st car on the road to hit that debris - you can probably run the statistics on that with the number of cars on the road and miles driven.
...

Exactly.
 
Through a wall.

Anyway, given that everyone has shown how the statistical analysis in this post is not correct, luvb2b, I think that you should delete it, or edit it to correct that it was wrong, so that people don't get further misinformed reading it.

- - - Updated - - -



And I have posted this in another thread, and for some reason you didn't respond to it (maybe it has to do with you being a new member and all your posts being fearmongering? hmm). In your assertion you have to show statistics, instead of continuing to make stuff up, clearly the things you are saying are made up. If you can cite something to back yourself up then please do so. I won't expect you to.

And i have responded with stats. That's the post that does not show up. Basically i will try to redo it here. In 1997 there are 377k fires in 2010 184,500. Miles driven went up 20%, so fires/mile driven is down about 3x. The only reason i could think of is safer cars.
 
you are right. the definition of "collision" is the key question here. in going through nfpa data the fires were categorized by "direct causal factor" as follows:

http://www.nfpa.org/~/media/Files/Research/NFPA reports/Vehicles/osautomobilefires.pdf


Mechanical failure or malfunction 69,100 (45%) 22 (11%) 217 (28%) $201 (38%)
Electrical failure or malfunction 35,800 (24%) 2 (1%) 113 (15%) $105 (20%)
Intentional 15,900 (10%) 23 (11%) 44 (6%) $112 (21%)
Exposure fire 8,200 (5%) 3 (1%) 14 (2%) $47 (9%)
Collision or overturn 5,700 (4%) 125 (60%) 144 (19%) $38 (7%)
Smoking materials 2,700 (2%) 8 (4%) 31 (4%) $11 (2%)

as best as i could determine, mechanical and electrical failures are not those caused by collisions. this is things like an electrical wire melting and starting a fire, or an oil filter leaking oil which catches fire. none of the other categories fit the tesla fires.

collision includes things like hitting other cars, trees, barriers etc. as far as i could determine. if someone else has the exact definitions it would help as the definition of collision is central to this conclusion.

luvb2b - as you've looked at the data, I'm curious if you've seen any data source which separates collisions (two cars running into each other for instance) from accidents involving vehicles hitting debris in the road.

I ask as the way I think about the 3 fires, 1 of the events was such an outlier that I have a hard time making much of anything from it. But the first and 3rd seem to have a similar causal factor - hitting debris in the road that punctures the bottom of the car, and leads to a fire. The difficulty is that if you define your Model S sample that precisely, can we also define an ICE sample with a similar causal factor - ICE car hits debris on road and catches fire?

If the data doesn't exist at that level of detail, then we're left to do what we can, with what we have.
 
I believe it is fair to say that MS's undercarriage is susceptible to damage by road debris. This damage can be expensive. This damage can cause a fire. This damage has, to date, always allowed the car to warn the occupant(s) and provide them with ample time to safely stop and exit the vehicle.

The NTSB will only affect action if a defect is found that poses a safety threat. There is zero evidence that MS poses a safety threat so, barring an incident with injury, I do not see a recall. This is a PR problem and an cost of ownership problem.

It is also sensational which guarantees it a news cycle (but only a single news cycle).

From a statistics point of view, I expect one MS per month to now run over a trailer hitch and catch fire..... I think not.