Discussion of statistical analysis of vehicle fires as it relates to Model S

[renim]

>LUVB2B
Firstly, its erroneous statistics to include Mexican EVs but exclude Mexican ICEs statistics in the analysis. Either adjust the ICE stats to include Mexican ICEs or adjust the EV stats to exclude Mexican EVs. Mexico is a Nissan country and shares many vehicles with USA, its wrong statistics to mix developed and developing world accident safety stats, even if the vehicles are identical.

>General
Fleet on road (all ages) is very relevant for regulatory bodies, particulary in regard to recalls. Elon is very correct to use on-road fleet stats in that context. Ie IF Tesla Model S has a lower fire rate than half the vehicles on the roads, then that half of vehicles on the roads should be recalled before Tesla would be recalled. If its 95% etc...

VMT, vehicle miles travelled seems to be how NHTSA would analysis this, so safety per 100M VMT is a key metric. http://www-nrd.nhtsa.dot.gov/Pubs/811701.pdf http://www-nrd.nhtsa.dot.gov/Pubs/811845.pdf As Tesla model S passed the 100M VMT a level of analaysis can be observed, the Tesla is very safe for its occupants. No fatalities have been reported, and while 'injury' has different definitions, No injuries have been reported where par value would expect around 80 injuries.

 

[kenliles]

1. yes, i am confident. not confident enough to be short, but confident enough to stay on the sidelines. i guess the main reason is this description on page 6 of this document: http://www.nfpa.org/~/media/Files/Research/NFPA reports/Vehicles/osautomobilefires.pdf

What are the leading causes of automobile fires?
Some type of a mechanical failure or malfunction was a factor on almost half (45%) of automobile
fires and 11% of the associated deaths. Mechanical failures may be due to leaks or breaks, worn out
parts, backfires, or similar issues. Electrical failures or malfunctions were factors in one-quarter
(24%) of the fires, but only 1% of the deaths.

you can see mechanical failures defined as basically something faulty on the vehicle, not the vehicle striking something.

2. intentional fires mean a fire where someone actually "set" the car on fire, this is the case of arson. a drunk driver crashing his car at high speed is not intentional. Here's the quote from the same page of the document above.

Ten percent of automobile fires were intentional; these incidents caused 11% of the deaths.
Intentional fires are excluded from the remainder of the analysis of causal factors. Because the
NFIRS field “cause of ignition,” includes unintentional, equipment or heat source failure, and act of
nature as separate code choices, the term “non-intentional” will be used to describe all fires that were
not intentionally set.

3. 13,300 car years being large or small is irrelevant. the binomial distribution properly incorporates this value of n in calculating probabilities. you could have 1,000 car years and 1 fire and the binomial distribution will tell you how likely you are to observe that for a given "p". it's so hard to explain this to someone who doesn't have a firm grasp of how it works.

here's another example. let's say you're interested in rolling all 1s on six fair dice. the odds of a single 1 is 1 in 6. the odds of doing 6 ones is (1/6)^6 = .00002143. that's the "p" and notice it is very close to the .0000392 probability of a collision fire.

now if you ask me the question in 1000 attempts at rolling six 1s on six fair dice, how likely is it that i will do it once? twice? three times? i can answer all of those questions *** precisely ***.

would you tell me in this case that 1000 attempts is not enough to answer the question? or that just doing it once or twice is too few times for me to answer the question? of course not. because you know that the probabilities can be exactly computed.

now understand that the only variable here is the "p", because the 13,300 car-years and the 3 fires are pretty much known exactly. we could debate the p being too high or too low, and you can adjust it yourself and run it through the binomial distribution using the excel formula i provided before. i say the "p" being based on millions of car-years is going to be very, very accurate.

4. as a model s owner, i'm only worried about a fire if i am in a collision. as an investor, i take the view that this is something the nhtsa will justifiably probe, in my view evidence is that something is likely is amiss, and sometime down the road it will require a fix. it may be the stock price has properly discounted all of that happening already, or perhaps not yet. am i freaking out about this because i own a model s? no.

i did all this research trying to decide whether or not to wade back into tesla, or whether or not to short tesla. since i've decided to do neither (aside from a intraday-trading position), i figured i might as well share it because it seems like many people don't quite understand how these numbers work.

got it- thanks for that explanation regarding the 'n' factor of baseline - yes, you're right, after a sufficient number that factor's size would not be meaningful

I remain skeptical on the sub-catorization of collision for ICE (to include road debris producing mechanical/electrical failure)- but for the moment will grant that. However that sub-catorization of ICE fires is made cognitively to attempt to match 100% of ModS 3 fire causes. Under that same assumption then, you would have to believe that 33% of all ModS fires going forward will be caused by slamming through a concrete wall at excessive speed. This seems to indicate a qualitative skewing produced by insufficient observations OR an insufficient comparative of the ICE sub-catorization.

On the other hand- you could argue that you didn't make enough sub-catorzation of ICE which would produce and even worse ModS result as follows:
In the same document (http://www.nfpa.org/~/media/Files/Research/NFPA%20reports/Vehicles/osautomobilefires.pdf) page 5
Only 17% of ICE fires are Highway related (the rest on residential, parking, commercial, etc). Whereas seemingly the ModS problem, if it exists, is only on the highway (or in Mexico at highway speeds in front of a concrete wall) and if high speed impact of road debris is the primary(only?) way to produce a ModS fire, then maybe the fix and comparison to ICE is a non(or easy fix)-issue.

I seriously doubt after all the crash testing results, anyone will include data or fixes for suicidal conversations with a concrete wall (and the ModS thwarted the injury/death part- nice job Elon and crew!)-
by the way, a little disingenuous though to point out the definition of 'Intentional' category doesn't work for that case. Yes- and I knew that of course - as you say 'a drunk driver crashing into a wall at high speed is not Intentional'. The point in calling it such was to illustrated the flaw of it's inclusion as an observation count- which concurrently requires publishing of the accurate statistic with the conclusions: That 33% of ModS Collision fires in your observation model of 3 fires (clearly characterized as precise and known as accurate - and that's certainly true) are caused by ModS slamming into a wall at high speed- with that caveat in your analysis, my observation is:
'maybe we need more observation'

thanks for you're hard work and post on this by the way- very much appreciated (fully meant)

 

[NigelM]

Re Fire #2: So I have to ask ... where's the analysis of 'how many people survive after driving an ICE car into a concrete wall at 30/40/50/60 mph'?? Shouldn't there also be an analysis of that? The fact that the car burned is immaterial. I believe (but I haven't looked at the stats) that most other cars would not have protected the occupants.

Can someone start an analysis of survivability of that type of situation? It wasn't road debris. It was a concrete wall and speed. But I'd really like to understand how the Model S did vs. ICE in that situation.

Who here is up to running THAT analysis?

Well I think B nailed it.

I'd just add that I'd also like to see a thread on the statistical probably of an ICE car being hit by heavy metal debris with a force of 25 tons actually asking the driver to please pull over carefully and exit the vehicle.

True anecdote from today: a friend who drives an ICE called me today and jokingly asks me if I have bought shares in a fire extinguisher company; I asked him back if he'd read the reports and he said yes so I then said "If you hit a metal tow hitch at 80mph on the Interstate would you rather be in your car or in a Model S?" His answer, "Oh no question, I'd rather be in a Tesla". Case closed.

 

[kenliles]

Well I think B nailed it.

I'd just add that I'd also like to see a thread on the statistical probably of an ICE car being hit by heavy metal debris with a force of 25 tons actually asking the driver to please pull over carefully and exit the vehicle.

True anecdote from today: a friend who drives an ICE called me today and jokingly asks me if I have bought shares in a fire extinguisher company; I asked him back if he'd read the reports and he said yes so I then said "If you hit a metal tow hitch at 80mph on the Interstate would you rather be in your car or in a Model S?" His answer, "Oh no question, I'd rather be in a Tesla". Case closed.

well said; and was in process of similar post around the notion of:
How much time does an ICE-Fire (punn intended unfortunately) give you to find out the result of the exampled ModS collisions?

 

[luvb2b]

I'm going to have to disagree, and I can't let this pass, as you seem to be building a short case, and you want people to think your thoughts are valid thoughts (they are not, for many reasons).

You are attempting to teach statistics using probability. We know the probability of rolling dice combinations a priori. We do not know the likelihood of a running over a trailer hitch such that it causes a fire in a Model S. The n of your number of occurrences will become large enough around 10. So your attempt to use probability, and all of your calculations are flawed.

yes it does seem like people are fearing a "short case". i've never said anything about valuation in light of the fires. the stock has already lost $9 billion of market value from the highs, and i think around $5-6 billion since the 3rd fire. that's a huge haircut. has the stock properly discounted the fires and a problem? possibly it has. or possibly not. that is a question of valuation, which is distinct from the statistical question i am addressing.

i mentioned in my post that precise wording is very important. the statistical test i presented has nothing to do with the probability of a tesla hitting a trailer hitch.

here's a concise statement of the hypothesis being tested:

null hypothesis: the risk of a collision related fire in a tesla model s is no greater than the risk of a collision in the average ice automobile.

my analysis shows that at the 97.5% confidence level, we can reject the null hypothesis. that is, the risk of a collision related fire in a model s is almost surely greater the risk of a collision related fire in an ice automobile.

i am not attempting to estimate the risk of a collision related fire in a model s - that would have substantial variation (+/-50% or more) with a sample size of 3. what i a doing is a statistical comparison of means, showing that it's highly unlikely that the mean of the rate of tesla collision fires is less than the mean of the rate of ice collision fires.

i hope that makes sense, and i hope someone else with a sound statistical background will jump in the discussion.

 

[EnergyMax]

yes it does seem like people are fearing a "short case". i've never said anything about valuation in light of the fires. the stock has already lost $9 billion of market value from the highs, and i think around $5-6 billion since the 3rd fire. that's a huge haircut. has the stock properly discounted the fires and a problem? possibly it has. or possibly not. that is a question of valuation, which is distinct from the statistical question i am addressing.

i mentioned in my post that precise wording is very important. the statistical test i presented has nothing to do with the probability of a tesla hitting a trailer hitch.

here's a concise statement of the hypothesis being tested:

null hypothesis: the risk of a collision related fire in a tesla model s is no greater than the risk of a collision in the average ice automobile.

my analysis shows that at the 97.5% confidence level, we can reject the null hypothesis. that is, the risk of a collision related fire in a model s is almost surely greater the risk of a collision related fire in an ice automobile.

i am not attempting to estimate the risk of a collision related fire in a model s - that would have substantial variation (+/-50% or more) with a sample size of 3. what i a doing is a statistical comparison of means, showing that it's highly unlikely that the mean of the rate of tesla collision fires is less than the mean of the rate of ice collision fires.

i hope that makes sense, and i hope someone else with a sound statistical background will jump in the discussion.

If you're asking about collision related fires, it seems you're using the wrong words as people traditionally use them. We've seen metallic debris fires in two cases. There are many photos of Model S collisions with other cars that have not resulted in a fire. Your words insinuate that collisions (which are relatively common) are relatively likely to result in a fire.

Much less common than a collision is a trailer hitch on a highway which gets run over. This is very rare, and your analysis does not take this into account.

How many cars run over trailer hitches per year? How many injuries? How much damage? These are the questions.

What was even the likelihood of two Model Ss running over such types of metal? We are way out there in improbabilities, I suspect. It could be a decade before another Model S runs over a trailer hitch. Collisions with other cars have yet to produce a fire.

 

[stopcrazypp]

null hypothesis: the risk of a collision related fire in a tesla model s is no greater than the risk of a collision in the average ice automobile.

The problem is if that the right comparison to make (as you can pick from many different types). From a safety standpoint, the driver cares about if the car will catch on fire after they hit debris and that null hypothesis does not tell you the answer (as it mixes in probability of collision in the first place which is affected by other factors, some of which are driver choices, like speed and following distance).

The null hypothesis that does is this: the risk of fire in Tesla Model S collisions is no greater than the risk of fire in the average ICE car collision.

And we don't have the data to support this without knowing the collision rate of Model S (currently the assumption is it's the same as ICE vehicles, which may not be a good assumption).

 

[kenliles]

my analysis shows that at the 97.5% confidence level, we can reject the null hypothesis. that is, the risk of a collision related fire in a model s is almost surely greater the risk of a collision related fire in an ice automobile.

i am not attempting to estimate the risk of a collision related fire in a model s - that would have substantial variation (+/-50% or more) with a sample size of 3. what i a doing is a statistical comparison of means, showing that it's highly unlikely that the mean of the rate of tesla collision fires is less than the mean of the rate of ice collision fires.

i hope that makes sense, and i hope someone else with a sound statistical background will jump in the discussion.

fair enough-
how do you square that with your engineering knowledge of the ModS (in your previous heavy long position, is this something you would not have predicted for example),
assumed Tesla testing results, years of Roadster data (not a design close enough for example?),
results of collision and other testing performed by safety labs, etc.- (they didn't test to real world cases sufficient to pickup on this perhaps?)?

I'm curious why your certainty of this statistical analysis exceeds your certainty of a long stock position, exceeds your belief in Elon and Co to represent same, while concurrently failing to exceed personal safety concerns (via your own ModS experience, fires don't equate to safety perhaps?)?

I think what I hear is that there's enough certainty to warrant no position short or long in stock, but not enough to prevent purchase (or retention) of the car. If everyone followed that course though, you'd have to be long since sales would not be effected

 

[Grendal]

The problem I have with this whole thread is the fact that we are again trying to make direct comparisons to ICE cars. It's such an easy trap to fall into.

The fundamental statistic that applies to safety is the fact that there have been no fatalities and no serious injuries in a Model S. That's it. The car is safe.

I don't care if the battery pack catches on fire statistically more than an ICE catches on fire. You can pick and choose the parameters of your statistics and make the Model S seem better and you can make it seem worse.

How about incidents of hitting large metal objects while travelling at high speed. Has that parameter been broken down statistically?

Because it was a separate incident. Do we have the statistics on cars travelling at high speed going through walls and slamming into trees?

The Model S might, and I emphasize the word might, be more prone to a fire when their battery pack is punctured. That does not make the car unsafe. Current facts indicate that in spite of the fires the car is safe since everyone has walked away safely.

Statistically an older gas car becomes more prone to fires as their fuel lines weaken, oil begins to leak, and gaskets begin to wear. Will a Model S get more dangerous over time? Probably not. It should remain exactly the same as when it was first bought. So statistically over the life of the car the Model S is probably equal or better than your average gas/diesel car.

See my point. Where do you put the parameters of what you are trying to decide? I guarantee you that I can focus on a certain area and make a Model S come out as statistically better than an ICE car. I can also focus on a different area (which someone against Tesla will do) and spin it so the Model S is much worse than an ICE car.

All of that said, I don't see any reason for Tesla to make improvements to avoid battery pack fires in the future.

 

[NigelM]

Luvb2b, since you're discussing "truth and lies" using statistics that appear to be controvertible while the term truth is an incontrovertible fact and lie has implications of intent to maliciously mislead, I would suggest you edit your title to something like "Discussion of statistical analysis of vehicle fires as it relates to Model S" rather then a sensationalist and provoking title such as the one you used. The title brings innuendo immediately to the forefront and stirs emotion rather than clinical discourse that could lead to useful information.

sounds good to me!

Mod Note: Done.

 

[daxz]

1. the data you site is "Automobile Fires in the U.S." the second fire was not in U.S.
2. I question the throwing out of the mechanical and electrical fire data. A lot of cars could be damaged from hitting objects and still driven only later to burst into flames and not be called a collision fire. I've work on lots of cars as a mechanic and the undersides of cars show all kinds of damage. I would also suspect that a lot a fires could be caused by objects stuck on hot exhaust systems or exhaust pushed against another part of the car.

 

[richkae]

1. yes, i am confident. not confident enough to be short, but confident enough to stay on the sidelines. i guess the main reason is this description on page 6 of this document: http://www.nfpa.org/~/media/Files/Research/NFPA reports/Vehicles/osautomobilefires.pdf

What are the leading causes of automobile fires?
Some type of a mechanical failure or malfunction was a factor on almost half (45%) of automobile
fires and 11% of the associated deaths. Mechanical failures may be due to leaks or breaks, worn out
parts, backfires, or similar issues. Electrical failures or malfunctions were factors in one-quarter
(24%) of the fires, but only 1% of the deaths.

you can see mechanical failures defined as basically something faulty on the vehicle, not the vehicle striking something.

I think you are mistaken and some fires that are caused by road debris must be included in the set of fires that are classified as mechanical failure.
While the statistical analysis is thorough, skepticism of the data quality is required.
How much investigation goes into determining this cause? If there is no obvious point of impact found ( other car, guardrail, tree, light pole ) then it goes into the "not a crash" bucket.
A minor piece of road debris, once it is combined with the twisted ash of the car - which is usually totally consumed - is probably unlikely to be noticed.
Between 2006-2010 there were 69,100 fires per year determined to be non-crash mechanical failure. How many of the "Mechanical failures may be due to leaks or breaks, worn out parts, backfires, or similar issues." are due to road debris? Even if road debris were detected, I would bet that a lot of the time it is put in the "not a collision" bucket because they are only concerned about collisions with other vehicles or blameable parties.

Since collision deductibles are typically higher than comprehensive ones ( $250 to $100 ), when asked "Did you hit something?" the average driver would say "No, it just spontaneously started on fire." It is worth their time to lie about that, but it is not worth the time of the insurance company to stiff them the $150 by spending a bunch of money to prove it was road debris ... especially since they will never be able to find who dropped the road debris to make them pay.

I think it is almost certain that some portion of those are actually caused by road debris, and since we dont know how many, much of the analysis of the stats is not useful.

If 10% or 20% of those 69,100 fires are caused by road debris then that changes all of the equations dramatically. ( There are also 35,800 fires from the electrical failure bucket. What damaged the wires? What dislodged the connector? Certainly road debris could do that also. )
In fact, I would not be surprised if the total failure from road debris is significantly higher than 20%, because it is appalling to think of all those spontaneous failures. As an engineer, I would be appalled if something i designed failed by incinerating itself when a part wore out.

Lastly, if Tesla ( or someone ) had not investigated the two fires caused by road debris, I think it is probable that they would have gone into some non-collision ( mechanical or electrical failure ) bucket. Certainly that was the accepted default, and what Tesla sought to disprove.

I would not dig too deeply into statistical analysis of data of unknown quality.

 

[omarsultan]

OK, with all of one college statistics class under my belt, I fear to venture into this head, but after the 3rd fire, I ran across the following report published by the AAA Foundation on road debris related accidents. Its 1990 data, but it would still seem to be somewhat useful. The TL;DR summary is the attribute 25,000 accidents each year to road debris with approximately 80-90 resulting fatalities. Not sure who's viewpoint this supports, but here you go: https://www.aaafoundation.org/sites/default/files/VRRD.pdf

O

 

[kenliles]

OK, with all of one college statistics class under my belt, I fear to venture into this head, but after the 3rd fire, I ran across the following report published by the AAA Foundation on road debris related accidents. Its 1990 data, but it would still seem to be somewhat useful. The TL;DR summary is the attribute 25,000 accidents each year to road debris with approximately 80-90 resulting fatalities. Not sure who's viewpoint this supports, but here you go: https://www.aaafoundation.org/sites/default/files/VRRD.pdf

O

that's interesting and closer to actual comparisons perhaps; If we knew how many of those caused ICE fires, we'd have something. It's interesting that produces a .34% death rate though

luvb2b- with your spreadsheet en-tacked is it easy to compute the predicted time for the next 'collision-enduced-fire' or better yet, how much time would have to pass without one to equal/exceed ICE moving beyond the Null hypothesis?

 

[serkol]

I think that luvb2b's calculations are fundamentally wrong, because he ignores the fact that 2 fires happened after running over a metal object that can pierce the car's underbelly. One fire happened outside of US and should be excluded from the calculations, because luvb2b compares these fires to the US data.

MS had a clearance much lower than an average car, lower than most sports cars. So if there's a trailer hitch on the road, it's possible that 100 cars will drive over it before MS finally catches it with it's battery.

MS was much more likely to catch a piece of road debris than an average car.

If you compare MS fires to all other car fires resulting from hitting road debris, MS was probably more likely to catch fire than an average car.

But because there were no other types of MS fires in US, it's save to say that in any other circumstances (collisions or whatever) MS is much less likely to catch fire.

Now when the last firmware increases the clearance, MS has the same chances of catching road debris as most other cars. So now it will have much less collisions with debris.

What does this mean? From now on MS will have much less chances to catch fire than an average car.

 

[NigelM]

am i allowed to argue with a moderator? there is a very standard accepted statistical methodology for conducting this kind of analysis, which is comparison of means testing. i took a slight shortcut because the number of ice car-years is so high that it doesn't require the full rigor of comparison means testing. so i don't know why you feel it's not valid, there's entire chapters written on how to do it.

Anyone can discuss with a moderator. ;-)

You didn't quote the final sentence of my post:

.....larger numbers tend to smooth out the random factors and make for more acceptable arguments.

The point is that you are measuring events at the extreme and assuming they are scale-able. I'm not disagreeing with your methodology, just with the appropriateness of using it at all. The Model S accidents all had outside factors which are not being evened out through volume data as is happening with the ICE sample.

 

[richkae]

MS was much more likely to catch a piece of road debris than an average car.

If you compare MS fires to all other car fires resulting from hitting road debris, MS was probably more likely to catch fire than an average car.

from my previous post, I do not believe the data supports such conclusions. There is no way to discern how many of the ICE fires result from running over road debris.

 

[serkol]

@richkae, I read your post and somewhat agree with you. However you base your argument on questioning the official stats, but we don't have any other stats. My point is that we should look more optimistically into the near future regardless of this.

I strongly believe that from now on MS has much less chances to catch fire. Additionally I think that Tesla should reinforce the underbelly. I was worrying that they may not have much time for this, but now I have a feeling that they have enough time to come out with an effective and not too expensive solution. Maybe Elon thinks that his "fire warranty" is the final solution... I would rather see it combined with some additional underbelly reinforcement, not too expensive, but something that, say, allows MS to hit that cursed trailer hitch with lowered suspension, and just skid over it. I don't know if it's possible, but I would love to see this done.

 

[brianman]

you are correct, as was the other poster. my original post had an error in the wording, even though the numbers were correct. i have fixed that now to read:
" i find that the risk for all non-intentional fires is roughly 4x greater in ice vehicles than it is for model s. "

"4x greater" is the same as "5x"

I think you mean, more simply, "4x".

- - - Updated - - -

I mostly agree with your post in that Elon has significantly exaggerated his statistics, however, there is still too little data on Model S fires to do any apples to apples comparisons. Three fires in a short period of time out of a sample set of 19,000 with an average lifetime of 6 months can't be compared to a hundred thousand plus fires out of several hundred million vehicles. It just doesn't work. The number of occurrences isn't statistically significant. In fact, it makes more sense that the Model S has an even lower fire rate and this was just a fluke because there were no fires for over a year and then several in rapid succession. It will take a few more years, or many more fires to say anything definitively.

Yes. I think the statistics "currently show" that it's impossible for a Model S to catch on fire in December. So far we have zero samples of that occurring.

- - - Updated - - -

the crazy Mexican collision

I think this makes a great event description.

For those that might find the adjective ordering too loose, perhaps we could go with "crazy collision in Mexico".

- - - Updated - - -

am i allowed to argue with a moderator?

Yes, keep it civil and you'll be fine. (And can be fun.)

 

[stopcrazypp]

The TL;DR summary is the attribute 25,000 accidents each year to road debris with approximately 80-90 resulting fatalities. Not sure who's viewpoint this supports, but here you go: https://www.aaafoundation.org/sites/default/files/VRRD.pdf

This is helpful despite being older data. Given 190 million cars on the road in 1990, that's 1 debris related accident per year per 7600 vehicles (unfortunately both data includes all vehicle types, not just cars, but it's still better than no data).

The thread here shows 3 different cases of Model S running over debris (without fire), if you include the two fire cases, that's 5 known debris related accidents for the Model S.
http://www.teslamotorsclub.com/show...odel-S-crashes-did-NOT-result-in-a-fire/page2

If you take 25k Model S that's 1 debris related accident per year per 5000 Model S, and if you take 13k (car-years) that the OP used, that's 1 debris related accident per year per 2600 Model S or about 3x higher chance of debris related accidents (and that's just the known cases).

That's definitely going to skew the numbers in the direction of the Model S not more likely to catch on fire from such accidents compared to ICE vehicles.