EEVblog 1422 - CAUSE of the Tesla Victoria Big Battery Fire

The report has been released on the Tesla Moorabool Victoria Big battery fire by Energy Safe Victoria, let's go through it and discuss the cause.

00:00 - Overview Tesla Moorabool Fire
02:44 - Finding on the cause of the fire
07:14 - Corrections and improvements required to get back online
09:26 - Physical pack spacing
11:01 - Statement of Technical Findings Report
12:30 - Root cause
13:45 - Contributory Factors
16:53 - Lessons leanred and preventing a recurrence
17:31 - Dave's hypothesis
20:49 - Fire propagation prevention
22:28 - Conclusion
24:50 - Other installations

Buy anything through that link and Dave gets a commission at no cost to you.

Donate With Bitcoin & Other Crypto Currencies!

#ElectronicsCreators #Tesla #Fire

[EEVblog]

TLDR; The megepack was taken offline at the time of the fault, so it was not charging or discharging. A leak in the cooling system caused a short circuit in some component that started the fire. So the power to sustain the short and subsequent fire must have come from the internal pack.
Also, offline mode disabled all protection systems and monitoring communications, so they had no way of knowing anything was wrong until someone yelled fire.

[bigclivedotcom]

Sometimes it takes an incident like this to encourage reassessment of safety systems. I'd guess the offline state would be to allow working on alarm systems without accidentally triggering emergency responses. The spacing is possibly a cost thing with available space and keeping packs in the vicinity of common electrical distribution and cooling systems.

This incident will probably reshape future installations and procedures. So in a way it's good that it happened. Especially in such a young section of the power distribution industry.

[IanScottJohnston]

I was charging a large lithium-ion strimmer battery in my workshop last week, like I've done many times before. However this time the charger showed a voltage error after about 30mins and the battery was HOT. I disconnected the charger and took the battery outside and put it on the patio (stone floor) and went off to get a screwdriver so I could take it apart (once it had as I returned to it, about 10ft away, it started hissing so I stepped back and it went BANG and the firework like flames were about 3ft in all directions.
lucky!

[mitsync]

I suspect the offline mode they used to temporarily take it out of service wasn't meant to be used that way. Having to actuate a physical keyswitch tells me that mode was probably meant for a complete shutdown so maintenance can be performed, as the operator probably wants to automate the wear-balancing that happens daily (which is what it seems the unit was shut down for). This could explain why EVERYTHING in that pack turned off, instead of only the charge/discharge you would expect for a standby like that.
I do agree that it is still weird that telemetry and monitoring can be turned off like that, and especially that it was actively used while SCADA was not active yet, but I highly doubt this was the intended shut-down mode for this situation, let alone the only mode available.

[JediBuddhist]

Some good old Asbestos sheets between the mega packs would work a treat.

[Turbochargedtwelve]

Who did the FMEA for this system? Allowing monitoring to be inactive during initial start up is crazy, that is the most likely time to have issues. Allowing monitoring and cooling to go down for any reason is crazy. Have we learned nothing from our nuclear disasters? Those should be on back up, independent power, enough to safely bring the machine to a standby state.

[WacKEDmaN]

20c part takes out $200K pack!
...sounds like they didnt think things well through to begin with... taking a pack offline should never stop alarms, monitors and protection....

[urbiso]

As a former SW developer of a Safety Analysis software for oil&gas industry, I'm realy surprised how badly implemented are safety standards in this energy battery field. Someone did very bad job with HAZOP/LOPA analysis, or most probably there was none. Having such devices in operation without real time monitoring is absolutely unimeginable in o&g industry, if this would happen, it would definitively cause automatic shutdown. They can be realy glad that only 2 packs were burned and no one was hurt. And to me, if the coolant leakage, which, I assume, is one of the most common failure which you can have, can cause catastrophic failure, then the pack is probably not well designed.

[WizardTim]

Geez they were running that thing blind without the SCADA monitoring!

The addition of the ‘battery module isolation loss’ alarm makes me think the failure could have been coolant leaking onto a DC contactor (or bus bar) for a battery module shorting and overloading it, however as the unit was in the offline service mode and no SCADA data was being monitored the unit never raised an alarm and thus never had its pyrotechnic fuse triggered so it remained shorted until it reached thermal runaway and caught fire.

And without a doubt this was not a safe failure, a better way of putting it would have been “despite the unsafe situation created by the catastrophic failure of a megapack, there was no loss of life and damage was contained to only the adjacent megapack with the assistance of approximately 150 firefighters and 30 firefighting appliances.”

I also suspect the solution ‘to fully mitigate the risk of fire propagation’ will not only involve firewalls but fire suppression.

[PaulSteMarie]

It sounds like the offline mode for maintenance is really intended to be just that: having the thing shut down and disconnected so you can work on it. They probably didn't foresee the need for monitoring while the thing is known to be offline. Unfortunately, you can't just turn off a bunch of fully charged batteries.

Chemical plants have to do that, but they have really complicated monitoring systems that go into different modes to prevent nuisance alarms. Watch some of the US CSB videos: they go into some wonderful information on the principles involved.

[MrSmeagolsGhost]

Oh 2nd thing, reason it’s a key switch that isolates everything.

It’s a part of the high voltage isolation procedures to physically isolate remote operation.

If the key is in and turned someone is working on it/it’s OOS and someone needs to physically attend to put it back on. Having it just drop all comms is a brute force way which makes things simpler. If you still return statuses etc there is a lot more testing needed to prove there is no way for it to be remotely operated/controlled while in local/maintenance mode.

For items like circuit breakers putting them in local mode will short/disconnect the control wiring but still allow status and position data to be sent back as it’s physically separate circuits.

In other instances physical links are removed or shorted to achieve the same thing.

[nfdrkkrz]

Yeah its ultimately up to the installer and site planner to implement pack spacing, but its also on Tesla to provide recommendations on pack spacing and potentially implement better heat shielding and fire mitigation

[--Zook--]

This incident had me seriously re-think my power cell storage placement and spacing in my Minecraft all the mods 6 world.






:)

[petehiggins]

They could perhaps fill the gap between modules with fire retardant foam as used by electricians to seal apertures in domestic fuse boxes. This is a foam that sets hard like builder's expanding foam.

[abpccpba]

Every responsible group/team did not do their due diligence. From what I have heard from Dave's reading is there are some horrendous design flaws that could have been found long before anything was constructed/programmed. : = (

[AndrewFremantle]

Very notably missing was a requirement (or even a suggestion!) for a fire mitigation system! You shouldn't be asking firefighters to place hoses between these smouldering bombs, you should have pipelines with nozzles pre-installed. Press button, get cooling mist between all the cabinets to mitigate the impacts of an uncontrolled

[CLipka]

16:15 - "If it shut down the cooling system how can the leak matter?" - The report clearly speaks of "a leak [...] that caused a short circuit that led to a fire". In other words, leaking coolant had managed to find its way onto electronics, shorted them out, and thus caused some overcurrent in a component, which in turn burst into flames because it wasn't designed to take that kind of abuse.

[jrb_sland]

There are almost always teething troubles with newly assembled systems. In these modern times, that also applies to the computerized control & alarm systems. Someone somewhere didn't think through the subtleties & interactions of various safety systems. After this demonstration of some shortfalls, all future systems will be far, far safer. Thanks, Dave, for the walkabout.

[chrishartley]

Hmmm, design idea, let's put the electronics where leaking coolant can drip onto it.

[aquilux-vids]

23:23 "failed safely" in this context means that:
a) it wasn't a chain reaction that took out the whole facility
b) the damage did not spread to other nearby but unrelated facilities
c) the failure didn't generate widespread hazards that could have caused injury in the general area (beyond the reach of the fire)
d) the failure did not create immediate hazards that would have injured anyone nearby (shock wave, shrapnel etc.)
e) the failure occurred in a measured and controlled way giving anyone in the immediate vicinity enough warning and time to move to a safe distance before risk of injury was present