Our data is GONE... Again - Petabyte Project Recovery Part 1

It's been a long time since we've had any serious data loss, but on this episode, we're discussing a software misconfiguration that has resulted in us losing an unknown amount of data on our petabyte project storage clusters.

Check out 45Drives at the links below

Buy Seagate 20TB Exos Drives

Purchases made through some store links may provide some compensation to Linus Media Group.

FOLLOW US ELSEWHERE
---------------------------------------------------

MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova

Outro: Approaching Nirvana - Sugar High

CHAPTERS
---------------------------------------------------
0:00 Intro

[The_Keeper]

Linus: "Right, *Now* we won't ever lose data again!
Data storage: "How many time do we have to teach you this lesson, old man?"

[markclayton]

The irony of a cloud storage provider sponsoring this segment is not lost on Linus. I like that.

[Lmpy]

LTT never ceases to amaze me on how professional and unprofessional they actually are at the same time.

[ashleymc]

"We never hired a full-time IT person" was stated and I immediately had the urge to bust out the popcorn and look at IT pros in the comment section.

[KoSiNeK]

I don't know why, but "server issues" episodes are my favourite LTT videos. Content like this just doesn't exist anywhere else.

[HAWKF]

Linus: Hates how USB and HDMI are being named.
Also Linus: New new new vault

[shwolverine]

Linus: "the way they name HDMI generations are so confusing"
also Linus: "we move the data from the old vault to new new vault and then name the old vault new new new vault with a bit of upgrade"

[OnlineWerds]

As a data center engineer your storage content is my favorite content. I'm terribly sorry for your issues here.

[loadnabox]

Linus, I have over a decade of experience in managing multi-petabyte ZFS with five nines uptime in large ISP's. I think you may have the wrong cause of the data and it may not (MAY NOT) be as lost as you think.

Please reach out to me

[TJ-vhps]

Postmortem reports like this are hugely valuable, but companies don’t usually share them. This is a great service to the community.

[Cluesman]

"a lot of power outages" + "transferring that much data might take months" sounds like a recipe for another video in this series.

[ulbuilder]

Your backups must be tested
So you know they work as expected
Offline is best
So you can rest
When lightening strikes unexpected

[DarrynJones]

"I'm the highest ranking person in the company, the highest ranking person in the IT team, and the person who decided not to hire a dedicated IT staff. There is no way to determine who's accountable here" - Linus 2022

[obedulloa]

If Linus manages his data the way he manages hardware... it's no surprise the data dropped

[jstadler]

As a full time Sysadmin i always wondered how you guys sustained your data without a real backup plan. As it turns out now, you didn't. Really sorry to hear that guys!
That's exactly why people like me get hired. Companies think they can do it on their own until they lose critical data to misconfigs and missing maintenance. Hurts to learn it the hard way.
I really recommend you guys to create offline backups to tape storage for all your archived content.
And respect for admitting having it done wrong so others can learn!
Keep on making such great content!

[LabGecko]

Tech Tips' data loss is due to one thing - quantum variability. :D
The data was in a state of flux until someone audited, at which point it was forced to exist or not exist. Some were observed to be the latter.

[lucasmenchone]

HR meeting with Linus: “All our data has been lost, i’m gonna fire someone…
But not before i fire up our segway to our sponsor…”

[leodoz]

Alternate title: The LMG group MIGHT hire an actual IT person

[QualityDoggo]

Just hearing "never hired a full time IT person" makes me go "uh oh... I don't like where this is going..." a good sysadmin who can help protect systems is a valuable part of any modern company

[RobertCrawfordRobert]

As soon as they switched from storage spaces I kind of saw this coming; I've got a 912tb S2D cluster that serves as storage for about 200 or so virtual machines and it's been rock solid and performance with NVME cache has been solid. One of the things I saw on Spiceworks was a warning about over engineering infrastructure.