We’ve NEVER done this before… - Mother Vault Part 1 - JBOD

We've built some cool machines on the channel, but nothing is even close to the plan for our 3.6-petabyte archival storage array.

Purchases made through some store links may provide some compensation to Linus Media Group.

FOLLOW US
---------------------------------------------------

MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova

Outro: Approaching Nirvana - Sugar High

CHAPTERS
---------------------------------------------------
0:00 Intro & History
4:19 The JBOD
9:15 The Computer
14:10 FILLED
15:26 Cabling & Zoning
16:23 It's... LOUD
18:19 IPMI
18:54 What about dual path & high availability?
19:58 It's aliveeee!
21:30 Performance testing & outro

[fnm]

At this point they may as well make a server for all the footage of them installing more servers

[Ascii_the_fallow_deer]

"The first petabyte project was built in 2015"
Me: *freezes half way through eating* "... 7 years?!?! What have I done with my life!?"

[sebastianturner]

The amount of times he has said “not one, not two, but three”
It just shows how big things are getting at LTT

[QuestGiver]

In the grim darkness of 2025, Linus tech tips finds itself barely recognizable as thousands of servers coat their buildings like a 2 meter thick layer of armor, shielding the world from the madman at it's heart. Inside the asylum, a monument to the sins of Linus' technology addiction, he slaves away streaming 24/7 as he tries in vain to "save the universe" in sever project after server project. His employees are wired directly into everything as little more than motherboards waiting to be dropped by Linus. As time goes by, Linus will himself finally become the master of the internet as he wires himself into his temple of tech tips, continually giving tips about tech so arcane that they are lost upon the ears of the masses. In the year 2025, Linus tech tips becomes unknowable, unrecognizable, feared, and worshipped.

[sakaraist]

These videos are truly genius. As a company they need to upgrade things, so they go way overboard and it makes an awesome video showing some unique tech configs we wouldn't otherwise see. They get sponsors to send in components for free, sponsorspots sold on the video and ad-revenue from the video it self. The epitome of efficiency.

[blackraen]

12:19 Biggest thing I have to deal with as a datacenter Storage Admin is constantly checking people that assume Storage doesn't need lots of host resources. Perfectly competent technical people somehow miss that processing and handling IO isn't free in terms of system resources, both at the Storage System AND Host Initiator sides.

20:20 Metadata dedicated is a godsend for huge fileservers like yours -- The idea is that filesystem metadata lookup is actually a really big performance hit because metadata reads/writes are very small, very random, and usually with a sync_required status so they hold up the line. With a metadata tier, you put all your metadata in a dedicated high performance tier/pool/spam/whatever and it not only speeds up your metadata activity, but it also keeps your other storage open for regular IO without having to shudder around working those small metadata IO packets in the middle of regular workload.

[JeffGeerling]

Next year, I'm expecting to see the same thing, in flash storage 🤪

The crazy thing is, you could put more storage per unit right now. It would just cost about 30x more lol

[xMcAwesomex]

One day it’d be cool if you guys made a video covering ALL the servers you guys have ever had dating back to the original. Covering all the names, capacity, reason for upgrading, all that stuff, that’d be awesome!

[The_Burrito]

"You can fix anything... with money."

Said with such dejection from a man that literally throws money at *all* of his problems.

[Veerorith]

I love how every time linus picks something heavy/expensive up you can hear everyone in the background collectively have a mild panic moment

[tousifraza]

*Everytime Linus needs a new server* : "WE HAVE A SERIOUS PROBLEM"

[christianedwards]

Linus always building some insane stuff for our entertainment even if it has no feasible uses for us... kudos

[hahanamegobrrr]

Linus in 2024 : The MOTHER VAULT is completely FULL!

[KibbleWhite]

Finally, they are doing it right, I've seen so many of their setups and thought, well if that works best for you and the team, then cool. But I always thought in the back of my head, that they really should separate the storage and compute and have the storage compatible to where you can swap out the compute unit and continue to roll without downtime. Your storage is starting to look strong now LTT, sweet !! ❤️

[nihung_mind]

Linus tech tips in 2035 :-
Building a zotabyte server for YouTube.

[XJA]

What boggles my mind is that despite it being a known issue for years and despite spending so much money on so many other projects... you guys still haven't hired a full time system administrator?

[big_matt]

Linus I watched your video around 2 years ago when you were deciding to quit due to juggling family and other psychological problems, now to see how far you have come I'm glad you didn't. Thanks :)

[dorixcraft]

There's one cool thing with ceph : it makes data loss *reaaaly* unlikely, because when an error is detected, as long as you have enough space left in the cluster, in terms of redundancy, and space, it will recover automatically and after that, even if you didn't change your bad drive, you'd have the same amount of redundancy as before the drive went bad, you just lost the drive's capacity, and usually you don't let such a cluster fill up to the point where it's unable to do that, and you expand before.
Also, ceph is a joy to expand : make the node join the cluster, create OSDs, done. Ceph will automatically rebalance the data if needed. Same thing if a node goes down, if there are enough nodes left to keep the required level or redundancy, it will automatically rebalance the data to make it happen.
We use this in our IT club at our uni, it saved us multiple times, keeping our infrastructure up and running even when we f things up. It is capable of recovering from very, VERY bad situations.
There was a time where we didn't have the "manpower" needed to replace bad drives in our cluster : if was fine : ceph kept these bad drives out of the cluster and rebuilt the redundancy, so even after all the drives of a node failed (bad HBA card), some within days of another, with no manual operation, there was no data loss, only capacity loss, which was fine, as we overprovisioned.
Truly a marvel.
In comparison, ZFS is only *good*, I think, a lot less flexible, certainly.
Edit : also, ceph is able to handle drives of drastically different capacities, and handling it as best as possible, without much intervention, if at all (depending on the crush rules, etc ...).

[MarkusHobelsberger]

Installing these in Lab 2 is quite an ingenious idea considering one of Lab 2's key features is its insane power infrastructure. Then just run 40Gbit or something fiber back to Lab 1 :D

[mustache_music]

Let's take a moment to appreciate the fact that the 1000 watts quoted here is powerful enough to run most small push style lawn mowers