Adding 810TBs of Tape Storage

Keeping the old stuff has always been a mess. Lots of focus has been placed on redundancy but not on backups. Here's the next step in our long term storage!

[davegem]

Having worked with many versions of LTO drives/tapes over the last 20 years I know all too well that they can/do fail, don’t fall into the trap of thinking that the data/tapes won’t degrade over time or get chewed up by a bad tape drive leader. I highly recommend you create two tapes for each backup and keep a good backup of your backup library master data base, so you don’t have to rescan all the tapes to rebuilt it. It’s also important that you follow the correct storage guidelines (temperature, humidity etc)

[steve_main]

As an IT guy all I have to say is "MAKE SURE YOUR BACK UP YOUR CATALOG DATABASE" this is the database that tells you what is on each tape and test its restore every so often on a test machine. This is so important and no one thinks about backing this data up. Make sure the backup is automated and is taken off site and ensure you have multi revisions of this database. If you don't have this catalog all your tapes are useless or you will have to re catalog all you tapes. If you only have 4 not an issue but once you get to a 100... you could spend a month cataloging your tapes!

[zncon]

If you really care to preserve everything then 3-2-1 backups are critical. Tape can be a key part of this, but it can't be the only thing.
3 - Total copies
2 - Different types of media
1 - Copy at a different physical location

[gobbel]

Big props to those tapes for ensuring that Dan stays the most photographed person on earth for the next years.

[darkavengerk]

As has been said below already, make sure you have two copies. One on-site and one off-site. The fact that you haven't lost data is more of a miracle than anything else.

Also tapes do suffer from bit rot as well (they are magnetic as well)
Insure you store them in a stable environment, while they are great for archive storage the archive part only works if they are treated as per their specs.

[neo]

To repeat what others already said... But since it's important and could avoid data loss. Do NOT expect 30 years out from those tapes. LTO is by no means safe from data/bit rot and for important data you should rotate every 5-10 years depending on storage conditions. The 30 years they market them with is during very specific conditions when it comes to temp and humidity, treat it as marketing buzz. Also, do NOT throw them around in luggage and avoid X-rays, strong magnetic fields and so on. And for the love of God, do NOT drop them. They are fragile and you just have to disassemble one to realize exactly how fragile they are. Have had more then one eaten by tape drives in the past (LTO4/5). You should have two copies of all critical data. If possible, use LTFS to avoid being locked into one vendors proprietary db's and system and keep backups of any eventual databases. But yes! When dealt with properly and when knowing about the quirks and limitations and gotchas LTO is very cost effective and nice. But like everything, LTO have its limitations and should not be treated as a indestructible bullet proof Jesus-medium that will save us all. 😎👍

[Chester]

Magnetic tape is so cool, i'm still shocked after i found out that a vhs can hold gigabytes of digital data depending on the length. 18tb in such a small package is unbelievable data density

[jeeves-]

While this is a big improvement vs random hard drives, as someone who archives data for a living I'd caution you against just putting everything on these tapes once and assuming they'll be safe for 30 years. That guarantee is more like a warranty than a specification, and I've had tapes fail much earlier than their rated lifespan. An immediate solution would be to use some form of parity or mirroring to ensure every file can't be lost with a single tape failing, but longer term I'd highly recommend backing up your archives using multiple forms of storage, in multiple locations, especially since this is your livelihood and if something like a flood hit wherever you store your tapes, you'd lose everything. A cheap option would be to make a JBOD server using your existing HDDs you just moved your data off of. JBOD is short for just a bunch of disks, and is exactly what it sounds like. It would also let you use a filesystem that automatically checks for and fixes bitrot, like ZFS, and mean you would have access to your entire archive without having to plug in a certain tape, or even multiple as LTO tapes can store a single file across multiple drives to ensure you're using all the storage it offers. This is why they have the barcodes as standard, as many commercial archives using LTO tapes will have a tape library, which are basically jukeboxes with tape instead of vinyl, and the software can automatically switch tapes to retrieve data spread across multiple drives.

[FarrellMcGovern]

I've worked with the LTO tape drives for a long time, although I worked primarily with Tape Library machines, that is, robots that you load dozens of tape into and has more than one drive. The backup software has a master database of all the files and tapes, and does backups, automatically changing the tape when the one in the drive becomes full. As other people have said you need *two* backups of everything, and store one copy in another site, preferably that is climate controlled. You should also periodically verify your backups. In the long term, you should also consider budgeting to change storage mediums every 5-10 years. Just like you are going from Hard Drives to LTO Tapes, in 5 - 10 years, you should be looking at the next great thing...maybe SSDs, or maybe holographic storage...

[radenthefridge]

Don't forget a cleaning tape or two for the drive! They need to be cleaned every so often (40+ hours of writing, for example) for best operation.

[mugzy]

For someone who only genuinely enjoys the content and knows nothing about the logistics and workings of your results, its really cool to see this. As much as I wish you both were super close to have more time in this channel, life moves on and project priorities change i am glad you keep slowmoguys still going!

[joelrobe]

Of all the YouTubers, your channel has been the one that has been in the back of my mind constantly. as someone who's a tech enthusiast, knowing how much raw data you generate every video and trying to come up with a solution had to come in occasional mental exercise for myself. I knew tape would probably be your archival solution but I didn't know the technology had progressed to the point where the write speeds would actually be faster than most hard drives. Your probably still needing a Linus grade storage server for, effectively, a scratch disk during editing but now you can of-load one your done and can stop having to build servers. Sweet

[BEACHYz]

Very interesting. Thanks for taking the time to go through the process of migrating large amounts of data from hdd's to LTO tape.

[joeyvachon]

Love the breakdown, it’s obviously not just a hobby for you guys! Keep bringing us that sweet delicious slo-mo greatness ♥️

[CaffeinatedHiFi]

I'm just going to echo what other people are saying and strongly consider duplicating your backups and storing them off site somewhere. Redundancy is just as important as backing up the data itself. It doesn't do you much good to back up all your data if it's all sitting in one location waiting to get destroyed in a fire or a tornado. You have a lot of very valuable data and you're now stepping into the world of enterprise level solutions. I would find an off-site backup location in another city, or state if you can mange it. Redundancy, redundancy, redundancy!!!

[stanley_]

That's bonkers! I didn't know such efficient storage even existed!
What also surprised me was that 8mm film converter on the shelf there; I didn't expect you of all people to have one of those, lol!

[Ttarler]

I’m a data scientist in the banking industry and I train models with tape archives all the time. Super reliable form of cold storage.

[jasonwohl]

Love that you now have more cohesive way to store your vids, wondering what happened with the NAS LTT helped you out with. Love all the vids!
Cannot stress enough that you should always have AT LEAST 2 copies of all your data (or anybody else reading this comment)

[MisterGrim]

"Phantom Operator" is the best job title I've ever heard of 🤘🏿

[Amouroso]

Love a slo mo guys 2nd channel video! So glad this channel is still active