Photo Storage with Amazon Glacier and S3

All the info I'm covering is there as well as more detail.

Last summer we talked about the launch of Amazon Glacier and the possibilities of using it for photo storage. In this video I want to do a followup since its now actually possible to use Glacier and it turns out its a lot easier to use than we thought it would be when the announcement came.

Its extremely important here to say quickly what Glacier is and what Glacier is not.

Glacier is cold storage. This is designed for files that you don't need to access very often, but you don't want to delete. Cold storage is perfectly safe and redundant. Amazon doesn't explain the process, but most likely the files are offloaded on to tape based storage or servers that run low power at low temperature to reduce the cost of "running". This means if you need to get to one of these files, you'll need to initiate a request and 4-5 hours later your file will be ready for download. There is a transfer fee if you go over a percentage of your total storage.

Glacier is not an FTP server or Digital Asset Management System. You can't flip through thumbnails or in the use case I'm giving in this video, do meta data association.

But the great thing about Glacier is that you can store old photos that you don't need to get to very often. I like to use this in tandem with a second flickr account. I make sure my names for files are consistent and upload JPG files to Flickr and Raw files to Glacier. I use all the metadata and set management in Flickr and keep my files organized so that if I ever do need the raw file - I can retrieve it from Glacier.

The best part is if you can deal with these restrictions, Glacier is 1 cent per gigabyte per month. Way cheaper than it would cost to run a server in your house and its probably more practical as well.

In this example - we use S3 (which is also pretty affordable at 9 cents a gig) - but I'll show you how to change the storage class from Standard to Glacier. You'll just manage your Glacier files in S3 - no front end, fancy app or database management needed!

[AquashiiMusic]

Man I REALLY appreciate you doing this. I have been scratching my head with watching so many other videos. Very clear thank you!

[EnricoMariaCrisostomo]

That's a nice idea. The easiest thing I can think of is mounting the S3 bucket as a filesystem and use it from your catalogue. Beware S3 is going to return an API error (just a warning) every time you try to "get" the file, and Lightroom may be willing to do that quite often.

Ted has made an important point: Glacier shouldn't something you'd use online. In fact, I tried some sync even with plain S3 mounted as a disk and that can be a real pain (long story short: forget about rsync efficiency).

[EnricoMariaCrisostomo]

That said, if you're worried about your catalogue, why don't you just back it up to S3 regularly, as you'd do with other files? In case of disaster, you get the stuff out of Glacier (the last catalogue and the photos). If you preserve the directory structure of your images, telling Lightroom to "rebase" the catalogue with a new root directory is one click away. And it works because I did it multiple times.

[theartofphotography]

Everything is pro-rated. S3 is 9.5 cents a month per gigabyte. If you have 1 day before it kicks to Glacier you'll pay 1/30 of that roughly for the 1 day it sat in S3. So yes, but its likely not noticeable. Transfer in to S3 is free.

[shrwnsan]

Thanks for this. Looking for options to transfer off Flickr, because of the current service updates. Any update to this workflow?

[trevorpinnocky]

It seems like creating an archival catalogue in Adobe Lightroom to be used in conjunction with Glacier would be a very tight solution. You'd be able to easy search through tags with the indexing benefits of Lightroom's db and then easily retrieve that from Glacier. Cheers.

[theartofphotography]

If you can archive straight to S3 you're in business.

[EnricoMariaCrisostomo]

Thanks Ted, another awesome podcast of yours!

[Indevor]

I'm a big fan of your videos. Would like to know what camera/lens you are using to shoot your videos. They are crystal clear. Regards and happy new year.

[santoshbulusu]

very informative, i just wanted to ask if i want to upload a file to flickr from gracier, do i have to download file from glacier to local storage and then upload to flickr?

[thunderchild]

Cheers, Ted. I'll be checking this out at the weekend.

[MrBantas]

Its a 5D Mark2 with a 24-70L, I guess.

[GrandMiserigueux]

Very long for very few usefull explanations.