Our Colocation Hosting versus AWS Costs Compared 2020 Edition

In this video, we compare our website hosting costs in colocation versus Amazon AWS. STH left AWS EC2 hosting in 2013 with our first colocation. After around 8 years, we have moved data centers and figured out what we need to maintain decent reliability and keep costs low. In this video, we are going to discuss how much we save annually. We also discuss the success of our previous efforts in 2018 estimating our AWS costs.

-----------------------------
Table of Contents
-----------------------------
00:00 Introduction
00:34 Brief History of STH Hosting
02:47 2018 Estimate 1-year Reserved and Partial Upfront on AWS
03:26 2018 Estimate Colocation
04:17 2018-2020 Our Self-Hosting Estimate Accuracy
08:32 STH 2020 Colocation v. AWS Assumptions
12:05 AWS 1-year Reserved Breakdown
14:22 AWS 3-year Reserved Breakdown (Mislabeled 1-year)
15:04 Building a 2020-2021 Colocation Estimate
18:38 Wrap-up

===================================
Other STH Content Mentioned in this Video
===================================

[ServeTheHomeVideo]

Quick one here. I saw there is a mislabeled slide at 14:22 where it says 1-year instead of 3-year. Making a note here and put it in the table of contents.

[MarkRose]

I administer a site on AWS that does about 2 billion page views a month, so medium to large scale. I have some observations about running on AWS. The amount of EC2 we use would fit in a single rack in a datacenter and burn under 5 killowatts.

The biggest benefit is having a team of people to manage the hardware, in multiple datacenters. Pretty much all our servers are cattle, and if one system goes down it simply gets replaced. You could take down an entire datacenter and we would know, but we'd be up. We just don't have to worry about hardware. The second benefit is having the ability to scale when required, without having to provision everything for peak load. Five times our traffic in ten minutes? Easy. And we only pay for the additional hardware as we need it. The biggest downside to being on AWS is the egress bandwidth fee. Egress bandwidth and CloudFront are negotiable when you have enough traffic.

As far as using Epyc for running websites, I've been disappointed by both Naples and Rome on AWS. The cheaper price is largely a false economy. Both Naples and Rome suffer from high memory latency which reflects in much higher CPU usage. For instance, a c5a.large (Rome) will run at 50% higher CPU than a c5.large running the same mcrouter and PHP load. And an r5ad.2xlarge (Naples) running memcache will use 500% of the CPU as a r5d.2xlarge. Yes, five times! I've seen similar poor performance running Java applications. Where I have found value in the Epyc instances are situations where the system is lightly used: stuff like Kubernetes master nodes, dev & qa environments, and so on. I hope Milan performs better.

People say AWS is expensive, but that's true only if you don't value your labour and can tolerate downtime if there are problems in your data centers. I can't imagine the number of people we would have to hire to have 24/7 monitoring of infrastructure in multiple datacenters to support the uptime promised to customers. It's an acceleration to not have to worry about hardware, especially for online-focused purposes. That being said, there's still a role for the traditional datacenter for other purposes. If I were running a render farm, I'd almost certainly build it out where power is cheap and not run it on AWS, for instance. Likewise if I hit sufficient scale that some cost such as bandwidth or file storage became cheaper for me to do on my own (exabytes). Netflix is famous for being entirely on AWS, except for their CDN, where the bandwidth savings are worth it. Similarly Dropbox moved off AWS when the storage costs became cheaper to in-house.

[FWVO]

Generally it comes down to this: "You either gonna do it your self or you gonna pay the big bucks."

[wilcosec]

TLDR: don't move to the cloud for cost savings, especially for the small to medium sized business with a more traditional web stack.

IMO, moving to AWS is great for business agility and where extreme elasticity or high availability is needed.

[MrBillrookard]

I was doing some web-dev stuff years ago where I had a co-located server about 200 miles away. I wound up bringing it all in-house, literally, as I got a business class connection to my home, some static IPs assigned, and built a small rack in my basement and have been serving those sites for about 10 years now (the website has been active since 1997). Considering my internet access is provided by the business connection, I pretty much pay out of pocket for the static IPs, plus electricity. I can thus expand the servers as I wish, add storage as I wish, add RAM as I wish, and if something goes down, I walk down a flight of stairs to figure out what's going wrong.

I've literally saved thousands a year doing this, and while I understand the advantages to not having to deal with the hardware layer of things, it's something I enjoy tinkering with - and of course saving money is always a good thing.

[ws]

I definitely would like to hear about some of the lessons learned about co-location since 2013. Its always interesting to learn how decisions were made. Based on what was needed, etc.

[pepeshopping]

Crazy amount of money for what the web site is/does.

[noahlistgarten]

Another excellent video! I don’t know if this has been covered in an article before, but would love a video describing how the website is currently set up/configured, from networking/firewall to server to hypervisor(s) to VMs to containers to CMS (as security allows, of course).

[GeoffGroves]

As partner and CEO of an AWS consulting partner, i thought this was really really well done and very accurate. Nice job!

[mikepb]

Sounds like site that could done using static site generation and host serverless. The would cut it down to a fraction of this.

[colinstu]

give us all you colo stories and lessons learned!

[steven]

Colo pricing and what you get for your monthly $$ must vary region to region, for our base monthly we get the 42u rack + 2 kW of constant draw (A + B combined), every extra kW above that has a fixed fee, data is not supplied, you pay monthly for a fibre cross connect to your supplier of choice and they charge you monthly for what ever speed/transfer you agree upon.

We moved to Colo after looking at how much refitted/rebuilding our server room would be to add generator backup, gas fire suppression, cooling and then the maintenance/power costs. We worked out that we could move into a Colo and pay for years worth of rack space that comes with all of that and more for the same cost but now it's monthly instead of up front. Hardware costs would be more, but in return the original server room becomes a DR/secondary location and you are only buying a few extra switches/routers as the bulk of the hardware was needed regardless of location.

[serverguru]

As a hardware guy tasked with investigating MSP and CSP business models this week, this is timely :) Thanks for the continued content Patrick.

[GGBeyond]

I went with co-location for my web projects because of the lower overall monthly cost compared to AWS and I'm able to pick up plenty of used servers off of Ebay for really cheap. I run a LAMP stack with Memcached and Elasticsearch.

[gdrriley]

can't wait for zen3 EYPC. Racks and racks of it will be in soon for the data center I work in.

[PaulJeffery]

What would the cost and ROI for using a CDN to cache images and static content? I would like to see what the cost delta is between co-lo bandwidth and CDN costs.

[Whipster-Old]

Awesome video. I see so many organizations put stuff in the cloud and wind up surprised about the cost.

[D_T]

Interesting video! For AWS, have you considered reserved instances only for "valleys" and using on-demand in auto-scaling group for varying traffic? This way you can save costs on idle hardware. Another suggestion I have is to look at Savings Plans, it's a new version of Reserved Instances that offer greater flexibility and same savings. I also highly recommend looking at Graviton2 ARM instances that are 3/4 the cost of x86 machines with similar performance.

Pro tip, to save on AWS outbound data charges, take a look at Lightsail instances, those come with Terabytes of bundled transfer (both in&out).

[rougebarbu]

Did you try to calculate the cost of renting bare metal servers at a facility like OVHcloud. They take care of all the hardware/network issues for you. We are running 6 servers with AMD 7371, 256G RAM + NVMe disks for about $2.5k/mo. Including bandwidth.

[brannancloward]

Very interesting breakdown. I really appreciate the helpful information! You have answered in 1 video every question I have ever had and could not find the answer to. THANKS!