Scaling Docker #1 - High Availability Docker DevOps with CoreOS, Etcd, Fleet, Flannel & Nginx

Let's build a high-availability production docker cluster!
So web development with Docker pretty much ROCKS - it enables you to build and scale world-class apps that used to only be doable for those working at companies like Google and Netflix.

The (not-really-bad) bad news is you have to start thinking like a world-class developer to make full use of it. To go from basic docker container deployment to a full-scale docker system, you'll need to implement a few things:

THIS VIDEO:
- Cluster computing - CoreOS is our solution here. CoreOS allows us to make an expandable cluster of computers so we can add more computers should our application need more power to scale. CoreOS also allows us to run services on any machine in the cluster and they can all communicate with each other.

- Scheduling: we shouldn't manually choose which app runs on which machine - this idea doesn't work at scale. Fleet will be our scheduler that decides where our apps/services run. If a CoreOS machine goes down, fleet will reschedule any services that it was running onto a new machine.

- Service Registration/Service Discovery: Since we never know what ip address or port a service will be running on, we need to register that service so other services can find it and communicate with it. We'll use Etcd and Flannel for this

- Public DNS: Once the app is working, we need a fixed way for the outside world to access it. We'll use Nginx and Confd for this. Confd will watch Etcd for service registration and when services come/go, it will build a new configuration file for Nginx and reload Nginx.

This docker tutorial will help you understand how to go through every step of this process.

-~-~~-~~~-~~-~-
Also watch: "Responsive Design Tutorial - Tips for making web sites look great on any device"
-~-~~-~~~-~~-~-

[luisobarbaro]

Thanks for consistently uploading high quality content! This past summer I watched your video "Watch this if you want to become a web developer". Since then I learned a ton of cool stuff, like react, redux, rethinkdb, node, webpack, etc. Now I'm in the process of scaling my web application :D.

[srik]

This was an awesome series of Tutorials

[kooraiber]

Thank you so much for these tutorials, they're so easy to understand!

[santicomp]

Are you going to do a kubernetes tutorial?, that would be awesome thanks
nice video by the way.

[pilkers]

Nice walkthrough, fleet + flannel seem interesting for orchestration, just wondering why you wouldn't stick with native docker swarm though, what advantages does fleet + flannel bring? Thanks again :-)

[arulpr]

This is so much easier now as Digitalocean now allows to create multiple droplets of same config

[plsreleasethekraken]

Is there a way to make this work if your cluster is across different regions? I seem to remember shared private networking only working inside the same region.

[noeskasoftware]

how does coreos know the cluster size? and if i set the cluster size to 3 do i need a minimum of 2 members to make etcd work?

[nsntg]

very informative for me :).. just want to ask. how can i configure the coreos cluster if i want to add one more coreos in the cluster? should i generate another discovery url which has a size of 5 then replace the old one that has a size of 3? thanks for the great tutorial btw

[cpakken]

Are the things you're showing in this video Digital Ocean specific? (eg Coreos configurations) How will this translate using another service like AWS?

[pgtrots]

Hey great series! Genuine question, do you consider this best practice above Kubernetes or Nomad?