3 Node Hyperconverged Proxmox cluster: Failure testing, Ceph performance, 10Gb mesh network

Why stop at 1 server? This videos goes over Proxmox clusters, what they can do, and how failure is handled.

Thanks to QSFPTEK for providing the network cables and transceivers used in this video. These products are available in the links below:

Let me know if you have any ideas that I can do with this cluster. I'd love to try more software and hardware configurations. This video also skipped many of the details when setting up this cluster. Let me know if you want me to go into more details with any parts of this video.

00:00 Intro
00:32 Hardware overview
00:57 Networking hardware setup
03:06 Software overview
03:30 Ceph overview
05:06 Network configuration
6:30 Advantages of a cluster
7:10 Ceph performance
8:45 Failure testing
9:00 Ceph drive failure
9:28 network link failure
10:00 Node failure
11:06 Conclusion

Full Node Specs:
Node0:
EMC Isilon x200
2x L5630
24GB DDR3
4x500GB SSDs
Node1:
DIY server with Intel S2600CP motherboard
2x E5 2680 v2
64GB
5x Sun F40 SSDs(20x 100GB SSD spresented to the OS)
Node2:
Asus LGA 2011v3 1u server
1x E5 2643 v4
128GB DDR4
4x500GB SSDs

[ZimTachyon]

You are genuinely great at presenting this content. You first hinted at not using a switch which caught my attention right away. Then you showed the triangle configuration to answer how you anticipate it would work. Finally you asked and answered the same questions I had like how do you avoid loops. Excellent presentation and extremely valuable.

[pauliussutkus]

Would love to have video with detailed explanation on how you setup proxmoxes, put them in cluster, setup mesh network using frr, cheking connectivity between the nodes(iperf3 or route ip 6 route). Adding subnet for cluster(ceph). Setup ceph and best pratice to use 2 copies of file or 2+1(parity). Also how to avoid fails. I think full tutorial of that would be great. Or you can divide into parts. Anyway good job.

[iamweave]

Nice. Usually I watch instructional videos at 1.25x or 1.5x -- yours is the first one I thought I was going to have to run it at lower than 1x!

[davidkamaunu]

Awesome presentation! and @03:13 free range chickens in the background!! 🤠👏

[allards]

Thanks you for this video, I never heard of the Proxmox Full Mesh Network Ceph feature before.

I recently bought three Mini-pc’s for the purpose of building an Proxmox HA cluster. I was planning on getting a small 2.5 GB switch for the storage.

Since the Mini-pc’s have two 2.5 GB ports I will use them in a Full Mesh Network buying separate USB-C to Ethernet adapters for the LAN connectivity.

For my Homelab such a setup is more than powerful enough.
Going to have a lot of fun (and frustration 😅) with an advanced Proxmox setup and Kubernetes Cluster on top of it..

[bluesquadron]

Running this kind of setup on three identical Elitedesk sff nodes with dedicated m.2 drive for ceph. Even with a single 1Gb connection to a router everything works great. Ceph likes memory so have to run with at least 24 GB.

[Ronaaronhunt]

Great content. It would be great if you could cover maintenance of the cluster. Things like upgrading a hard drive and/or replacing one of cluster PCs if there is a hardware failure.

[TooLazyToFail]

This is a cool project! I (unintentionally) learn things that help me at work every time you post one of these.

[subpixel]

Great content. I'd really like to watch a deep drive on network setup that covers separate networks for Ceph (>=10Gb), VM access outside of the cluster, and a intra-cluster management network (<=1Gb)

[flahiker]

Great Video. A suggestion to test your setup is to simulate a power outage and see how the cluster responds. I have a 3 node ProxMox cluster running Ceph and I am setting up an extra cheap PC to run NUT to manage the UPS. My goal is to simulate a power outage (un plug the UPS) and have the cluster "Gracefully" shutdown and restart when power is restored.

[MickeyMishra]

I love it when old hardware gets used. Sure it may take more power, but in my experience? Mixing and matching may be hard to do? But its overall a better idea for uptime. Chances that 3 sets of gear fail at the same time from different product lines? Yea, not going to happen!

This is wonderful that more people are using the DAC cables. I stepped away from home server stuff years ago, but its nice seeing other folks keep the hobby alive.

[GapYouIn]

Good stuff! Good to see someone show that you can just grab the commodity hardware from wherever and make a cluster that is fault tolerant lol. I run several ceph clusters and it definitely gets better with scale, but still a lot to be desired. Interesting to see it so well integrated with proxmox.

[dtom]

Love the content. I currently have a 4 node cluster in production with PVE and Ceph. I currently have VM storage on SSD’s and cold storage on spinning rust with ssd db/wal, but would like to see something on ec pool. I know you can create 4+2 on 3 nodes by sending them in pairs, but I can’t quite get my head around the CRUSH rule for it. The logic behind this is to increase storage efficiency.

[cmacpher]

Since you asked, how about reliable VL intensive OLTP database using no data loss log shipping, very fast failover on multi-node active/passive HA cluster config with enterprise class database products like Oracle and HANA. Hit it hard, every server hardware, OS, network, database, heartbeat, corruption, simulated WAN, DC environment, and disaster failure scenario you can come up with. Show that this product can compete in enterprise environments. Perhaps it can. Enjoy the challenge. I look forward to viewing more of your videos. Amazing talent you have, loved the chickens.

[lquezada]

You’re a champ - thanks for all that information in such a short time. I am currently working with passthrough, trying to get my rtx 2060 to detect on a windows 11 vm. Hopefully I can figure it by this week.

[rocketi]

I love the random chickens behind you. Great content!

[MikeDent]

Amazing knowledge and enthusiasm. I think Proxmox should employ you.

[CDWD-Project]

Great video, I just finished getting our last VMs off a way overpriced Nutanix cluster and I was looking at putting Proxmox on it and you have me sold. No I did not buy the Nutanix as they got it 6 months before I took over.

[RyouConcord]

god dang wizard indeed. your content is rad man

[martyewise]

Thanks! Super vid! Searching for parts and planning construction of my own PVE cluster.