Fully Routed Networks in Proxmox! Point-to-Point and Weird Cluster Configs Made Easy with OSPF

Are you playing with Proxmox clustering, but want faster networking without paying for multi-gig switches? For small clusters, sometimes it can make sense to use fast point to point links between nodes. This could be in a small 2 or 3 node cluster, where you can use dual port 10 gig cards and direct attach cables without a switch. Maybe you've got a wacky 5 node cluster with quad port gigabit cards on each node, and don't want to buy a 20 port switch and do link aggregation. Or maybe you want to be the crazy guy who uses intel NUCs with thunderbolt between them. Whatever your use case, this video will help you setup your fully routed cluster network properly.

This is accomplished by creating point to point links between each node in any topology you can think of, and allowing OSPFv3 to exchange route information across all of the links. Once we have configured OSPF on all of the relevant interfaces, the cluster route map will automatically be generated and updated if any links go down and the shortest path will be chosen based on link speeds and priorities.

To see the example FRR configuration I used, see my blog post:

Feel free to chat with me more on my Discord server:

Timestamps:
00:00 - Introduction
00:38 - Concept
02:13 - Install and Configure FRR
11:58 - Testing
14:07 - Crazy Setups

#proxmox #ipv6 #routing

[eDoc]

Half your videos are things I've thought about doing but never put in the effort to try. This is no exception. Keep it up! I imagine a not-too-future video will involve inter-VM traffic passing through these links.

[Ownermode]

Great video, this was just what I was looking for. I was thinking about modifying the routing table by hand. Totally forgot about OSPF!

[Darkk]

Awesome video! Good use of OSPF for fault tolerance. :) Also, brilliant naming for your internal IPv6 fd69:beef:cafe::555. I know we're allowed to use A through F for the address but that is simply a genius way of making use of those limited letters.

[jaykavathe]

Can you please make another video on moving existing ceph network onto the ring network you just created please? Will be very helpful method to understand ceph network configuration

[GeoffSeeley]

This was great! Learned something new today even after 25+ years in IT.

[KeithWeston]

Thanks! Consistently the best information presented clearly and thoroughly.

[randyandy-og]

Man I got a bunch of NICs coming in the mail to set this up, but I'm also a little bit stupid. Everytime I google this people are just like "yeah you just connect your nodes directly", which...like...duh...So having someone ACTUALLY walk through the process was so helpful

[swiftlabbuildstuff]

I found this video excellent. I usually learn by example but I was hesitant to give the "ring network" a try since all the examples I found were 3 node ring networks. This video gave me the confidence that I could make this work with a 5 node node Proxmox cluster. I already have dual 10g LACP bond physical switched networking per node, which is plenty fast. After configuring the dual Thunderbolt on each node in a ring, I found it pretty easy. Even with a node (and its links) down, any other node is no more than a few hops away. Maybe still faster than the 10g LACP bond. Need to test though. I love that you show how to run the iperf, traceroute and nload commands for checking the connectivity. Very easy to follow. My next challenge is that this Proxmox cluster already hosts a Ceph cluster using IPv4 addressing on the 10g bond, but now I want to move the Ceph backend network over to the ring network. From what I can find, Ceph can't run dual stack - needs to be all ipv4 or ipv6. I'll be looking forward to your blog post/video on Proxmox Ceph running on the ring network. Once again, thanks for an excellent video!

[yankee-in-london]

Next level. Love your videos as it really helps push me beyond the basics and I love it.

[geesharp]

Man, it's been a while since i configured OSPF on network equipment. Brings back memories.

[berniemeowmeow]

Great video! Very cool. Appreciate you going deeper on these topics. Love learning new stuff.

[yasaralzakout]

You always have a great step by step videos. Thank You.

[ExillNetworks]

Awesome work! Fantastic video!
At 5:23, I didn't know that!
I have been playing with Linux for years, but I dident know this! Thank you so much!!

[LampJustin]

Awesome one! This is exactly the setup I planned for my Homelab and my 40G adapters. The only difference is that I'd go with BGP (unnumbered) instead of OSPF :)
Btw if you use vtysh you can build the config like with cisco. It's so much nicer with tab-completion and the occasional "?" for help. To save the config use write . The changes will be applied instantly so keep that in mind.

[westlydurkee]

A video series on networking like setting up a local DNS, LDAP, and Samba Active Directory server would be great. I really like the way you explain things keep up the good work!

[TheOnlyEpsilonAlpha]

Okay, i came to that video because of something else: My Proxmox Instance (a test one) has the issue, that i can't ping anything outside the network.

But i was stuck into the video cause i noticed: FINALLY someone with network expertise, the "beef:cake" IPv6 is freaking funny, and it's interesting to have failover routes if the major route fails.

But my initial issue ist still there... and i see youtube recommends me your Video "Proxmox Networking: VLANs, Bridges, and Bonds" that looks more what i'm searching for!

[postnick]

I got some 2.5 gigabit cards. So far I just direct connect between proxmox and Trunas for slightly faster backups and nfs.

[simo]

Most video are basic. This is advanced. Awsome.

[RobertoRubio-ijms]

Each time I watch one of your videos, my entire datacenter goes on overhaul. Amazing content. Do you do consulting?

[Felix-vehs]

Now I know why there are loopback addresses on OSPF and BGP ^^