Renumbering my PROXMOX Cluster because of an IP Oops!

Sometimes you need to change the IP addresses on your Proxmox cluster, and it's never a particularly fun time. But, come along for the tale of my late-night frustration as I realized I'd have to renumber my cluster, and learn from my mistakes so you don't make them too.

In this case it was caused by Comcast changing my IPv6 prefix (something they haven't done in 2 years), but it can happen any time you need to move IP ranges for any reason. So, make sure you have backup ring networks for Corosync, and be aware of how to bring the cluster back from any accidents.

Feel free to chat with me more on my Discord server:

[matthewschuster]

Man, I really enjoy your content. I also really appreciate the manner in which you reply to others in the comment section. You have lots of patience. Thank you for what you're doing.

[NetBandit]

Maybe the only tech tuber who doesn't pretend like IPv6 doesn't exist

[ChazTFC]

Yeah back in 2017 I setup a ProxMox cluster for a LAN party, had to re-IP right before the event... Glad to see it's still broken.

[Felix-vehs]

1:00 you can also get a IPv6 PI /48 prefix as a private person, but you need to apply for your own AS number, and get your own IPv6 prefix from a LIR of your choice.
Then you can find a hosting provider which supports BGP and PI address space (like Vultr) and peer over a IPsec+GRE tunnel. I plan to do this but still have to learn a lot to do this. :)

[salat]

I get a new prefix every 24h :) I think it's a good habit to also use ULAs (RFC 4193) in a "managed" network - they are under your control and never need to change. Combined with another address family and/or OSPF is a great way to stay connected..

[EverettWilson]

I reIPed Proxmox once. Angry seems like an understatement.

[hawwestin]

Been there, done that. In my case investigation where those addresses were hardcoded took really long time. Plus some of those files you cannot edit until you stop corosync and pve-cluster and start local pmxcf.
It have FULL CIDR so even expansion of network from /24 to sth bigger will brake some things. I had strike twice with this. First static IP address made by infra was made by somebody lazy and he make arp static entry in dynamic pool. And some day different new device negotiate same ipv4. Second time network CIDR change from 24 to 21.
It will be good to post those instruction you made as some of those things you mention are not obvious until they strike you

[markstanchin]

Hello, yes another question, when I try to edit the files you suggest I get permission denied, and this is actually on the node itself. Node 1 Actually. The files still have the old 192.168 address. I couldn’t set up another back net work because I don’t have the interfaces in two of Lenovo nodes that I have. I guess there’s a way you can add a interface with USB. Also I believe there is a port they can be installed to main board you can swap out the current module installed which in this is a serial port.

[andre_warmeling]

Very good content and I am glad you could recover your setup... But why would your external IP interfere with your LAN? What's the reason for having the cluster on public V6 space?

I'm sorry if that's a silly question, btw.

[RobertRidleyE]

I suggest adding ULA (unique local addresses) to your network. All local configuration will never change (unless you make the change) and the address range is non routable on the Internet. (fd00::/8). If you ever have a prefix change again, the only thing that will break are those services pointing to those ISP assigned prefix address ranges.

[markstanchin]

Hello, thanks for the video. You have one of the only videos on this subject. My issue is similar but self-inflicted I changed my home net work from flat 192.168 to subnets 10.10, 10.20 ect. My main reason I ran out of IP addresses. So I did this without really thinking it through with the multi node proxmox cluster ceph shared storage etc.. so I’m like now what. So I went online for info on changing IP address. Your video came up. I was hoping for more of a live tutorial on the commands, etc. been I’m very new at this. I still have the other network running sort of it can’t access the Internet, so I can access the proxmox etc.. none of the VM’s can access the Internet and the nodes can’t talk to each other. I’m just wondering what would be my best method should I create the back network as you suggested I’m not sure what would be my best method. Thanks I appreciate your time and knowledge.

[MrRhysstevens]

Why don't you add the new ips to the corosync config 1st? You can have more than one. Then the configuration would replicate throughout. Then do the other changes. Then remove the old corosync ips once working on the new net work?

[Darkk]

Ya, I don't use IPv6 internally when it's tied to the WAN unless been given my own static IP from the ISP. Not fun to wake up one morning to find things broken and figure out what the hell happened. Changing the IPs in ProxMox cluster is tricky but can be done. So for my servers I rather stick to IPv4 internally where I have full control of it. IPv6 can be attached as secondary so it would not mess up the critical services.

[Mr.Leeroy]

Not sure if I understand the subject fully, but why chorosync net was in public IPv6 space anyway?
If it is a management network, couldn't it just be setup in local address space?

[frzen]

I changed the ip on proxmox and it still replies to pings on the old ip address as well as the new one.... ipv4 though but I burned over an hour on this today trying to figure it out

[superslammer]

What is your advantage to running ipv6 at home? DNS sounds like it would be way way easier to fix something like this with.

[AlyssaNguyen]

It took me way too long to figure out that every time you said "etsy, " it was supposed to be "et cetera." 😒