Home Networking: 100TB 10Gbit Server - What hardware do you need?

I'm finally ready to proceed with the server new 100TB server build. In this video I highlight all hardware used and why. This includes everything from disks to motherboard and CPU including discussing PCIe lane usage, CPU clocking and memory used.

Jump links:
01:10 Home Usage Server, Not Enterprise
02:33 Using a desktop motherboard and dividing PCIe lanes
04:19 PCIe x1 Graphics card
06:45 Final PCIe slot layout
08:00 SAS cables
08:50 Fan needed on PCIe cards or not?
09:45 Stock cooler and fans now, Noctua later!
10:10 64GB memory
10:50 No ECC memory rant
12:20 Power Supply
13:00 OS, VM/Container SSDs, L2ARC
15:50 Power Loss protection SSDs for ZIL/sLOG
17:40 HDD storage and RAID levels
20:20 Preparing hardware before install

░▒▓[Associated blog post]▓▒░

░▒▓[Shopping links]▓▒░
Most links will ask if you are looking on Amazon or Aliexpress. I prefer Aliexpress myself. These are affiliate links and help me out greatly!

Motherboard and CPU:

Memory:

Storage:

Graphics card:

Network 10Gbit Equipment:

Network 10Gbit Cards:

SSD Hot Swap cage:

LSI Card + Cables:

Other:

░▒▓[Music]▓▒░
Intro&Outro
TheFatRat - Xenogenesis

[rdwatson]

Thanks for going through your decision process on components. That's a key step many videos skip, they throw together hardware without any discussion of why or just say "because that what forums recommend".

[nephilimcrt]

I'm an experienced IT engineer and I have no idea what the hell you're talking about :'D
But you obviously know what you're doing and why you're doing it, so no reason for others to criticize, I'd say.

[gordslater]

Hi - old vid but very good one full of useful info.

I'm currently buidling a similar Proxmox box but using old Socket 775 (!) kit - 8GB non-ECC and several 500GB drives, plus a 128GB SSD, on onboard SATA only. This is my second build on this old hardware - the first runs the intended VMs just fine (mainly OpenWRT or Alpine boxes so very light disk space or usage, the occasional minimal debian server VM)

Just wanted to say, apart from budget and performance, just about *everything* you say in this vid is very relevant to the old, free (unused) hardware builds I'm currently doing. It's just on a different scale and zero budget :)

[fpul]

First time ever seeing your videos, but you just got another sub from me. I'm super excited to see the rest of your videos on this build. I have a very elementary education on servers, but feel like I can learn a ton from you since you do so well explaining things. Looking forward to binge watching your stuff!

[Actionjack]

Great heads up on lane planning. It really helped.

[lukasibanda]

Great server builds. I want to build VMWare and Proxmox servers for CUCM, TrueNas and DLP. Keep it up. Next time show us how much in total you spent to determine whether to buy or build a server

[jeverett]

Great content. I really appreciate the diligent planning and solution engineering to make optimal use of what you have. New subscriber here and will soon be using your affiliate link for some Amazon purchases.
When recycling old workstation motherboards to home servers, I too have opted for (in my case) a PCI VGA adapter when I wanted to reclaim PCIe slots/lanes for more useful cards. I've always wanted to play around with trying to pipe grub/linux console out to either a motherboard serial port if available, or a USBSerial adapter if that is the only RS232 option -- which would avoid spending a slot/PCIe lanes on (rarely used)) server video output (but then newer server motherboards with IPMI and/or embedded video took away the necessity to be that inventive).

[marcosoliveira]

Man! A lot of very interesting information throughout the video. Thank you!

[fatalty]

I absolutely support your choices of using old desktop hardware. This is not a production environment, so cutting corners when your use-case is home/personal/fun/etc. is completely acceptable. The only reason I don't run my old desktop hardware in my rack is that I get old enterprise hardware from my employer every year or so. Most of it is 5-8 years old, hurts my power bill a bit and is a bit noisy, but then again... Free is free. :D

[mdd]

A very fine and detailed summary of hardware needs!

[JavaManRules]

Great stuff!! Keep-up the good work as I do enjoy your channel!!!

[victorbart]

Not my choice of HW but I see you ideas behind it! Love the see more of it :) Why not a 16 port SAS controller?

[stephenbruce]

I am looking to build another server and you have pointed out a few things that honestly I never gave much thought to so I figured I would subscribe to see your build and look for some ideas to take into account for my next server. Before any new build I like to put quite a bit of thought into it so that I will be good for the long haul. My primary server is now 7-8 yrs old been running 24-7 with very few problems but nothing lasts forever and so for a while now I have been thinking about building a replacement

[chadwilkins]

Good overview. I thought I would be able to do more with an APU but they actually end up hurting you by taking away all the channels. Sales guy saying there is no difference between the B570 and B550 was not accurate. Those 4 channels are needed with network setups.

[scheimong]

I ran into the same issue with limited PCIE lanes in my most recent NAS/server build. So what I did was put in a temporary GPU (I have a spare GT730 lying around) so that I can configure BIOS and install my OS (RHEL-8), then I updated packages and enabled SSHD and VNC Server. Afterwards I can just pull out the GPU and run the machine headless, so I have all the PCIE lanes available to my HBA and NIC. Whatever further configuration and maintenance I have to do can be done over the network.

Now granted, if I am to do any configuration with my BIOS, I would have to pull out a card and put in my temporary GPU again. But just like your use case, my machine is not a critical system and can have some downtime if necessary.

Just note that while modern CPUs, consumer or enterprise, technically all support headless operation, some consumer motherboards may impose weird restrictions that prevent you from booting without VGA. It's also often difficult to find out whether a particular model of consumer motherboard supports it, since it's a very niche inquiry so manufacturers usually don't put that information in the most noticeable spot. So if anyone reading is looking to run their setup this way, make sure you read the motherboard manual (which you can usually find online in PDF format) thoroughly BEFORE you make the purchase.

In my case I'm running R5-3600 on Asrock B550m Steel Legend. This board is particularly nice for my use case because it has 6 onboard SATA ports (which means I only need 1 extra HBA to handle all 12 drive bays and still have 2 ports to spare; it supports headless boot; and the greatest of all, it explicitly supports ECC UDIMM memory. FYI in case you are looking for hardware recommendations. I got the CPU and MB for a total of 250 USD, which is an absolute steal IMO.

[HuMaNiTaRiAn]

Can you do a wthout zil and with zil cache to compare? I'm running with just disks and wondering if I would see any significant improvement in adding a zil cache

[erwin]

I like your idea to use the ZIL and L2ARC for 3 pools… Would you do that for special devices (metadata) as well?

[ShintaSaINt]

Thanks for your time, knowledge and content! Looking forward to test of server build, please bring more proxmox content, I've been hearing about it but don't know what she why is important. Regards Shane trinidad

[Uplinkpr]

I think I personally would have gone with one LSI card and a SAS expander to keep your pci lanes in check, but thats me.

[SyberPrepper]

I've been looking forward to this build. Thanks for taking us through your choices. If you were buying new, but still trying to save money, would you go with a 1950 threadripper and newer motherboard to reduce some of your restrictions?