100 Gig Networking in your Home Lab

-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------

Become a Channel Member!

Support the channel on:

My Hardware:

My Studio Equipment:

0:00 Intro
0:23 The Mikrotik CRS504-4XQ-IN 100G Switch/Router
1:02 Why I bought this
3:48 40/100G is cool
4:11 RouterOS interface and setup
6:04 Let's run a speed test!
6:29 Why isn't it fast?
7:09 I have a problem...
7:44 100G networking!
9:10 Was it worth it?
10:29 Okay I want better performance...
11:17 Let's set up SMB Direct and RDMA
12:22 Conclusion

[JeffGeerling]

8:31 time to upgrade your workstation, then!

[kumakell]

For the record, FS doesn't just "seem" like a good store, they are pretty much THE store for fiberoptic networking. They make EVERYTHING for fiber, and they're just about *the* go-to for third-party transceivers for the entire market. Their switches are used in enterprise as well, just at a much lower volume than their optics and accessories.

[Bytional]

40G is a bit awkward right now, homelab users mostly not there yet but enterprise users are upgrading from it to 100G already.

[TrTai]

As a network engineer.... Just use the switch. The point to point to bridge is possible but it brings nothing but pain.

[cjchico]

I use this switch for vSAN, breaking out to 2x 25GbE per each Dell server. Works great and haven't had any issues yet. RouterOS is very odd and definitely takes some getting used to.

[TheRealClutch]

I now have a way to describe my hobbies, crippling inability to be content with what I have.

[ajhieb]

I had a similar upgrade path, where I started with 3 servers connected via 3 dual port ConnectX-3 cards, but wanted to connect to some other stuff, so I found a Mellanox SX-6036 40Gbe switch on eBay for about $150 and I've been thrilled with that thing. It's not "sit next to it" quiet, but it's not even close to being the loudest thing in my rack. If you're not averse to used equipment, it's a great option.

[TDCIYB]

iSCSI should be the best option to improve speed for editing. And supported by both TrueNAS and Windows.

[AndrewWells]

9:30 I'm disappointed that the next cut after "that would be stupid, right?" wasn't with those PC parts on the table. Current tech really isn't designed for single transfer speeds at that level. That switch was probably designed to be at the core layer of a network. Still fun to play with though.

[torgrimt]

looking forward to your review of the threadripper in a week ;)

[CassegrainSweden]

This must fall under the category: do I need it, hell no, do I want it, hell yeah 😂

[iroesstrongarm]

That "oh god" from you wife definitely had undertones of her full awareness that too much money was spent. 😂

[MM-vlic]

I've been using the ConnectX-3 40/56Gbs for 5 plus years with a Mellanox SX6036G ("G" = I found when testing, it's best to use 100G+ ram disc on each machine to eliminate storage device also being a bit retro... Asus X99 workstation MBs with Xeon e5-1660/1680 v3 over clocked 4.2Ghz with 2666 ECC ram 40 PCIe lanes or more with the PLX switch chip, other MBs are Supermicro X10 series... Remember those Enterprise cars need air flow.... I use retro desktop cases with a side fan to cool NICs, Raid/HBA cards and NVMe drives....

[phillipmartinez]

When's the threadripper video coming out?!

[NeilHyndman]

Amazing video! I enjoyed watching it and I learned a LOT! Thank you!

[juniornunes]

the "see i fixed the saggy servers!!!" gesture was great lol

[chaosfenix]

I love this video. Yes I know there isn't a ton of use for it now but I think there really could be if it was more common. I would love to see some sort of direct video output over the network solution. HDMI 2.1 is only 48Gbps and Displayport maxes out at 80Gbps. Currently remote video is handled by encoding the video signal with something like h.264, sending it on the network, and then decoding the video before presenting the picture on your display. I would love for there to be an option where we could skip those encoding/decoding steps as those add extra latency and can introduce compression artifacts, and instead just send the entire video stream on the network with some overhead left over for USB peripherals. Again, I know this is a stupid upgrade now, but I would love to see this tech better utilized in the future. We have been stuck on 1Gbps for so long that it limitation itself has impacted what we can do with the bandwidth.

[thenextension]

FEC74 is a protocol for communicating to devices that congestion is occurring and to adjust behavior to relieve the congestion. There are a few other variants of it like FEC-RS and IEEE FEC

[RandomTechWZ]

10gb networking seems to be the sweet spot for home labs and your wife sounds exactly like mine when I tell her changes I made to my home lab lol.

[jeremybarber]

Good stuff right here. Greatly appreciate the ride along as you learn.