Windows file transfers are FAST

-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------

Become a Channel Member!

Support the channel on:

My Hardware:

My Studio Equipment:

0:00 Intro
0:51 Fixing my issues from before
1:08 PCIe lanes issue
2:09 Fixing the slow file transfer speeds
3:46 Testing our speeds...we need more
4:33 Now that's fast
5:12 Real world file transfer speeds
5:39 My thoughts on the results
6:45 Conclusion

[SpoonHurler]

100gbps... I could move my meme folder in... well still minutes but less minutes 😂

[Koop]

Hey Raid Owl I have to say this- I LOVE your videos. You explain the issues you run into extreme well and I really appreciate how you walk through and troubleshoot your issues. I think it's awesome that you listen to the feedback of viewers and react accordingly. The last video you had on this everyone said "time to upgrade your hardware!" and you saw that, thought about it, and did it your way while explaining why you made the decisions you did. You're extremely relatable- you do things for the fun of experimenting with it and explain your mistakes and/or lack of knowledge with humility. We all benefit and learn with you because of it, thank you. So often you'll see content creators who curate their content to show them in the best light- the smarted most capable people who never make mistakes and never show the issues they encounter even if there were millions. Of course there isn't anything wrong with presenting concise content like that- a 10 hour video of troubleshooting wouldn't be all that entertaining- but I feel you personally have a fantastic mix of explaining issues while still being concise.

Just want to say again I really appreciate how you create your content. I personally have a home lab setup very similar to yours. I am considering the pros and cons of the same exact switch you bought. I am curious about the same levels of performance that you exactly are trying to achieve. I'm gaining a lot from your work and I just have to say again- appreciate you so much! Keep it up!

BTW if you upgrade from your MikroTik switch help me out and sell me yours! haha. Joking but maybe not really lol.

Also REALLY funny comedy in this video. You had me laughing as the Threadripper slowly faded in. Amazing. I would LOVE to see you do this workstation build as that's also something I've been curious about doing. Let's see it!

Best of luck and looking forward to more videos!

[trevorlyman]

If you’re looking for more pcie lanes for a more reasonable price consider an Epyc system with a gen 2 or 3 processor. The motherboards start around $500. I managed to get a 7352 24c for $200 on eBay. Then you can get all the gen 4 lanes you could need! Alex from the Self Hosted Podcast has a good video on this.

[MarcoGPUtuber]

This is as fast as me clicking the notification of a Raid Owl video

[npoitevin]

Wait… My 10Gb router and switches with optical links are not the fastest ones anymore ? Dang I feel so low

[DeFi-Macrodosing]

My gigabit connection is feeling unloved right now.

[DPCTechnology]

Great stuff, great storytelling, love it!!

[rustusandroid]

For RoCE to work from end to end, your switch needs to have Priority Flow Control enabled. Without that, RDMA is useless. Jumbo frames help a bit as well. But what do I know, I've just worked at datacenters for the last 15 years...

[RGMLIFE]

would be interesting to deepdive into the Truenas performace stuff, disable compression etc, windows server is such a license hell

[Prime_Tecnologia_Automacao]

I have a feeling I'm going to see part three of this saga, ... and.. with a happy ending....lol

[kevinoneill]

0:00 "I have a problem..." finding a shirt that didn't come from the bottom of the pile of (clean?) laundry. Jkjkjk, I'm just jealous that I'm not your "full 100G connection with RDMA that actually gets 100G speeds." Fun video. Nice work, thanks!

[TheXDKane]

If I remember correctly, Windows file copy only uses a single cpu core, a tool like ChoEazyCopy can use multiple cores for file transfer. So me guess is that your 100 gig network is limited by software design. Hope this helps.

[SOF]

Seeing those speeds is crazy to me, meanwhile I am happy with my 2.5Gb transfer between my desktop and my server.

[hugevibez]

Here's the thing, the main goal of 100gbps is not to get the best transfer speeds possible, the goal is to help with things like network congestion. It's about getting non-blocking throughput when you have thousands of clients connected on the network at the same time. The ultra high-speeds are mostly meant for the network. Your analysis is correct that you will always be dealing with overhead, and this overhead only seems to increase as network speeds go up. Using RDMA with SMB Direct is basically the best possible scenario in terms of overhead, as you already described. What it is useful for though, is being able to handle ceph, live VM migrations, internet traffic and file transfers at the same time and having it all run smoothly. If you want to achieve this properly you are going to have to seriously rethink your network topology though to create a true fabric infrastructure, since you want to make sure your traffic is non-blocking.

[Yock]

When running SSDs/NVMEs in a pool under Windows you need to remember to run through a lot of hoops to get TRIM to work.
I found that out the hard way my self.

[chaosfenix]

I am glad you are upgrading your setup. It is kinda weird how for the longest time we have always been told that you don't need faster than gigabit because your WAN connection is a lot slower than that anyway. Now I have 10G internet at my home and am actually needing to upgrade my home network because most of the devices on the network don't actually support that. I kinda feel like your home LAN should be at least 2-3x what your wan connection is which means I need to seriously do some upgrades. And do I need those speeds? Sure don't but I do try and utilize it as best as I can. I have a tor node setup and I have my own nas and server setup to account for as well. I am really hopeful that now that Gigabit and faster speeds are becoming more common that we will finally move on from the idea that gigabit is fast enough. I would love to see SMPTE 2110 proliferate for virtual systems and I do think that it is getting easier and easier to just run your own homelab.

[austinclement]

Always a fan of what’s coming from RaidOwl!

[jensunger]

Thanks for confirming!! i went through a similar journey. I switched from 10G to 100G and had all running with x8 and x16.
It was a Rabbitbhole to get to at least 3GB/s with SMB while i expected lime you to have 7 to 10GB/s.
It was so complex and I wonder, , are we the only ones??? There is almost no good info out there and I think we are at the forefront of Homelab.

[MMWielebny]

RoCE requires not only nic support but switch support as well to make 100 gigs possible as switch will make proper QoS and pass packets faster. On some conferences some ppl claimed that dpdk is also need but I am not sure if this is True... Also if you are using RoCEv1 instead of RoCEv2 you should get slighty faster speeds.

Intel with their iWARP make claims that iWARP needs only nic support without switch but never had chance to test it out.

[SnowReborn]

i just migrated 80TB of data abandoned NTFS in favor of TrueNas ZFS, but... smb and nfs(better than smb) is just unsable for any amount of high IO application. too much overhead, the only workaround is iscsi.... but puts me back to NTFS... are there any workaround to utilize rdma /smb direct with truenas zfs pools?