any recommended 2.5G or 5G PCIe X1 network card

Message

jpsollie · Post by **jpsollie** » Sun Apr 12, 2020 7:09 am

I am looking forward to the upcoming 2.5G or 5G ethernet speeds, and I'll try to upgrade my gentoo NAS with a NIC which suits my needs.

- PCIe X1 <-- the mobo only has pcie x1 slots left
- 2.5G or 5G <- I am not ready for 10G ethernet
- low amount of CPU interaction required (eg: TCP checksum offload, low amount of IRQs compared to the realtek 8111 network traffic, etc ...)

Any ideas?

thx

mike155 · Post by **mike155** » Fri Apr 17, 2020 3:21 am

What about RTL8125 based PCIe cards like the Delock 89531?

chipset: Realtek RTL8125
external connector: 1 x 2.5 Gigabit LAN RJ45 jack
internal connector: 1 x PCI Express x1, V2.1
driver: a RTL8125 driver seems to be included in the Linux kernel (vanilla 5.4)
nice price tag: those cards cost less than 50 € / US-$.

axl · Post by **axl** » Fri Apr 17, 2020 4:02 am

Could someone please explain to me how a 5G network card would work without a 5g switch? What would you connect to it to get those speeds? Sorry for my ignorance.

jpsollie · Post by **jpsollie** » Fri Apr 17, 2020 5:10 am

About the Realtek chip:
Is it better than the RTL8169? My mainboard has a RTL8169 built in, and I'm definitely not happy with it: offload features are limited, no MSI-X and high amount of IRQ interuppts
About your 5G question:
Yes, I do not have a 5G switch yet, the idea was simply 'if I have to buy a dedicated NIC anyway (because I'm not happy with the built-in), I'd better take one which can follow the 2.5/5G move in the upcoming years

axl · Post by **axl** » Fri Apr 17, 2020 5:13 am

jpsollie wrote:Yes, I do not have a 5G switch yet, the idea was simply 'if I have to buy a dedicated NIC anyway (because I'm not happy with the built-in), I'd better take one which can follow the 2.5/5G move in the upcoming years

You mean directly connect it with another computer? Wouldn't you need another network card just to get out of the peer to peer thing?

jpsollie · Post by **jpsollie** » Fri Apr 17, 2020 6:22 am

Absolutely not. I'd keep my gigabit switch, but once the 2.5 or 5G becomes mainstream and less pricey (which I expect in the next 2-3 years), I'll buy a multi-gigabit switch

axl · Post by **axl** » Fri Apr 17, 2020 6:25 am

Ohhh. Ok. Thank you for your clarification.

I have one these 5G on-board network cards. I was thinking of ways to make more of it. Anyway, thanks.

Anon-E-moose · Post by **Anon-E-moose** » Fri Apr 17, 2020 10:44 am

Highly unlikely to find a 5g speed on an x1 card, I would think that 2.5 is the fastest you'll get especially on pcie rev2.

You'll also need a router capable of 2.5/5g speeds to use it at that speed and you're stuck with the slowest speed that you connect to.

Buying now, with an eye on future proofing is, a little foolish, IMO, at least as far as 2.5/5g speeds, it's relatively new and I don't think the specs are set in stone yet.

mike155 · Post by **mike155** » Sat Apr 18, 2020 1:15 am

Anon-E-moose wrote:at least as far as 2.5/5g speeds, it's relatively new and I don't think the specs are set in stone yet.

Well, IEEE Std 802.3bz-2016 (the standard for 2.5GBASE-T and 5GBASE-T) was approved in 2016. See: http://www.ieee802.org/3/NGBASET/email/msg00996.html.

szatox · Post by **szatox** » Sat Apr 18, 2020 10:09 am

Does your NIC really have to be a single lane one?
PCIe has some HA built in, if you put a "wrong" size adapter in a bank you either underutilize a big bank or degrade performance of the adapter, but it is supposed to work anyway, so maybe it's not as big problem as it seems?

Another thing, have you considered building your home network on infiniband?
I've got a few 16Gbps second-hand adapters to play with them a bit... They were dirt-cheap, and work pretty well, especially if you can leverage RDMA. E.g. NFS can do that. Had some issues getting wires for them (FU, paypal) but there used to be quite a few sellers from China on eBay.
As a bonus, infiniband adapters can be switched to ethernet mode too (16G IB -> 10G Eth), though plugs are not compatible. Chances are it could work with SFP modules though, there is some room for research.

Post by **NeddySeagoon** » Sat Apr 18, 2020 10:34 am

jpsollie,

You get what you pay for. Realtek are not known for high end NICs with all the bells and whistles.
As the NIC capability increases, so does the cost. That usually means looking at Intel, who don't have a 5Gbit part yet and the 2.5Gbit part is still newish.

Why is the lack of offload and MSI-X a problem?
Do you have any performance measurements to show how it hurts?
What would you be doing with the resources that these additional features freed up?

mike155 · Post by **mike155** » Sun Apr 19, 2020 12:48 am

axl wrote:Could someone please explain to me how a 5G network card would work without a 5g switch? What would you connect to it to get those speeds? Sorry for my ignorance.

There are consumer-friendly switches that support 2.5GBASE-T and 5GBASE-T. Take a look at the Netgear switches XS505M, XS508M, XS512M or GS110MX, for example.

jpsollie · Post by **jpsollie** » Sun Apr 19, 2020 6:08 am

NeddySeagoon wrote:jpsollie,
Why is the lack of offload and MSI-X a problem?
Do you have any performance measurements to show how it hurts?
What would you be doing with the resources that these additional features freed up?

The NIC is attached to a X570 south bridge, which is connected via an X4 PCIe link to the CPU. On this link are also a x8 SAS controller and the 2nd GPU (next to the onboard stuff such as sata, usb, etc ...). Lower offload features means more interrupts, which cause problems due to the limited PCIe bandwidth available, and the AMD non-transpartent PCIe bridge seems to switch devices, not buffer them and push all data over the link. That's why.

axl · Post by **axl** » Sun Apr 19, 2020 6:18 am

mike155 wrote:
axl wrote:Could someone please explain to me how a 5G network card would work without a 5g switch? What would you connect to it to get those speeds? Sorry for my ignorance.
There are consumer-friendly switches that support 2.5GBASE-T and 5GBASE-T. Take a look at the Netgear switches XS505M, XS508M, XS512M or GS110MX, for example.

Thank you for that. I feel old. From dialup... to 5G. Man the time flies.

I still remember the T connectors from old bnc networks. I love playing with those like they are lego. I suppose everyone did.

Post by **NeddySeagoon** » Sun Apr 19, 2020 11:06 am

jpsollie,

It sounds like the NIC is the least of the problems in that arrangement.
All that on a 4 lane PCIe link.

What else can you rearrange?
A saying to do with Grannys and eggs comes to mind since you appear to have done your homework here.

Anon-E-moose · Post by **Anon-E-moose** » Sun Apr 19, 2020 11:11 am

A pcie x1 card won't alleviate any bandwidth problems, it will only add to them.

What's the motherboard?

Edit to add: Having your kernel set up with proper preemption (tuned for your use case) is also important.

jpsollie · Post by **jpsollie** » Sun Apr 19, 2020 12:07 pm

NeddySeagoon wrote:jpsollie,

It sounds like the NIC is the least of the problems in that arrangement.
All that on a 4 lane PCIe link.

What else can you rearrange?
A saying to do with Grannys and eggs comes to mind since you appear to have done your homework here.

The mobo (gigabyte X570 UD) also supports PCIe bifurication, so maybe I should just split the x16 slot attached to the ryzen CPU into 2 x8 slots. I'll be able to attach the 2nd GPU directly then, probably a better solution