Gigabit Ethernet slow/unstable - recommended gigabit cards?

Ge(nto)org · n00b Joined: 09 Aug 2004 Posts: 17

Hi!

The performance of my Gigabit Ethernet is really bad: When I try to scp a file from one host to the other, it first starts with about 15MB/s but immediately decreases step by step to 3MB/s. A NFS transfer has similar problems except that it is also stalled every Minute for a few seconds, then the transfer again has about 15MB/s which decrease again until it is stalled and again...
With 100MBit cards on the same hosts I get constant 11MB/s with scp and NFS!

I have a onboard Marvell 88E8001 using the skge module, RTL-8169 cards using r8169, a Gigabit Ethernet Switch and Cat6 cables. I have tried the following setups:
skge <-> switch <-> r8169
r8169 <-> switch <-> r8169
skge <-> r8169
r8169 <-> r8169

Also the RTL-8169 cards have a LED indicating the speed and it says that they are connected at gigabit speed.

I'm really stuck now, and don't know what to check next :cry:

Any help greatly appreciated!

alex.blackbit · Advocate Joined: 26 Jul 2005 Posts: 2397

is maybe the cable crap?
i had the problem with the nfs stalls before. i never solved it. if you find any hints, please let me know.

agent_jdh · Veteran Joined: 08 Aug 2002 Posts: 1783 Location: Scotland

If it's any consolation, I've got 2x r8169 based cards, netgear gigabit switch, cat6 cables, and it's pretty flakey as well. A lot of the time on booting it doesn't even use gigabit speed, the switch light shows 100Mbit. Often requires _several_ reboots to get 1Gb. The transfer speeds varies wildly as well - sometimes it screams, sometimes it crawls, and on any large transfers it wobbles between the two. I can't get the driver to work with large frames (mtu should be able to go up to about 7k) either, which is a shame.

To summarise (!), I don't think rtl8169 gigabit ethernet cards are very good in linux - be it a hardware or software problem.
_________________
Jingle Jangle Jewellery

Ge(nto)org · n00b Joined: 09 Aug 2004 Posts: 17

Ge(nto)org · n00b Joined: 09 Aug 2004 Posts: 17

Now, I compiled in the "Use Rx and Tx Polling (NAPI) (EXPERIMENTAL)" option into the r8169 module and the following happened:
In the
skge <-> switch <-> r8169 and
skge <-> r8169
setups I get a constant scp transfer rate of 19MB/s now, the NFS issue still exists (maybe it's only NFS related).

(The other setups
r8169 <-> switch <-> r8169
r8169 <-> r8169
still don't work.)

However, I didn't buy a Gigabit Ethernet card to get only 8MB/s more...

Could anyone recommend a Gigabit card that works smoothly under Linux?

Thx, georg

ljubo · Posted: Wed Jul 04, 2007 11:16 am Post subject:

I can recommend Intel PRO/1000 GT desktop adapter, or any other Intel Gigabit Ethernet Adapter.
Regards, Ljubo

alex.blackbit · Advocate Joined: 26 Jul 2005 Posts: 2397

that's right. intel cards have good driver support and work as expected.

godish · n00b Joined: 05 Oct 2004 Posts: 50 Location: Boston MA

I 3rd that, Never had any issues with intel gig adaptors
_________________
Nobody makes me bleed my own blood... NOBODY!

Ge(nto)org · n00b Joined: 09 Aug 2004 Posts: 17

K, now I bought a Intel PRO/1000 GT desktop adapter which uses the e1000 module.
With scp I get about 17MB/s when using
e1000 <-> skge
e1000 <-> switch <-> skge.

Grml, I really thought the Realtek chipset caused the problems, but since I don't think that Realtek and Marvell cards are crap the problem has to be somewhere else...

alex.blackbit · Advocate Joined: 26 Jul 2005 Posts: 2397

i am sorry to hear that. but... hmm, the problem MUST be somewhere.

Cyker · Veteran Joined: 15 Jun 2006 Posts: 1746

The RealTek Gbe chips have been reported to underperform, apparently due to IRQ spamming. Jumbo frames and enabling any off-loading options can help.

I have a Yukon2 and it's pretty quick, but has a tendency to randomly die during heavy transfers.

The built-in nVidia Forcdeth NIC is the fastest performer here.

Out of curiosity, how are you measuring the transfer?
I used to use BWM, but found that it was reporting speeds much lower than were really happening...

Ge(nto)org · n00b Joined: 09 Aug 2004 Posts: 17

pdr · l33t Joined: 20 Mar 2004 Posts: 618

I just scp'd a 5.5GB file from my server to my workstation. Pretty consistent 22-24MB/s, but had one "stall" where it dipped to 18MB/s.

The file is also on an NFS mount, so I timed a cp of it from the NFS mount to local drive. Now maybe caching is mucking with this (didn't run "sync"), but it took 2m31.359s real time, which comes out to 37MB/s. As a test (disk speed) I also timed copying it into the same directory on the server - time was 4m26.491s (which implies I am disk-bound on timing).

This file is on an encrypted, lvm, raid-5 partition on the server, but I'm happy with the speed, given the other benefits :twisted:

SETUP:

Server nic: I think I'm using the onboard skge, but could be the onboard forcedeth. Its an Asus A8N SLI Deluxe (love this motherboard for socket 939!)

Switch: netgear Gb switch. Running cat-6.

Workstation nic: PCIe card (Koutech PEN120 - USD 30 at Newegg.com) - is a Marvell 88E8053, using sky2 driver. I originally got this card because the previous motherboard (Asus P5V-LM or something - the microatx P5 board) had an attansic which I don't want to use (not in 2.6.20-r8?); new board (P5B Deluxe) has a Marvell chip so I could use it now, but I think I'll stick with the PCIe card for now.

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

A common misconception is forgetting that gigabit ethernet is faster than the (shared) PCI bus its attached to, and faster than the hard disk thats sending it data. Transfer rates peak due to initial transfer from cache to cache, then slow to disk/ PCI bus speeds.

PCI express mounted gigabit is faster, but is still limited by the disk or whatever your source/sink can deliver.

In disk to disk transfer, 100mb ethernet is only slightly slower than the real disk capability, so gigabit only transfers a bit faster because the disk is now the limit.

Cyker · Veteran Joined: 15 Jun 2006 Posts: 1746

LinuxLoader · n00b Joined: 04 May 2005 Posts: 18 Location: Bulgaria

when u try to copy with scp u use small packets .. thats a pain for the NIC . What are da CPU on the machines , try to use smp_affinity if they are smp . other thing is ... did u on the dma on the harddrive ? some times it`s didn`t workign and it`s a deam hell ... scp is not prefer for tests ... try ftp or http
_________________
Watch your thoughts; they become words.Watch your words; they becîme actions. Watch your actions;they become habits. Watch your habits; they become character.Watch your character;It becomes your destiny.

Galahad · Tux's lil' helper Joined: 12 Feb 2003 Posts: 126

What do you expect from scp? It fully encrypts the whole connection through a ssh tunnel!

use ftp if you want speed.

Ge(nto)org · n00b Joined: 09 Aug 2004 Posts: 17

alex.blackbit · Advocate Joined: 26 Jul 2005 Posts: 2397

it's of course true that scp causes overhead because of the encryption. but isn't that done in the host cpu and not in the nic?
but nevertheless nfs can be a big problemchild. one more thing you can try is samba. it may sound strange... but my experience was good using it between unix-like machines.

chovy · Guru Joined: 03 Dec 2004 Posts: 453

chovy · Guru Joined: 03 Dec 2004 Posts: 453

I am definitely capped at 10Mbs -- I copied a fat file (700mb) and it started at 15Mbps, and quickly settled at 8.9 - 9.5Mbps.

Any idea how to get my speed up from 10Mbps to 1000Mbps (let alone 100Mbps would be an improvement at this point).

I feel like a config file somewhere is limiting my throughput.
_________________
Woof, Woof! Add "[solved]" to the title! Woof, Woof!

xbmodder · Guru Joined: 25 Feb 2004 Posts: 404

Best performance with Intel gigabit (server) + Netgear Switch + Intel Gigabit (Server+)

Worst: ForceDeth + Cisco

Filesystem is cifs
_________________
http://xbmodder.us/

chovy · Guru Joined: 03 Dec 2004 Posts: 453

Maybe, but i'm not even seeing 100Mbps let alone 1Gigabit/sec

I want to know why I am only transferring at 10Mbps first. I have the kernel config params set for my Gigabit card. Is there anything else I need to do?

I also read about Cat5e cables instead of Cat5....would that be limiting me to 10mbps?
_________________
Woof, Woof! Add "[solved]" to the title! Woof, Woof!

eccerr0r · Posted: Fri Aug 31, 2007 8:54 pm Post subject:

I've a bunch of gigabit cards, and had mixed results with them:

RTL8169 x 3 - 2 of them DIED or stopped working at Gbit speeds. One of them still works fine.
RTL8110S - On-Motherboard (MSI) - OK
Syskonnect/3Com 3c940 (SKGE) - On-Motherboard (Abit) - OK
Broadcom Tigon3 - On-motherboard (Dell Inspiron) - OK
Marvell (Sky2) On-Motherboard (Foxconn) - OK

From past experience I can't recommend the RTL8169 unless you look carefully at the boards. I think I notice a pattern of the failing boards, but not certain yet (only 3 boards!). The on-board ethernets have worked OK for me though.

I haven't been able to get anywhere near Gbit saturation speeds. I typically send my data unencrypted on my personal LAN, and so far on a 10/100/1000 shared cat5 system (meaning, no packet size hacking) I can only get 40MB/sec. Many of my machines are restricted by PCI as well, but I'm happy if I can get 20MB/sec one way through the Gbit link... at least it's 2x what I can get over Fast Ethernet.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?