| View previous topic :: View next topic |
| Author |
Message |
trumee
Guru
Joined: 02 Mar 2003
Posts: 551
Location: London,UK
|
 Posted: Fri Sep 09, 2016 2:29 am Post subject: Mellanox ConnectX-2 EN (MNPA19-XTR) suspend to ram |
 |
|
Hello,
I have a Mellanox 10 GBe card which doesnt work after a suspend to ram (S3 sleep).
Following is the kernel buffer (dmesg) when 'pm-suspend' is used:
| Code: |
[15680.420895] PM: Syncing filesystems ... done.
[15680.435333] PM: Preparing system for mem sleep
[15680.435468] Freezing user space processes ... (elapsed 0.001 seconds) done.
[15680.436821] Freezing remaining freezable tasks ... (elapsed 0.000 seconds) done.
[15680.437814] PM: Entering mem sleep
[15680.437833] Suspending console(s) (use no_console_suspend to debug)
[15680.437947] sd 8:0:0:0: [sdb] Synchronizing SCSI cache
[15680.438138] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[15680.438253] serial 00:07: disabled
[15680.445437] sd 0:0:0:0: [sda] Stopping disk
[15680.648034] igb 0000:05:00.0: System wakeup enabled by ACPI
[15680.676345] igb 0000:06:00.0: System wakeup enabled by ACPI
[15680.686825] PM: suspend of devices complete after 249.045 msecs
[15680.687371] PM: late suspend of devices complete after 0.544 msecs
[15680.688225] ehci-pci 0000:00:1d.0: System wakeup enabled by ACPI
[15680.688305] ehci-pci 0000:00:1a.0: System wakeup enabled by ACPI
[15680.698640] PM: noirq suspend of devices complete after 11.274 msecs
[15680.698895] ACPI: Preparing to enter system sleep state S3
[15680.943781] PM: Saving platform NVS memory
[15680.943786] Disabling non-boot CPUs ...
[15680.943814] intel_pstate CPU 1 exiting
[15680.943947] Broke affinity for irq 42
[15680.944966] smpboot: CPU 1 is now offline
[15680.951832] intel_pstate CPU 2 exiting
[15680.951961] Broke affinity for irq 43
[15680.952977] smpboot: CPU 2 is now offline
[15680.960821] intel_pstate CPU 3 exiting
[15680.960947] Broke affinity for irq 44
[15680.961964] smpboot: CPU 3 is now offline
[15680.970770] intel_pstate CPU 4 exiting
[15680.971914] smpboot: CPU 4 is now offline
[15680.979738] intel_pstate CPU 5 exiting
[15680.980862] smpboot: CPU 5 is now offline
[15680.985731] intel_pstate CPU 6 exiting
[15680.986858] smpboot: CPU 6 is now offline
[15680.994687] intel_pstate CPU 7 exiting
[15680.995803] smpboot: CPU 7 is now offline
[15681.004824] ACPI: Low-level resume complete
[15681.004876] PM: Restoring platform NVS memory
[15681.005208] Enabling non-boot CPUs ...
[15681.005246] x86: Booting SMP configuration:
[15681.005247] smpboot: Booting Node 0 Processor 1 APIC 0x2
[15681.024083] cache: parent cpu1 should not be sleeping
[15681.024171] CPU1 is up
[15681.024189] smpboot: Booting Node 0 Processor 2 APIC 0x4
[15681.043106] cache: parent cpu2 should not be sleeping
[15681.043195] CPU2 is up
[15681.043212] smpboot: Booting Node 0 Processor 3 APIC 0x6
[15681.062127] cache: parent cpu3 should not be sleeping
[15681.062217] CPU3 is up
[15681.062235] smpboot: Booting Node 0 Processor 4 APIC 0x1
[15681.081147] cache: parent cpu4 should not be sleeping
[15681.081214] CPU4 is up
[15681.081228] smpboot: Booting Node 0 Processor 5 APIC 0x3
[15681.100073] cache: parent cpu5 should not be sleeping
[15681.100141] CPU5 is up
[15681.100155] smpboot: Booting Node 0 Processor 6 APIC 0x5
[15681.119082] cache: parent cpu6 should not be sleeping
[15681.119148] CPU6 is up
[15681.119161] smpboot: Booting Node 0 Processor 7 APIC 0x7
[15681.138101] cache: parent cpu7 should not be sleeping
[15681.138171] CPU7 is up
[15681.144884] ACPI: Waking up from system sleep state S3
[15681.191261] ehci-pci 0000:00:1d.0: System wakeup disabled by ACPI
[15681.191320] ehci-pci 0000:00:1a.0: System wakeup disabled by ACPI
[15681.201854] PM: noirq resume of devices complete after 11.732 msecs
[15681.202169] PM: early resume of devices complete after 0.209 msecs
[15681.202224] igb 0000:05:00.0: System wakeup disabled by ACPI
[15681.202240] igb 0000:06:00.0: System wakeup disabled by ACPI
[15681.202302] rtc_cmos 00:02: System wakeup disabled by ACPI
[15681.203272] serial 00:07: activated
[15681.204689] sd 0:0:0:0: [sda] Starting disk
[15681.428697] usb 1-1.1: reset full-speed USB device number 3 using ehci-pci
[15681.429684] usb 2-1.5: reset high-speed USB device number 3 using ehci-pci
[15681.507679] ata3: SATA link down (SStatus 0 SControl 300)
[15681.513681] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[15681.515660] ata4: SATA link down (SStatus 0 SControl 300)
[15681.515738] ata5: SATA link down (SStatus 0 SControl 300)
[15681.516647] ata2: SATA link down (SStatus 0 SControl 300)
[15681.518647] ata6: SATA link down (SStatus 0 SControl 300)
[15681.554581] ata1.00: configured for UDMA/133
[15681.587566] usb 2-1.3: reset full-speed USB device number 5 using ehci-pci
[15681.663840] ep_81: parent 2-1.5:1.1 should not be sleeping
[15681.673622] PM: resume of devices complete after 471.737 msecs
[15681.673781] PM: Finishing wakeup.
[15681.673782] Restarting tasks ... done.
[15684.427969] igb 0000:06:00.0 eno2: igb: eno2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[15691.445557] mlx4_core 0000:07:00.0: mlx4_cmd_post:cmd_pending failed
[15691.445560] mlx4_core 0000:07:00.0: Could not post command 0x49: ret=-5, in_param=0x0, in_mod=0x1, op_mod=0x0
[15691.445563] mlx4_core 0000:07:00.0: device is going to be reset
[15692.446405] mlx4_core 0000:07:00.0: device was reset successfully
[15692.446410] mlx4_en 0000:07:00.0: Internal error detected, restarting device
[15692.446428] mlx4_core 0000:07:00.0: Could not post command 0x43: ret=-5, in_param=0x0, in_mod=0x1, op_mod=0x0
[15697.234128] mlx4_core 0000:07:00.0: Internal error mark was detected on device
[15697.234226] mlx4_en: enp7s0: Close port called
[15701.146201] mlx4_core 0000:07:00.0: PCIe BW is different than device's capability
[15701.146204] mlx4_core 0000:07:00.0: PCIe link speed is 5.0GT/s, device supports 5.0GT/s
[15701.146206] mlx4_core 0000:07:00.0: PCIe link width is x4, device supports x8
[15701.178341] mlx4_en 0000:07:00.0: Activating port:1
[15701.180543] mlx4_en: 0000:07:00.0: Port 1: Using 64 TX rings
[15701.180544] mlx4_en: 0000:07:00.0: Port 1: Using 4 RX rings
[15701.180546] mlx4_en: 0000:07:00.0: Port 1: frag:0 - size:1518 prefix:0 stride:1536
[15701.180845] mlx4_en: 0000:07:00.0: Port 1: Initializing port
[15701.182347] mlx4_core 0000:07:00.0 enp7s0: renamed from eth0
[15701.182530] mlx4_core 0000:07:00.0: mlx4_restart_one was ended, ret=0
[15703.420072] mlx4_en: enp7s0: Link Up
|
ifconfig -a shows that the card is alive but it no longer has an ip address.
| Code: |
enp7s0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:03:d0:5e:1e:c0 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
|
My /etc/conf.d/net has a static ip address defined for it:
| Code: |
config_enp7s0="192.168.1.2 netmask 255.255.255.0"
mtu_enp7s0="1500"
|
Card details:
| Code: |
#ethtool -i enp7s0
driver: mlx4_en
version: 2.2-1 (Feb 2014)
firmware-version: 2.9.1200
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
#lspci
07:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
|
|
|
| Back to top |
|
 |
trumee
Guru
Joined: 02 Mar 2003
Posts: 551
Location: London,UK
|
| Posted: Fri Sep 09, 2016 7:58 pm Post subject: |
|
|
Following advice on irc i used the script /etc/pm/sleep.d/00network as mentioned here: https://wiki.gentoo.org/wiki/OpenRC_notebook_roaming_How-To
Looking at the /var/log/pm-suspend.log, it did seem that the interface 'enp7s0' was brought up and given the ip address. However, ifconfig showed that although the interface was available but no ip address was given to it. dmesg showed a bunch of error messages may lead to the problem. I also booted off a usb drive running Arch Linux and Networkmanager. Suspend/Resume with NetworkManager worked fine in Arch. So i guess this is either an openrc or gentoo issue.
#/var/log/pm-suspend.log
| /var/log/pm-suspend.log: |
Fri Sep 9 14:23:09 CDT 2016: Running hooks for resume
Running hook /usr/lib64/pm-utils/sleep.d/99video resume suspend:
/usr/lib64/pm-utils/sleep.d/99video resume suspend: success.
Running hook /usr/lib64/pm-utils/sleep.d/98video-quirk-db-handler resume suspend:
/usr/lib64/pm-utils/sleep.d/98video-quirk-db-handler resume suspend: success.
Running hook /usr/lib64/pm-utils/sleep.d/95led resume suspend:
/usr/lib64/pm-utils/sleep.d/95led resume suspend: not applicable.
Running hook /usr/lib64/pm-utils/sleep.d/94cpufreq resume suspend:
/usr/lib64/pm-utils/sleep.d/94cpufreq resume suspend: success.
Running hook /usr/lib64/pm-utils/sleep.d/90clock resume suspend:
/usr/lib64/pm-utils/sleep.d/90clock resume suspend: disabled.
Running hook /usr/lib64/pm-utils/sleep.d/75modules resume suspend:
Reloaded unloaded modules.
/usr/lib64/pm-utils/sleep.d/75modules resume suspend: success.
Running hook /usr/lib64/pm-utils/sleep.d/50unload_alx resume suspend:
/usr/lib64/pm-utils/sleep.d/50unload_alx resume suspend: success.
Running hook /usr/lib64/pm-utils/sleep.d/01grub resume suspend:
/usr/lib64/pm-utils/sleep.d/01grub resume suspend: disabled.
Running hook /usr/lib64/pm-utils/sleep.d/00powersave resume suspend:
Blacklisting 01grub.
Blacklisting 90clock.
/usr/lib64/pm-utils/sleep.d/00powersave resume suspend: success.
Running hook /etc/pm/sleep.d/00network resume suspend:
* Bringing up interface enp7s0
* 192.168.1.2 ... [ ok ]
|
dmesg
| dmesg: |
[ 42.632813] igb 0000:06:00.0 eno2: igb: eno2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 99.517109] mlx4_core 0000:07:00.0: command 0x49 timed out (go bit not cleared)
[ 99.517113] mlx4_core 0000:07:00.0: device is going to be reset
[ 100.517897] mlx4_core 0000:07:00.0: device was reset successfully
[ 100.517901] mlx4_en 0000:07:00.0: Internal error detected, restarting device
[ 100.517909] mlx4_en: enp7s0: frag:0 - size:1518 prefix:0 stride:1536
[ 100.518014] mlx4_en: enp7s0: Failed activating Rx CQ
[ 100.518088] mlx4_en: enp7s0: Failed starting port:1
[ 104.718723] mlx4_core 0000:07:00.0: Internal error mark was detected on device
[ 108.586009] mlx4_core 0000:07:00.0: PCIe BW is different than device's capability
[ 108.586011] mlx4_core 0000:07:00.0: PCIe link speed is 5.0GT/s, device supports 5.0GT/s
[ 108.586012] mlx4_core 0000:07:00.0: PCIe link width is x4, device supports x8
[ 108.617279] mlx4_en 0000:07:00.0: Activating port:1
[ 108.619418] mlx4_en: 0000:07:00.0: Port 1: Using 64 TX rings
[ 108.619420] mlx4_en: 0000:07:00.0: Port 1: Using 4 RX rings
[ 108.619421] mlx4_en: 0000:07:00.0: Port 1: frag:0 - size:1518 prefix:0 stride:1536
[ 108.619747] mlx4_en: 0000:07:00.0: Port 1: Initializing port
[ 108.620907] mlx4_core 0000:07:00.0: mlx4_restart_one was ended, ret=0
[ 108.621211] mlx4_core 0000:07:00.0 enp7s0: renamed from eth0
[ 110.861516] mlx4_en: enp7s0: Link Up
|
[Moderator edit: added url tag to Gentoo wiki reference so that readers can more readily read the referenced guide; decorated existing code blocks by adding ="description" to them. -Hu] |
|
| Back to top |
|
|
crystall
n00b
Joined: 31 Aug 2004
Posts: 15
|
| Posted: Mon Sep 12, 2016 9:46 am Post subject: Re: Mellanox ConnectX-2 EN (MNPA19-XTR) suspend to ram |
|
|
| trumee wrote: |
Card details:
| Code: |
#ethtool -i enp7s0
driver: mlx4_en
version: 2.2-1 (Feb 2014)
firmware-version: 2.9.1200
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
#lspci
07:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
|
|
You might be hitting a driver/hardware limitation here. If you search through Mellanox pages you'll find that in their own OFED distrubtion suspend-to-RAM is enabled only for the InfiniBand version of your card (ConnectX-2 IB). It's not stated explicitly but there's no mention of S3 support for the EN version of it.
http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_2_3-2_0_0.pdf
Try bringing this up with their support. Personally I've overseen a deployment of a few dozens of ConnectX-2 IB cards running on Gentoo and S3 was working correctly. We had a bunch of ConnectX-2 EN cards too but they were on machines that would never go into suspension so we never hit this particular issue. |
|
| Back to top |
|
|
trumee
Guru
Joined: 02 Mar 2003
Posts: 551
Location: London,UK
|
| Posted: Sat Nov 05, 2016 4:19 pm Post subject: |
|
|
Finally got it working.
| Code: |
/etc/pm/config.d/gentoo
SUSPEND_MODULES="mlx4_en"
|
| Code: |
/etc/pm/sleep.d/00network
#!/bin/bash
# properly stop and start the network
if [ -r "${PM_FUNCTIONS}" ]; then
. "${PM_FUNCTIONS}"
elif [ -r "${FUNCTIONS}" ]; then
. "${FUNCTIONS}"
else
# pm-utils version is too old, or something else is wrong
exit $NA
fi
case "$1" in
hibernate|suspend)
exec service net.enp7s0 stop
;;
thaw|resume)
sleep 0
;;
*) exit $NA
;;
esac
case "$1" in
hibernate|suspend)
exec rc default
;;
thaw|resume)
exec service net.enp7s0 start
exec service netmount start
;;
*) exit $NA
;;
esac
|
| Code: |
/etc/pm/sleep.d/06_nfs
#!/bin/bash
#
# Unmount NFS file systems before suspending, to
# prevent hangup when the system resume and NFS
# is not there.
#
# By Ralf, Unmount NFS http://palebluedot.nl/jml/computer-stuff/3-linux/26-umountnfs.html
case "$1" in
hibernate|suspend)
{
for fs in $(/bin/mount -t nfs | /usr/bin/awk {'print $3'}) ; do
echo "unmounting $fs";
/bin/umount -fl $fs;
done
} 2>/dev/null
;;
# do nothing when resuming
thaw|resume)
{
echo "Nothing to do with NFS"; } 2>/dev/null
;;
*)
;;
esac
exit $?
|
|
|
| Back to top |
|
|
crystall
n00b
Joined: 31 Aug 2004
Posts: 15
|
| Posted: Sat Nov 05, 2016 5:52 pm Post subject: |
|
|
| Awesome, thanks for sharing! |
|
| Back to top |
|
|
janos666
n00b
Joined: 15 Nov 2015
Posts: 30
|
| Posted: Wed Nov 16, 2016 6:11 pm Post subject: |
|
|
I have no problem with sleep/resume on the Windows client side and I never use sleep or hibernation on the Linux server side. However, I think these cards are inconveniently left alone by Gentoo devs.  .
Does anybody know why the mlx4_en driver got stuck at v2.2-1 (Feb 2014) in the mainline kernel tree (up to Linux 4.8.8 ) if the latest official Mellanox package has source code with 3.4-1 version number?
I didn't research this story but I assume they stopped updating the mainline code due to some licensing issues which emerged on the road (the fresh sources are always available from Mellanox, so I see no other reason to get stuck behind by several years on the kernel side). And then there is the sys-fabric/ofed package for Gentoo which can download, build and install drivers for nearly every Mellanox cards except the Ethernet NICs (HCAs are supported from the Connex-X family but not the EN cards), so it seems rather inconvenient to try and update the drivers manually (I am not sure I could do it properly myself and it would be inconvenient to maintain).
Last edited by janos666 on Wed Nov 16, 2016 9:24 pm; edited 4 times in total |
|
| Back to top |
|
|
crystall
n00b
Joined: 31 Aug 2004
Posts: 15
|
| Posted: Wed Nov 16, 2016 9:07 pm Post subject: |
|
|
| janos666 wrote: | | I have no problem with sleep/resume on the Windows client side and I never use sleep or hibernation on the Linux server side. However, I think these cards are inconveniently left alone by Gentoo devs. . |
This is not a Gentoo issue, Mellanox has not been very good at sending their code upstream, see this recent post by Linus Torvalds for an example of the problems with their contributions:
https://lkml.org/lkml/2016/3/16/684
Here's the relevant snippet:
| Quote: | Because I'm very close to not taking any rdma changes that touch
networking any more. Ever.
The Mellanox people are on my shit-list until they show that they can
actually act like responsible people and not just monkeys throwing
shit at the walls. |
Linus rants aside, Mellanox seem content with keeping most of their driver code out-of-tree, in the OFED package which is very inconvenient IMHO. |
|
| Back to top |
|
|
janos666
n00b
Joined: 15 Nov 2015
Posts: 30
|
| Posted: Wed Nov 16, 2016 9:17 pm Post subject: |
|
|
| crystall wrote: | | janos666 wrote: | | I have no problem with sleep/resume on the Windows client side and I never use sleep or hibernation on the Linux server side. However, I think these cards are inconveniently left alone by Gentoo devs. . |
This is not a Gentoo issue, Mellanox has not been very good at sending their code upstream, see this recent post by Linus Torvalds for an example of the problems with their contributions:
https://lkml.org/lkml/2016/3/16/684
Here's the relevant snippet:
| Quote: | Because I'm very close to not taking any rdma changes that touch
networking any more. Ever.
The Mellanox people are on my shit-list until they show that they can
actually act like responsible people and not just monkeys throwing
shit at the walls. |
Linus rants aside, Mellanox seem content with keeping most of their driver code out-of-tree, in the OFED package which is very inconvenient IMHO. |
Well, yes, but there could be some ebuilds to download the sources from Mellanox and either patch the kernel or build a loadable module (according to a kernel-builtin useflags). There are similar packages for Realtek NICs. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21602
|
| Posted: Thu Nov 17, 2016 2:20 am Post subject: |
|
|
| From a technical perspective, yes, Gentoo could probably provide ebuilds to wrap this functionality. However, many users believe, whether or not correctly, that having an ebuild in the main tree is a declaration that the package will receive some level of support from Gentoo. I have never dealt with Mellanox before, but if their work is as bad as Linus suggests, supporting it when users report problems could be a substantial amount of effort. There may not be any Gentoo developers who have the expertise, the time, and the motivation to provide that support. |
|
| Back to top |
|
|
janos666
n00b
Joined: 15 Nov 2015
Posts: 30
|
| Posted: Thu Nov 17, 2016 9:46 pm Post subject: |
|
|
It's not like I currently need any specific features or fixes but I always wondered why I seem to have somewhat asymmetric bandwidth between Linux (gentoo-sources driver) and Windows (usually the latest from Mellanox):
| Quote: | C:\Users\media\Downloads>iperf3.exe -c 192.168.1.11
Connecting to host 192.168.1.11, port 5201
[ 4] local 192.168.1.111 port 12141 connected to 192.168.1.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1006 MBytes 8.43 Gbits/sec
[ 4] 1.00-2.00 sec 1.05 GBytes 9.00 Gbits/sec
[ 4] 2.00-3.00 sec 1.05 GBytes 9.01 Gbits/sec
[ 4] 3.00-4.00 sec 1.02 GBytes 8.77 Gbits/sec
[ 4] 4.00-5.00 sec 1.05 GBytes 8.99 Gbits/sec
[ 4] 5.00-6.00 sec 1.04 GBytes 8.96 Gbits/sec
[ 4] 6.00-7.00 sec 1.05 GBytes 9.06 Gbits/sec
[ 4] 7.00-8.00 sec 1.06 GBytes 9.10 Gbits/sec
[ 4] 8.00-9.00 sec 1.04 GBytes 8.96 Gbits/sec
[ 4] 9.00-10.00 sec 1.06 GBytes 9.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec sender
[ 4] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec receiver
iperf Done.
C:\Users\media\Downloads>iperf3.exe -R -c 192.168.1.11
Connecting to host 192.168.1.11, port 5201
Reverse mode, remote host 192.168.1.11 is sending
[ 4] local 192.168.1.111 port 12147 connected to 192.168.1.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 787 MBytes 6.60 Gbits/sec
[ 4] 1.00-2.00 sec 824 MBytes 6.92 Gbits/sec
[ 4] 2.00-3.00 sec 829 MBytes 6.96 Gbits/sec
[ 4] 3.00-4.00 sec 829 MBytes 6.95 Gbits/sec
[ 4] 4.00-5.00 sec 831 MBytes 6.97 Gbits/sec
[ 4] 5.00-6.00 sec 829 MBytes 6.95 Gbits/sec
[ 4] 6.00-7.00 sec 802 MBytes 6.73 Gbits/sec
[ 4] 7.00-8.00 sec 829 MBytes 6.96 Gbits/sec
[ 4] 8.00-9.00 sec 807 MBytes 6.77 Gbits/sec
[ 4] 9.00-10.00 sec 823 MBytes 6.90 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 8.00 GBytes 6.87 Gbits/sec 241 sender
[ 4] 0.00-10.00 sec 8.00 GBytes 6.87 Gbits/sec receiver
iperf Done. |
These are single-port ConnectX-2 10Gbe cards (direct connection through optics), both are getting 8 PCI-E 2.0 lanes from the CPU (well, right now for this test, that is, sometimes I need the 8x slot for a second VGA card on the client side). |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
