ATI Crossfire

[Acidphase]

Hello,

I've been trying to get my 5970 and 5870 cards to crossfire now for quite some time. I have tried just about every possible combination.
Before you ask yes I tried to the duel GPU card(5970) on its own and not just going after a tri-fire setup which you can see below.

my xorg.conf was setup with:

Code: Select all

aticonfig --adapter=all --initial   

I have added the chains then reboot , then --cf on reboot.

Code: Select all

 ~ $ aticonfig --lsch
                                                                                                                                                                        
CrossFire chain for adapter 0, status: enabled                                                                                                                          
  0. 03:00.0 ATI Radeon HD 5900 Series                                                                                                                                  
  1. 04:00.0 ATI Radeon HD 5900 Series

However this is false:

Code: Select all

~ $ aticonfig --lscs

    Candidate Combination: 
    Master: 3:0:0 
    Slave: 4:0:0 
    CrossFire is disabled on current device
    CrossFire Diagnostics:
    CrossFire can work with P2P mapping through GART
    Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA 
    Candidate Combination: 
    Master: 3:0:0 
    Slave: 4:0:0 
    CrossFire is disabled on current device
    CrossFire Diagnostics:
    CrossFire can work with P2P mapping through GART
    Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA 
    Candidate Combination: 
    Master: 3:0:0 
    Slave: 4:0:0 
    CrossFire is disabled on current device
    CrossFire Diagnostics:
    CrossFire can work with P2P mapping through GART
    Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA 

I've had all sorts of combinations(including tri-fire) which stated it was enabled with --lsch but when I check it against --lscs it's never enabled.

Code: Select all

Linux 3.2.1-gentoo-r2 x86_64 AMD FX(tm)-8150 Eight-Core Processor AuthenticAMD GNU/Linux

Code: Select all

Asus Crosshair V Formula

Code: Select all

lspci -k
00:00.0 Host bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)                                                       
        Subsystem: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 port B)                                                                  
00:00.2 IOMMU: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory Management Unit (IOMMU)                                                                            
        Subsystem: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory Management Unit (IOMMU)                                                                        
00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port B)                                                               
        Kernel driver in use: pcieport
00:05.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port E)
        Kernel driver in use: pcieport
00:06.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port F)
        Kernel driver in use: pcieport
00:07.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port G)
        Kernel driver in use: pcieport
00:09.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port H)
        Kernel driver in use: pcieport
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx1 port A)
        Kernel driver in use: pcieport
00:0b.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (NB-SB link)
        Kernel driver in use: pcieport
00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
        Subsystem: ASUSTeK Computer Inc. Device 84dd
        Kernel driver in use: ahci
00:12.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
        Kernel driver in use: ohci_hcd
00:12.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
        Kernel driver in use: ehci_hcd
00:13.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
        Kernel driver in use: ohci_hcd
00:13.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
        Kernel driver in use: ehci_hcd
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
        Subsystem: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller
        Kernel driver in use: piix4_smbus
00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller
00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
        Kernel driver in use: ohci_hcd
00:16.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
        Kernel driver in use: ohci_hcd
00:16.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
        Subsystem: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
        Kernel driver in use: ehci_hcd
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3
        Kernel driver in use: k10temp
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4
        Kernel driver in use: fam15h_power
00:18.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb)
        Kernel driver in use: pcieport
02:04.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb)
        Kernel driver in use: pcieport
02:08.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb)
        Kernel driver in use: pcieport
03:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Hemlock [Radeon HD 5900 Series]
        Subsystem: Advanced Micro Devices [AMD] nee ATI Device 2542
        Kernel driver in use: fglrx_pci
        Kernel modules: fglrx
03:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series]
        Subsystem: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series]
04:00.0 Display controller: Advanced Micro Devices [AMD] nee ATI Hemlock [Radeon HD 5900 Series]
        Subsystem: Advanced Micro Devices [AMD] nee ATI Device 2042
        Kernel driver in use: fglrx_pci
        Kernel modules: fglrx
05:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection
        Subsystem: ASUSTeK Computer Inc. Device 8457
        Kernel driver in use: e1000e
06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
        Subsystem: ASUSTeK Computer Inc. Device 8488
        Kernel driver in use: xhci_hcd
07:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
        Subsystem: ASUSTeK Computer Inc. Device 8488
        Kernel driver in use: xhci_hcd
08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
        Subsystem: ASUSTeK Computer Inc. Device 8488
        Kernel driver in use: xhci_hcd
09:00.0 Audio device: Creative Labs X-Fi Titanium series [EMU20k2] (rev 04)
        Subsystem: Creative Labs Device 0043
        Kernel driver in use: snd_ctxfi
0a:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cypress XT [Radeon HD 5870]
        Subsystem: ASUSTeK Computer Inc. Device 032e
        Kernel driver in use: fglrx_pci
        Kernel modules: fglrx
0a:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series]
        Subsystem: ASUSTeK Computer Inc. Device aa50

I see people they say it works fine in Ubuntu I really don't like Ubuntu been using Gentoo for a while now however I really don't like running a dual card in single chip mode and having a second card rendered useless either. Obviously all works fine in Windows 7 x64 which I hate to say lol.
I would really love to see the day where I don't have to dual boot my system just to play games.

Any help would be accepted and appreciated.

[DaggyStyle]

afaik, xfire isn't supported on linux. no matter what driver it is.

[xming]

I thought that catalyst supports crossfire under linux, the open source drivers doesn't.

[DaggyStyle]

I thought that catalyst supports crossfire under linux, the open source drivers doesn't.

right, my bad, but have anyone tried that? I've not seen a single benchmark on the matter.

[Acidphase]

I thought that catalyst supports crossfire under linux, the open source drivers doesn't.

right, my bad, but have anyone tried that? I've not seen a single benchmark on the matter.

Crossfire has been around in Linux since 2008.
http://www.phoronix.com/scan.php?page=a ... inux&num=1

[Acidphase]

I thought that catalyst supports crossfire under linux, the open source drivers doesn't.

This has been the case now for years and crossfire is dodgy as you can see depending on OS , I bet if I loaded up OPen SuSe it would crossfire. I hear allot of the AMD devs use Open Suse lol.
However this may change depending on how fast the open source guys pick up things and go as there is some interesting changes coming in kernel 3.3 take a look:
http://www.phoronix.com/scan.php?page=n ... px=MTAzODQ

[hedmo]

hi all

i am using crossfire at typing moment and it works quite good but not 100%:


mybox / # uname -a
Linux mybox 3.1.0-rc3-mm1 #7 SMP Sat Jan 14 08:46:08 Local time zone must be set--see zic x86_64 AMD Phenom(tm) II X6 1055T Processor AuthenticAMD GNU/Linux
mybox / #



Acidphase

not all r600 -> are supported.like my first cards xt2900 (crossfire could not set a master for the chain)but my xfx 6870 do.
if you realy want to see if your cards are supported..set +qt4 for ati-drivers.if they are you will have that option in the menu




DaggyStyle wrote:


right, my bad, but have anyone tried that? I've not seen a single benchmark on the matter.

do you know what i need to do a test like that.the only thing i am using them for is to play games=resolution=3x1400x1050=desktop area=4200x1074 .


regards hedmo

[Acidphase]

not all r600 -> are supported.like my first cards xt2900 (crossfire could not set a master for the chain)but my xfx 6870 do.
if you realy want to see if your cards are supported..set +qt4 for ati-drivers.if they are you will have that option in the menu

I have seen this option before and have enabled it in the past upon reboot it disappears. My cards have to be supported 5970 and 5870 all tri-fire fine in windows and for sure
dual GPU cards in other Linux distros. I just want to get it going in Gentoo lol.

In Windows 7 x64 for BF3 getting 200fps+ 1920x1080 with 4x anti-aliasing, 16x anisotropic filtering and maximum in-game settings seen 550fps in WoW lol , and Skyrim runs like a charm.
I don't expect see these frame rates in Linux via wine but I won't need them to that high already well within the playable realm.

[hedmo]

Acidphase

yes.i think your cards i supported.my xt2900 is supported to but not with the 2x xt2900.if i had one xt2900 and one 2900
it should worked for me.if i still wanted to get the crossfire working, i had to downgrade to ati-drivers-9.x and struggle with
-9.x old problems.you are talking about tri-fire.i have not seen all your cards,do you have a card with dual-gpu? if that
so.you already have crossfire on that one.about game i have stoped looking on the fps.even if i watch movies on 2 screens,play
on the 3 and scoll my cube i will still get 85 fps

[DaggyStyle]

hi all

i am using crossfire at typing moment and it works quite good but not 100%:


mybox / # uname -a
Linux mybox 3.1.0-rc3-mm1 #7 SMP Sat Jan 14 08:46:08 Local time zone must be set--see zic x86_64 AMD Phenom(tm) II X6 1055T Processor AuthenticAMD GNU/Linux
mybox / #



Acidphase

not all r600 -> are supported.like my first cards xt2900 (crossfire could not set a master for the chain)but my xfx 6870 do
if you realy want to see if your cards are supported..set +qt4 for ati-drivers.if they are you will have that option in the menu




DaggyStyle wrote:


right, my bad, but have anyone tried that? I've not seen a single benchmark on the matter.

do you know what i need to do a test like that.the only thing i am using them for is to play games=resolution=3x1400x1050=desktop area=4200x1074 .


regards hedmo

dude, just a side node, it is a good practice to insert as pace after the usage of a dot in case newline will not be in use, it makes it easier to understand you.

I assume that your first part was a question (if so, please use the question mark next time) I have no clue, sorry.

I don't use the closed driver.

[Acidphase]

Edit:
Just as a side note, even though it says I'm a n00b over there. I may be a n00b to the forums but I am not a n00b to Linux.
I have been manipulating/running Linux/Unix for over a Decade now via locally and remotely. I'm slightly a n00b to Gentoo as I have
been using it for roughly 5 months now. .

Acidphase

Yes, I think your cards are supported. My xt2900 is supported too, but not with the 2x xt2900, if I had one xt2900 and one 2900
it should have worked for me. However if I still wanted to get the crossfire working, I had to downgrade my ati-drivers to 9.x then struggle with the 9.x bugs.
You are talking about tri-fire. I have not seen all your cards. Do you have a card with dual-gpu? If yes
then you already have crossfire on that card. As far as games I have stopped looking on the fps, if i watch movies on 2 screens, and play
on the 3 and scroll my cube I will still get 85 fps.
(Edited to make more coherent, just for you DaggyStyle lol)

@hedemo
How can you not see all of my cards? I posted with the question the output of:

Code: Select all

lspci -k

@hedemo
On the Dual GPU notion that they're already crossfired by default this is not true. For one you have to have the devices listed in the xorg.conf before a chain can even be made.
After that you have enable the chain, also as I stated previously I've already tried just the dual card on it's own as well as TRI-Fire.
I'm curious though now that you said that lol can I see the output of your:

Code: Select all

aticonfig --lscs 

-------

Do you know what I need to do a benchmark like that? The only thing I'm using them for is to play games @resolution=3x1400x1050=desktop area=4200x1074

I think Phoronix's test suite may have what you're looking for take a look here DaggyStyle http://www.phoronix-test-suite.com/

[DaggyStyle]

Do you know what I need to do a benchmark like that? The only thing I'm using them for is to play games @resolution=3x1400x1050=desktop area=4200x1074

I think Phoronix's test suite may have what you're looking for take a look here DaggyStyle http://www.phoronix-test-suite.com/

I'm well aware of that phoronix test suite, but as I said, I haven't seen before the creation of this thread xfire benchmarks, I didn't said I want to execute one.

[Acidphase]

Do you know what I need to do a benchmark like that? The only thing I'm using them for is to play games @resolution=3x1400x1050=desktop area=4200x1074

I think Phoronix's test suite may have what you're looking for take a look here DaggyStyle http://www.phoronix-test-suite.com/

I'm well aware of that phoronix test suite, but as I said, I haven't seen before the creation of this thread xfire benchmarks, I didn't said I want to execute one.

Lol my bad , it was hedemo that asked for it. He didn't quote you properly. He added his Question within your quote.

[Acidphase]

I guess this goes in the you're "SOL" category? lol

[hedmo]

Acidphase

sorry for taking this long time to reply. i had to disabled iommu in bios and remove it in grub otherwise crossfire was disabled
for me to"



mybox ~ # aticonfig --lscs
Candidate Combination:
Master: 7:0:0
Slave: 2:0:0
CrossFire is enabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART
Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA
Candidate Combination:
Master: 7:0:0
Slave: 2:0:0
CrossFire is enabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART
Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA
mybox ~ # lspci -k | grep VGA
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [ATI Radeon HD 6800 Series]
07:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [ATI Radeon HD 6800 Series]
mybox ~ #

[Acidphase]

Acidphase

sorry for taking this long time to reply. i had to disabled iommu in bios and remove it in grub otherwise crossfire was disabled
for me to"



mybox ~ # aticonfig --lscs
Candidate Combination:
Master: 7:0:0
Slave: 2:0:0
CrossFire is enabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART
Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA
Candidate Combination:
Master: 7:0:0
Slave: 2:0:0
CrossFire is enabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART
Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA
mybox ~ # lspci -k | grep VGA
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [ATI Radeon HD 6800 Series]
07:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [ATI Radeon HD 6800 Series]
mybox ~ #

Thanks for the reply , I can't shut off my IOMMU totally as it seems my input devices are being pumped through it. I can shut of IOMMU via the bios but then I hade to recompile the kernel remove some options that I had built in mainly the amd_iommu but as far as straight up IOMMU aka IBM Calgary this cant removed for whatever reason as I will not have any input device support via USB. So I can not set iommu=off in grub.

What's bothering me is that the IOMMU device has always been enabled and Crossfire works just fine in Windows as well as the IOMMU Device (which would be the AMD one) shows up in windows device manager as well.
I wonder if AMD will built support the AMD IOMMU Device V2 in the next Catalyst release because their is a definite link between the two as fglrx fails to build with this feature enabled in the Kernel (this feature only shows up in 3.3.x kernels).

With that being said some how or another all of them are tied to one another in some form or another. What I mean as in all is:
CONFIG_AGP
CONFIG_AGP_AMD64
Which are he "Gart" related aspects of this scheme.
CONFIG_CALGARY_IOMMU

The all have to do with PCI and DMA, I'm not a kernel Developer nor a Developer for Graphics Drivers and don't know whats going wrong where, how, and when lol but it's getting too the point where it's pissing me off so much I may become both lol.

Edit:
Hmm seems I wasn't so far from the truth and hopefully in the near future Crossfire won't be Such a Night Mare take a look "Multi GPU support on Linux platforms".
http://ir.amd.com/phoenix.zhtml?c=74093 ... id=1646939

[Acidphase]

It seems when you have the physical crossfire bridge connected is where the Linux Kernel fails. In a TRI-Fire setup or more that is.

When the MSI-X sets up the IRQ for each device it's having a problem giving an IRQ for the 3rd GPU it isn't seeing it as a proper vga device.
However it' saying a bridge is possible.

Code: Select all

dmesg | grep vga
[    0.233493] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.233493] vgaarb: device added: PCI:0000:0a:00.0,decodes=io+mem,owns=none,locks=none
[    0.233493] vgaarb: loaded
[    0.233493] vgaarb: bridge control possible 0000:0a:00.0
[    0.233493] vgaarb: bridge control possible 0000:03:00.0
[   49.834414] vgaarb: this pci device is not a vga device
[   49.834421] vgaarb: this pci device is not a vga device
[   49.834429] vgaarb: this pci device is not a vga device
[   49.834436] vgaarb: this pci device is not a vga device
[   49.834443] vgaarb: this pci device is not a vga device
[   49.834450] vgaarb: this pci device is not a vga device
[   49.834457] vgaarb: this pci device is not a vga device
[   49.834464] vgaarb: this pci device is not a vga device
[   49.834471] vgaarb: this pci device is not a vga device
[   49.834478] vgaarb: this pci device is not a vga device
[   49.834485] vgaarb: this pci device is not a vga device
[   49.834492] vgaarb: this pci device is not a vga device
[   49.834498] vgaarb: this pci device is not a vga device
[   49.834506] vgaarb: this pci device is not a vga device
[   49.834513] vgaarb: this pci device is not a vga device
[   49.834520] vgaarb: this pci device is not a vga device
[   49.834527] vgaarb: this pci device is not a vga device
[   49.834534] vgaarb: this pci device is not a vga device
[   49.834541] vgaarb: this pci device is not a vga device
[   49.834548] vgaarb: this pci device is not a vga device
[   49.834555] vgaarb: this pci device is not a vga device
[   49.834562] vgaarb: this pci device is not a vga device
[   49.834569] vgaarb: this pci device is not a vga device
[   49.834576] vgaarb: this pci device is not a vga device
[   49.834582] vgaarb: this pci device is not a vga device
[   49.834589] vgaarb: this pci device is not a vga device
[   49.834597] vgaarb: this pci device is not a vga device
[   49.834604] vgaarb: this pci device is not a vga device
[   49.834612] vgaarb: this pci device is not a vga device
[   49.834628] vgaarb: this pci device is not a vga device
[   49.834636] vgaarb: this pci device is not a vga device
[   49.834644] vgaarb: this pci device is not a vga device
[   49.834652] vgaarb: this pci device is not a vga device
[   49.834661] vgaarb: this pci device is not a vga device
[   49.834669] vgaarb: this pci device is not a vga device
[   49.834676] vgaarb: this pci device is not a vga device
[   49.834690] vgaarb: device changed decodes: PCI:0000:0a:00.0,olddecodes=io+mem,decodes=none:owns=none
[   49.834702] vgaarb: this pci device is not a vga device

Now a truncated output of

Code: Select all

dmesg | grep pci

[   50.438714] fglrx_pci 0000:03:00.0: irq 100 for MSI/MSI-X
[   50.454599] fglrx_pci 0000:04:00.0: irq 101 for MSI/MSI-X

Quite odd , even odder is the output of aticonfig --lscs especially compared against the above state that a bridge is possible.

Code: Select all

aticonfig --lscs
    Candidate Combination: 
    Master: 3:0:0 
    Slave: 4:0:0 
    CrossFire is disabled on current device
    CrossFire Diagnostics:
    CrossFire can work with P2P mapping through GART
    Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA 
    Candidate Combination: 
    Master: 3:0:0 
    Slave: 4:0:0 
    CrossFire is disabled on current device
    CrossFire Diagnostics:
    CrossFire can work with P2P mapping through GART
    Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA 
    Candidate Combination: 
    Master: 3:0:0 
    Slave: 4:0:0 
    CrossFire is disabled on current device
    CrossFire Diagnostics:
    CrossFire can work with P2P mapping through GART
    Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA 

What's odd is the 3 entry as the PCI BUS ID's are wrong on a Duel GPU setup would just see the first two entries. Whic makes the next command even more odd lol.

Code: Select all

aticonfig --lscc

Master adapter:  0. 03:00.0 ATI Radeon HD 5900 Series
    Candidates:  1. 04:00.0 ATI Radeon HD 5900 Series
               
Master adapter:  1. 04:00.0 ATI Radeon HD 5900 Series
    Candidates:  1. 04:00.0 ATI Radeon HD 5900 Series
               
Master adapter:  2. 0a:00.0 ATI Radeon HD 5800 Series
    Candidates:  1. 04:00.0 ATI Radeon HD 5900 Series

That middle entry sure is odd it has it as master and a slave at the same for a possible chain. I think this is a result that between the the fglrx module and the way the kernel is looking at things as it can't make
a 3 tier chain internally like Windows can.

In short between the fglrx module, the way kernel is looking at things, The MSI-X assignments, and the wonderful code of the proprietary drivers themselves it is in a state of FUBAR when it comes to CrossFire with more
than 2 GPU's at least for now.

As far as the IOMMU there is more than one implementation of it. The one that hdemo is disabling is the AGP aperture portion which by default reserves 64MB of RAM. With the AMD 890-990FX chipsets which basically are the same exact thing with the exception I have a bit more control with the 990FX IOMMU in BIOS and kernel side.

Code: Select all

dmesg | grep IOMMU
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    5.564370] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40

I enabled IOMMU for AMD-v which is designed for virtual setups more specifically with new features like GPU Pass-through etc.. at the same time I did not allow it to create the 64MB aperture which is a seperate option on my North Bridge for IOMMU.
In light of this and now at least how it's all making sense to me now lol. I will most likely go back to KMS as I have more control and I can use all the features of my IOMMU including the V2 introduced in the 3.3.x Kernels. IMMOUv2 gives you more support for PCI PRI and PASID.
Mind you this is markd "experimental". I have feeling this coupled with another new feature introduced in the 3.3.x kernels called DMA_SHARED_BUFFER a.k.a DMA-BUF will open the door for multi-gpu setups using open source drivers. (Which is already being implmented and worked on with Nvidia open source)

[Acidphase]

[topic=]How to enable Crossfire on a AMD MultiGPU card[/topic]

Hello I'm just going to paste this here as I wrote it in a response to a PM I received on the forum feel free to criticize or share it lol

Hello <person>,

Okay I just re-set it up on my other box, for starters you have to disable IOMMU in the kernel so make sure you have all of these off "N".

Code: Select all

CONFIG_AMD_IOMMU 
CONFIG_IOMMU_SUPPORT 
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT

Then rebuild your kernel make install etc..
(note: you should run emerge -av @x11-module-rebuild if you have the x11 overlay if not reinstall all the xorg drivers and the ati-drivers. Why? #1 your input devices may not work anymore because the DMA map just changed and #2 the drivers will not work either for the same reason so X wont load)

So once you have all that squared away reboot load from the new built kernel if you can login as root before going to X or whatever flavor DE(Desktop Environment) you're running. Then:

Code: Select all

cd /etc/X11
mv xorg.conf xorg.conf.backup
aticonfig --initial
mv /etc/ati/amdpcsdb /etc/ati/amdpcsdb.backup

Or (this is the file where crossfire settings are saved so as a precaution remove it or back it up so whatever you did previously doesn't interfere)

Code: Select all

rm /etc/ati/amdpcsdb
/etc/init.d/xdm start

Or if you like to use startx

Code: Select all

su <user>
startx

If you're wondering why I didn't pass the --adapter option in the aticonfig --initial it's because it's unneeded because the second GPU is technically not a entire display adapter.
So now you're back in the DE and we can finally set up the crossfire chains and then enable them.
OKay so load up your favorite terminal and type:
(you don't want to or need to do this ass root the normal <user> is fine)

Code: Select all

aticonfig --cfa --adapter=0,1
ALT+CTRL+F1 

If you used startx previously then
Drop back to tty1 and:

Code: Select all

ctrl +c (kill X)
startx

If not:

Code: Select all

/etc/init.d/xdm stop
/etx/init.d/xdm start 

(I don't use restart on purpose sometimes causes problems why I don't know it just does lol)

Load up the terminal again (almost done bare with me AMD makes things fun eh?). This time type:
aticonfig --cf on --adapter=0
This will enable the previous chain we created earlier, guess what restart X again" lol
Once more load terminal and type:

Code: Select all

aticonfig --lscs

Hopefully you'll see something like this:

Code: Select all

Candidate Combination: 
    Master: 8:0:0 
    Slave: 7:0:0 
    CrossFire is enabled on current device
    CrossFire Diagnostics:
    There is CrossFire Side port connection between GPUs
    CrossFire can work with P2P write through peer aperture
    Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA 

Congratulations Crossfire in Linux I hope this works and helps.

Regards,
-Acidphase