Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
InfiniBand - a cheap way to _fast_ network, PC to PC?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Sun Feb 19, 2017 6:01 pm    Post subject: InfiniBand - a cheap way to _fast_ network, PC to PC? Reply with quote

I stumbled into this youtube video.

Then I went straight to eBay and bought cheapest 20Gbps IB cards and a cable. Reason? I need faster transfers between my desktop and server for backups copying raw video etc.
Time will tell if I just threw close to 100€ into a well...

Meanwhile I'd like to hear if any of you have experience with InfiniBand and if you use it with Gentoo specifically.


Last edited by Zucca on Wed Feb 22, 2017 7:22 pm; edited 1 time in total
Back to top
View user's profile Send private message
R0b0t1
Apprentice
Apprentice


Joined: 05 Jun 2008
Posts: 264

PostPosted: Wed Feb 22, 2017 5:16 am    Post subject: Reply with quote

I've been looking for something similar. Has it worked?

InfiniBand seems like a cheaper, better, faster option than a local highspeed copper (of some other technology) or fiber network. I suppose supercomputing consumers may have driven the price down - it seems too cheap for what it is compared to the alternatives.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Wed Feb 22, 2017 6:14 pm    Post subject: Reply with quote

R0b0t1 wrote:
I've been looking for something similar. Has it worked?
The cards have not arrived yet.
R0b0t1 wrote:
than a local highspeed copper (of some other technology) or fiber network.
InfiniBand can also be copper or optical.

But yeah. It seems quite cheap. I thought of going to 40Gbps, but I suspect I couldn't really utilize that fast connection.
I ordered the 4x cards and a cable. There is also a 12x connector. Which is quite wide.
One thing I haven't figured out completely yet is why there's RAM memory on some cards. Maybe if you directly attach a stogare device(s) to the cable the it can act as a fast cache maybe...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10587
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Feb 22, 2017 6:24 pm    Post subject: Reply with quote

Ooohh. I'm interested in the outcome of this experiment, too. I have a baby cluster at home that I would love to network with InfiniBand.

Edit: I don't see a combination that at 4x links produces 20Gb/sec throughput. What type of cards did you order?

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Wed Feb 22, 2017 7:10 pm    Post subject: Reply with quote

John R. Graham wrote:
What type of cards did you order?
Mellanox MHGS18-XTC. Oh, and look at the price (at introduction, I guess). ;P I paid around $30-$35 for each.

Note that I'm using bps as "bits per second".
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10587
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Feb 22, 2017 7:59 pm    Post subject: Reply with quote

Got it. I think that rate is without encoding overhead. The actual data throughput is 16 Gbit/sec. Come to think of it, though, Gigabit Ethernet probably advertises raw line rate, too, so this is probably fully 20x faster than GigE.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Wed Feb 22, 2017 8:10 pm    Post subject: Reply with quote

John R. Graham wrote:
so this is probably fully 20x faster than GigE.
I do hope so. :)
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Wed Feb 22, 2017 9:35 pm    Post subject: Reply with quote

Since it seems relevant to the topic, has anyone ever tried IP over Firewire? Seems like a cheap way to get up to 3.2Gbps uninterrupted bandwidth between two PCs if they both have FW ports sitting otherwise idle; I just need a second one to play with...
Back to top
View user's profile Send private message
R0b0t1
Apprentice
Apprentice


Joined: 05 Jun 2008
Posts: 264

PostPosted: Wed Feb 22, 2017 9:52 pm    Post subject: Reply with quote

Zucca wrote:
One thing I haven't figured out completely yet is why there's RAM memory on some cards. Maybe if you directly attach a stogare device(s) to the cable the it can act as a fast cache maybe...


Unfortunately I don't have a very in-depth answer for you, but suffice it to say that routing logic isn't stateless. Some of the RAM may be for buffering received data but it may also be used for configuring the behavior of the network interface (i.e. acting on the raw internet protocol and underlying physical interface data). As InfiniBand is not as widely implemented as the various forms of ethernet they may not have miniaturized the hardware to the point where then necessary memory is on-die with the controlling logic.

Probably a fruitless endeavor, but the cards may have a miniature RTOS that you could run code on if you could figure out how to access it.

Thanks for info w.r.t. what you ordered - I'll give it a look when time and money permit.


Last edited by R0b0t1 on Thu Feb 23, 2017 5:24 pm; edited 1 time in total
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Thu Feb 23, 2017 10:53 am    Post subject: Reply with quote

Ant P. wrote:
Since it seems relevant to the topic, has anyone ever tried IP over Firewire? Seems like a cheap way to get up to 3.2Gbps uninterrupted bandwidth between two PCs if they both have FW ports sitting otherwise idle; I just need a second one to play with...
Isn't the FW3200 made mainly for server environments? I recall that FW800 was the best consumer could "get".
But yeah. FW3200 would yield to quite high speeds too. I haven't tried. I think FW has DMA. I think that drops latency quite a lot, making it proper for certain networking tasks.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Fri Feb 24, 2017 11:37 am    Post subject: Reply with quote

I have small backfiring on my planned setup.
The PCIe 16x slot on my desktop motherboard only acts as 4x, while the card needs at least 8x. I read that with 4x I can (could) still get around 16Gbps, so I'll settle with that until I upgrade to (possibly) Ryzen.

In more distant future I might upgrade to Thunderbolt networking.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10587
Location: Somewhere over Atlanta, Georgia

PostPosted: Fri Feb 24, 2017 11:59 am    Post subject: Reply with quote

If your PCIe slots are v2.0, then that's correct, but it's the theoretical maximum. In practice, I'd expect half that, which would still be enough to declare your experiment a resounding success, in my opinion.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Fri Feb 24, 2017 12:18 pm    Post subject: Reply with quote

John R. Graham wrote:
If your PCIe slots are v2.0, then that's correct, but it's the theoretical maximum. In practice, I'd expect half that, which would still be enough to declare your experiment a resounding success, in my opinion.

- John
Yes. PCIe 2.0. A half of 16Gbps sounds a bit pessimistic, but I can cope with it.

Wikipedia wrote:
Like 1.x, PCIe 2.0 uses an 8b/10b encoding scheme, therefore delivering, per-lane, an effective 4 Gbit/s max transfer rate from its 5 GT/s raw data rate.
... does GT mean GigaTransfers? Is there some error correcting bits (checksums) so that theoretical speed is actually 4/5th of the speed?
I'm not too familiar with the inner side of PCIe technology...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10587
Location: Somewhere over Atlanta, Georgia

PostPosted: Fri Feb 24, 2017 3:09 pm    Post subject: Reply with quote

Zucca wrote:
... does GT mean GigaTransfers? Is there some error correcting bits (checksums) so that theoretical speed is actually 4/5th of the speed?
I'm not too familiar with the inner side of PCIe technology...
Yes, GT means that. For the rest, during the design of any signaling medium, the designers have to answer two questions:
  1. How do you distinguish the boundary between one symbol and the next?
  2. Since it's desirable to be able to transmit all possible symbols in a message, what mechanism is used to distinguish between actual message and metadata (start of message, length of message, checksum info, etc.)?
Of course there are many other questions to be answered, but these two will suffice to explain the 4/5 overhead you've read about. In very general terms, there are two ways to distinguish those things:
  1. With out-of-band signaling: using extra wires (or radio channels, or carrier pigeons) to delineate the boundaries between symbols and message elements.
  2. With in-band signaling: using the same wires (or radio channels, or carrier pigeons) with special signal patterns to delineate such things. Those special patterns eat up a portion of the bandwidth that could otherwise be used to move data, but then the physical interface is simpler, using less wires (or...well, you get the idea).
Both question 1 and question 2 can independently be answered with answer 1 or answer 2, by the way.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Fri Feb 24, 2017 5:06 pm    Post subject: Reply with quote

I'm interested too, although probably more at a 10gbps rate. Still, IB sounds interesting and as yet I have found precious few Linux posts about it.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3105

PostPosted: Fri Feb 24, 2017 8:18 pm    Post subject: Reply with quote

I'm watching this topic too.
I wonder how you managed to get those adapters at 35$ each. When you mentioned this price I started looking around, but the things I found tend to be MUCH more expensive. Prohibitively expensive.

My more technical questions are:
Can you use IB as a boot medium? I mean something like PXE or SAN boot.
Can you use its RDMA feature for high performance storage sharing? (Stretched RAID anyone?)
Back to top
View user's profile Send private message
R0b0t1
Apprentice
Apprentice


Joined: 05 Jun 2008
Posts: 264

PostPosted: Fri Feb 24, 2017 8:38 pm    Post subject: Reply with quote

szatox wrote:
I'm watching this topic too.
I wonder how you managed to get those adapters at 35$ each. When you mentioned this price I started looking around, but the things I found tend to be MUCH more expensive. Prohibitively expensive.

My more technical questions are:
Can you use IB as a boot medium? I mean something like PXE or SAN boot.
Can you use its RDMA feature for high performance storage sharing? (Stretched RAID anyone?)


For the first: You should be able to, but the UEFI firmware needs a driver. If the IB cards support a common subset of communications commands, similar to graphics cards for their display capabilities, it should work. I've seen netbooting ARM boards over USB to Ethernet converters but this was something with explicit support in the bootloader.

Very interested in #2.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Fri Feb 24, 2017 10:38 pm    Post subject: Reply with quote

szatox wrote:
I wonder how you managed to get those adapters at 35$ each. When you mentioned this price I started looking around, but the things I found tend to be MUCH more expensive. Prohibitively expensive.



Also I think you could find used IB stuff from Amazon as well...

Also just in case... I'll mention this:
Be sure to pick ones with PCIe, not PCI-X... unless, of course, you happen to have PCI-X slots on your MB.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Sat Feb 25, 2017 11:09 am    Post subject: Reply with quote

szatox wrote:
Can you use its RDMA feature for high performance storage sharing? (Stretched RAID anyone?)
There are IB-SATA cables. I guess you have to specifically configure the IB port to act as SATA host. I don't know if the IB host card needs some specific hardware feature to be able to act as so.

Earlier I found a used IB switch from ebay for about $100. I think it was 10Gbps one. Just imagine how many SATA drives you could attach... If it's even possible that way.

I really wish there was someone with actual experience with using IB on large servers.


And thanks John for the insights of PCIe!
Back to top
View user's profile Send private message
unixlike
n00b
n00b


Joined: 18 Dec 2016
Posts: 4

PostPosted: Mon Feb 27, 2017 9:09 am    Post subject: Reply with quote

I've seen such cards on ebay myself since there were always posts about it as a solution on hardforum five years ago, on gentoo forums two years ago.. But I didn't buy because of the drivers. Aren't those cards gonna use the mthca driver as opposed to the newer (world apart cards) that use mlx5?
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3105

PostPosted: Mon Feb 27, 2017 7:05 pm    Post subject: Reply with quote

Well, it's getting more interesting.
Talking about RDMA, I didn't mean IB-SATA wires, I've been thinking about computer-computer connection where one machine uses hard drives owned by the other. Pretty much the same way it would work with iSCSI, but skipping all the CPU-related overhead.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Tue Feb 28, 2017 11:25 am    Post subject: Reply with quote

szatox wrote:
but skipping all the CPU-related overhead.
I'm currently looking for options for that. Haven't decided yet.
This may be something to consider... If it's possible.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Tue Feb 28, 2017 12:24 pm    Post subject: Reply with quote

Apparently the post man was on sick leave.
The post office has a ninja in place of him.
Came to the gates and took an U-turn so fast that nobody saw him. Not even the dogs outside.
I think that's a record.

So I'll be picking up the cards myself from the local post office then.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3105

PostPosted: Sat Mar 18, 2017 11:52 am    Post subject: Reply with quote

So.... did you manage to get anything usable out of it?
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3311
Location: Rasi, Finland

PostPosted: Sun Mar 19, 2017 2:29 pm    Post subject: Reply with quote

I went and bought some other parts too. I decided to try out one thing I've always thought of trying out (You'll propably see that later).
I have all but one part now. I could, however, plug in the IB cards today already. I would need to maybe rebuild the kernel for the new motherboard and switch that motherboard in before I could actually test those IB cards.

The last part should arrive next week, so for the conveniece I'll pull all the things together at the same time.

Sorry for the delay. I'm very impatiently waiting to see the results.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page 1, 2, 3, 4  Next
Page 1 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum