Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Virtual NIC for Virtualbox, InfiniBand on host [SOLVED]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1461
Location: KUUSANKOSKI, Finland

PostPosted: Sat Jul 07, 2018 7:51 pm    Post subject: Virtual NIC for Virtualbox, InfiniBand on host [SOLVED] Reply with quote

I need to run a virtual server on VirtualBox as a part of a course I'm currently going.
Then virtual server needs:
  • ip of 10.20.85.20
  • gateway 10.20.0.1
  • subnet mask 255.255.0.0


Everything would be easy just by giving another ip to ib0 (IP-over-InfiniBand interface on the host) and then choosing bridged connection from VirtualBox settings. The problem is that VirtualBox doesn't have support for interfaces of type "Infiniband" (I assume). And I would like to avoid (physically) routing, yet another cable to my PC.
So now I'd (probably) need to create some kind of VLAN, maybe, for the 10.20.0.0 network? Then to get out of the VLAN to the internet the gateway address should probably be the address of host computer's ib0 ip address?

Am I in the right tracks? This is more advanced networking than what I'm used to. I'm learning here.
Please, do point out if there's some other simple way to get this working. The server addresses must not change.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...


Last edited by Zucca on Sun Jul 08, 2018 8:35 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42564
Location: 56N 3W

PostPosted: Sat Jul 07, 2018 8:06 pm    Post subject: Reply with quote

Zucca,

Am I missing something here?

Create a new bridge with no hardware.
On the bare metal, route 10.20.85.20 to the new bridge.

In Virtualbox, connect the network card to the new bridge.

You can route 10.20/16 to the bridge if you want and assign 10.20.85.20/16 statically to the VM.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1717

PostPosted: Sun Jul 08, 2018 9:33 am    Post subject: Reply with quote

Yes, another bridge (without any physical interfaces) seems to be the easiest way to go.
You will need IP forwarding enabled in kernel and (most likely) masquerade in iptables, just like any LAN router.
Unfortunately IPoIB does not support VLANS. I'm looking for a better way to handle this. E.g. Etherner over IB may or may not be a viable option. Or - maybe - eIPoIB.
They all have some drawbacks compared to each other, but each of them should be a workable solution.
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1461
Location: KUUSANKOSKI, Finland

PostPosted: Sun Jul 08, 2018 10:28 am    Post subject: Reply with quote

NeddySeagoon wrote:
Zucca,

Am I missing something here?
Nothing really. I'm just missing my skills. :) I've totally forgotten how to manage ip networking. :\ This is what happens when you "quicklearn" something just for one-time purpose.
Luckily I get to learn more ip(v4) networking on the course.

NeddySeagoon wrote:
Create a new bridge with no hardware.
On the bare metal, route 10.20.85.20 to the new bridge.

In Virtualbox, connect the network card to the new bridge.

You can route 10.20/16 to the bridge if you want and assign 10.20.85.20/16 statically to the VM.
I tried this last night, but only got to the point where I could ping my host machine's ip. I tried to route the traffic, but with no success. Now (next day) after reading szatox's post I remembered that I need to "sysctl -w net.ipv4.ip_forward=1".

Now I'm not sure if I need to set some nftables rules too or will just normal routing configuration do it.

Quick details:
  • Host PC network 10.0.11.0/28 (InfiniBand)
    • host ip 10.0.11.2 (dev ib0)
    • gateway 10.0.11.1
  • Guest PC network 10.20.0.0/16
    • guest ip 10.20.85.20
    • another guest in the future 10.20.85.25
    • gateway 10.20.0.1


@szatox: You could circumvent VLAN incapabilities etc., which aren't present (the incapabilities I mean) on normal Ethernet by switching your ib interfaces to Ethernet mode, but I guess by doing that you'll lose RDMA features of those cards. :(
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1717

PostPosted: Sun Jul 08, 2018 11:03 am    Post subject: Reply with quote

Zucca, as I mentioned you will probably need NAT like here:
Code:
-A POSTROUTING -s 10.20.0.0/16 -o ib0 -j MASQUERADE
(or -i vmbridge0 -o ib0)

Quote:
@szatox: You could circumvent VLAN incapabilities etc., which aren't present (the incapabilities I mean) on normal Ethernet by switching your ib interfaces to Ethernet mode, but I guess by doing that you'll lose RDMA features of those cards. :(
Of course I could. I suppose you could do that too, since direct support for ethernet seems to be a common trait for Mellanox HCA, but there are drawbacks to this approach too. Losing RDMA is one of them, degrading wire speed to 10G is another, then comes the advertised reliability and congestion control/path switching (I haven't stressed either of those networks nearly enough to draw any conclusions on the latter aspects).
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42564
Location: 56N 3W

PostPosted: Sun Jul 08, 2018 12:30 pm    Post subject: Reply with quote

Zucca,

As 10.20.0.0/16 is outside the Host PC network 10.0.11.0/28
How does the gateway at 10.20.0.1/16 get anywhere?

I have a horrible hack for that but it doesn't work with iproute2, so I have an interest in the answer.

As 10.20.0.0/16 is not routable over the internet, you will need NAT somewhere to get to the outside world.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1461
Location: KUUSANKOSKI, Finland

PostPosted: Sun Jul 08, 2018 8:32 pm    Post subject: Reply with quote

Thanks guys. It works now.
I needed compile all the missing nft modules, since this PC doesn't normally do any special networking things.

shellcmd: ifconfig brvbox0 :
brvbox0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.20.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::6c8d:36ff:fe29:13b3  prefixlen 64  scopeid 0x20<link>
        ether 6e:8d:36:29:13:b3  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 122  bytes 9804 (9.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
shellcmd: route :
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.11.1       0.0.0.0         UG    6      0        0 ib0
10.0.11.0       0.0.0.0         255.255.255.240 U     0      0        0 ib0
10.20.0.0       0.0.0.0         255.255.0.0     U     0      0        0 brvbox0
shellcmd: nft list ruleset :
table ip nat {
        chain prerouting {
                type nat hook prerouting priority 0; policy accept;
        }

        chain postrouting {
                type nat hook postrouting priority 100; policy accept;
                oifname "ib0" masquerade
        }
}
... and of course ipv4 forwarding is enabled. I may need to create some init script or similar to automate all these when needed.

Oh. Btw, if you see anything wrong here, please say so. I may need to specify the broadcast address later...
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum