| View previous topic :: View next topic |
| Author |
Message |
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
 Posted: Sat Jul 07, 2018 7:51 pm Post subject: Virtual NIC for Virtualbox, InfiniBand on host [SOLVED] |
 |
|
I need to run a virtual server on VirtualBox as a part of a course I'm currently going.
Then virtual server needs:
- ip of 10.20.85.20
- gateway 10.20.0.1
- subnet mask 255.255.0.0
Everything would be easy just by giving another ip to ib0 (IP-over-InfiniBand interface on the host) and then choosing bridged connection from VirtualBox settings. The problem is that VirtualBox doesn't have support for interfaces of type "Infiniband" (I assume). And I would like to avoid (physically) routing, yet another cable to my PC.
So now I'd (probably) need to create some kind of VLAN, maybe, for the 10.20.0.0 network? Then to get out of the VLAN to the internet the gateway address should probably be the address of host computer's ib0 ip address?
Am I in the right tracks? This is more advanced networking than what I'm used to. I'm learning here.
Please, do point out if there's some other simple way to get this working. The server addresses must not change.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
Last edited by Zucca on Sun Jul 08, 2018 8:35 pm; edited 1 time in total |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W
|
| Posted: Sat Jul 07, 2018 8:06 pm Post subject: |
|
|
Zucca,
Am I missing something here?
Create a new bridge with no hardware.
On the bare metal, route 10.20.85.20 to the new bridge.
In Virtualbox, connect the network card to the new bridge.
You can route 10.20/16 to the bridge if you want and assign 10.20.85.20/16 statically to the VM.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3137
|
| Posted: Sun Jul 08, 2018 9:33 am Post subject: |
|
|
Yes, another bridge (without any physical interfaces) seems to be the easiest way to go.
You will need IP forwarding enabled in kernel and (most likely) masquerade in iptables, just like any LAN router.
Unfortunately IPoIB does not support VLANS. I'm looking for a better way to handle this. E.g. Etherner over IB may or may not be a viable option. Or - maybe - eIPoIB.
They all have some drawbacks compared to each other, but each of them should be a workable solution. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Sun Jul 08, 2018 10:28 am Post subject: |
|
|
| NeddySeagoon wrote: | Zucca,
Am I missing something here? |
Nothing really. I'm just missing my skills. :) I've totally forgotten how to manage ip networking. :\ This is what happens when you "quicklearn" something just for one-time purpose.
Luckily I get to learn more ip(v4) networking on the course.
| NeddySeagoon wrote: | Create a new bridge with no hardware.
On the bare metal, route 10.20.85.20 to the new bridge.
In Virtualbox, connect the network card to the new bridge.
You can route 10.20/16 to the bridge if you want and assign 10.20.85.20/16 statically to the VM. |
I tried this last night, but only got to the point where I could ping my host machine's ip. I tried to route the traffic, but with no success. Now (next day) after reading szatox's post I remembered that I need to "sysctl -w net.ipv4.ip_forward=1".
Now I'm not sure if I need to set some nftables rules too or will just normal routing configuration do it.
Quick details:- Host PC network 10.0.11.0/28 (InfiniBand)
- host ip 10.0.11.2 (dev ib0)
- gateway 10.0.11.1
- Guest PC network 10.20.0.0/16
- guest ip 10.20.85.20
- another guest in the future 10.20.85.25
- gateway 10.20.0.1
@szatox: You could circumvent VLAN incapabilities etc., which aren't present (the incapabilities I mean) on normal Ethernet by switching your ib interfaces to Ethernet mode, but I guess by doing that you'll lose RDMA features of those cards. :(
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3137
|
| Posted: Sun Jul 08, 2018 11:03 am Post subject: |
|
|
Zucca, as I mentioned you will probably need NAT like here:
| Code: | | -A POSTROUTING -s 10.20.0.0/16 -o ib0 -j MASQUERADE |
(or -i vmbridge0 -o ib0)
| Quote: | @szatox: You could circumvent VLAN incapabilities etc., which aren't present (the incapabilities I mean) on normal Ethernet by switching your ib interfaces to Ethernet mode, but I guess by doing that you'll lose RDMA features of those cards.  |
Of course I could. I suppose you could do that too, since direct support for ethernet seems to be a common trait for Mellanox HCA, but there are drawbacks to this approach too. Losing RDMA is one of them, degrading wire speed to 10G is another, then comes the advertised reliability and congestion control/path switching (I haven't stressed either of those networks nearly enough to draw any conclusions on the latter aspects). |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54244
Location: 56N 3W
|
| Posted: Sun Jul 08, 2018 12:30 pm Post subject: |
|
|
Zucca,
As 10.20.0.0/16 is outside the Host PC network 10.0.11.0/28
How does the gateway at 10.20.0.1/16 get anywhere?
I have a horrible hack for that but it doesn't work with iproute2, so I have an interest in the answer.
As 10.20.0.0/16 is not routable over the internet, you will need NAT somewhere to get to the outside world.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland
|
| Posted: Sun Jul 08, 2018 8:32 pm Post subject: |
|
|
Thanks guys. It works now.
I needed compile all the missing nft modules, since this PC doesn't normally do any special networking things.
| shellcmd: ifconfig brvbox0 : | brvbox0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.20.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::6c8d:36ff:fe29:13b3 prefixlen 64 scopeid 0x20<link>
ether 6e:8d:36:29:13:b3 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 122 bytes 9804 (9.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 |
| shellcmd: route : | Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.0.11.1 0.0.0.0 UG 6 0 0 ib0
10.0.11.0 0.0.0.0 255.255.255.240 U 0 0 0 ib0
10.20.0.0 0.0.0.0 255.255.0.0 U 0 0 0 brvbox0 |
| shellcmd: nft list ruleset : | table ip nat {
chain prerouting {
type nat hook prerouting priority 0; policy accept;
}
chain postrouting {
type nat hook postrouting priority 100; policy accept;
oifname "ib0" masquerade
}
} |
... and of course ipv4 forwarding is enabled. I may need to create some init script or similar to automate all these when needed.
Oh. Btw, if you see anything wrong here, please say so. I may need to specify the broadcast address later...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
