Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to route network traffic for private link? [SOLVED]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
dufeu
l33t
l33t


Joined: 30 Aug 2002
Posts: 873
Location: US-FL-EST

PostPosted: Fri Nov 24, 2017 6:23 pm    Post subject: How to route network traffic for private link? [SOLVED] Reply with quote

I can't seem to find an example for this case. What I want to do is have two PCs directly connected with 10Ge {SPF+} as well as keep the current 1Ge connections to my LAN for each PC.

PC1 is my primary workstation while PC2 is my NAS. The NAS offers both nfs and samba based shares across my network. Ny BluRay reader/writer is on PC1. I both read/restore from the BD reader to the NAS and burn from the NAS to the BD writer. The current 1Ge connections across my switch limits me to a max throughput of real life measured 900Mbps. The BD r/w unit maxes at 300Mbps.

Both PCs are connected to the same switch. The LAN segment is 192.168.1.0/24, PC1's 1Ge nic is 192.168.1.200 and PC2's 1Ge nic is 192.168.1.202.

I do very large data transfers between PC1 and PC2 on a regular basis. What I want to do is keep all traffic between PC1 and PC2 off my LAN altogether.

I've recently added Mellanox 10Ge cards to each PC and the cards seem to work fine.

I have some experience routing two different subnets with their own respective routers through a two nic box, but this case is not that. It's also been many years {I stopped building my own routers} since I've worked with iptables.

I believe I need to set on both PCs:
Code:
echo 1 > /proc/sys/net/ipv4/ip_forward


On PC1 {192.168.1.200} where eth0 = 1Ge and eth0 = 10Ge:
Code:
iptables -A -o eth1 -d !192.168.1.202 -j drop
iptables -A -o eth0 -d 192.168.1.202 -j drop

On PC2 {192.168.1.202} where enp9s0 = 1Ge and enp3s0 = 10Ge:
Code:
iptables -A -o enp3s0 -d !192.168.1.200 -j DROP
iptables -A -o enp9s0 -d 192.168.1.200 -j DROP


I think the IP addresses of the respective 10Ge cards don't matter. If any other PC asks the NAS for data, the NAS will send the packets to that PC via the 1Ge nic. If my workstation asks for data, then the NAS should sent it via it's 10Ge nic to the workstations 10Ge nic.

Is this correct? Or did I get it backwards? Or, should I just use 'route add'? I keep feeling I've missed something.

One thing I'm not sure of is if my workstation will recognize the nfs shares properly. i.e. do I need to change the nfs mount entries in /etc/fstab on my workstation and add corresponding entries to /etc/exports on my NAS?
_________________
People whom think M$ is mediocre, don't know the half of it.


Last edited by dufeu on Sat Nov 25, 2017 2:57 pm; edited 1 time in total
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 6345
Location: almost Mile High in the USA

PostPosted: Fri Nov 24, 2017 6:58 pm    Post subject: Reply with quote

If both machines are connected to a switch, the hardware switch will prevent traffic from permeating the rest of the network, unless someone somehow sets the switch to promiscuous mode.

If you actually want 11Gbps (the original 1Gb and the new 10Gb) then this will require some trickery as you'd need to bond the network adapters.

If you just want to use the 10Gb link for just the two machines together, easiest way is to just make the 10Gb cards on their own subnet and just use that subnet when you need that speed boost, else use the 1Gb network address.

e.g.
Box1:
eth1g: 192.168.1.201/24 def route 192.168.1.1
eth10g: 192.168.2.2/24
Box2:
eth1g: 192.168.1.202/24 def route 192.168.1.1
eth10g: 192.168.2.3/24

Note there is no default route for 10g. Thus no general traffic will be sent out that port. Also you don't need a router on the 192.168.2.x net, it's just one fast subnet. Unfortunately you will need to treat the new ethernet as a different host, but that's no big deal since you only have two machines (one machine - the server) that needs to keep two sets of configuration (like two exports, two sets of samba rules) - one for the fast port and one for the slow port.

You don't want to forward... not sure how that helps you. You might well ditch the 1Gb card and share everything on the same 10Gb switch and let the switch handle things - this is the easiest solution. The iptables you present have a problem when a remote machine tries contacting one of these machines and gets the wrong ethercard, and that packet gets dropped... If you were experimenting with this and ended up with the problem in https://forums.gentoo.org/viewtopic-t-1071888-highlight-.html , that would explain the problems you see in the other thread, bad iptables rules definitely would cause that weird behavior.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
dufeu
l33t
l33t


Joined: 30 Aug 2002
Posts: 873
Location: US-FL-EST

PostPosted: Sat Nov 25, 2017 2:54 pm    Post subject: Reply with quote

eccerr0r wrote:
If you actually want 11Gbps (the original 1Gb and the new 10Gb) then this will require some trickery as you'd need to bond the network adapters.
Bonding is one of those things I try hard to avoid. Always. Bonding is really only useful when you have multiple users always needing simultaneous access to the same server. If you want faster throughput between two individual computers, you have to have the same multiple nics set up and bonded on both computers. Too much effort and aggravation. Bridging and vlans are plenty enough fun to troubleshoot already.

eccerr0r wrote:
If you just want to use the 10Gb link for just the two machines together, easiest way is to just make the 10Gb cards on their own subnet and just use that subnet when you need that speed boost, else use the 1Gb network address.

e.g.
Box1:
eth1g: 192.168.1.201/24 def route 192.168.1.1
eth10g: 192.168.2.2/24
Box2:
eth1g: 192.168.1.202/24 def route 192.168.1.1
eth10g: 192.168.2.3/24
Aack! I don't know why I was so fixated on either routing or forwarding. I completely didn't get even an inkling of using multiple subnets cross my mind.

eccerr0r wrote:
Note there is no default route for 10g. Thus no general traffic will be sent out that port. Also you don't need a router on the 192.168.2.x net, it's just one fast subnet. Unfortunately you will need to treat the new ethernet as a different host, but that's no big deal since you only have two machines (one machine - the server) that needs to keep two sets of configuration (like two exports, two sets of samba rules) - one for the fast port and one for the slow port.
I have no problem with having two sets of rules - one set for each subnet. It's a perfectly sensible thing to do.

eccerr0r wrote:
The iptables you present have a problem when a remote machine tries contacting one of these machines and gets the wrong ethercard, and that packet gets dropped...
Assuming the two PCs 10G nics are directly linked, this wouldn't be a problem, correct? The rest of the physical network would only see the 1G nics for both PCs.

eccerr0r wrote:
If you were experimenting with this and ended up with the problem in https://forums.gentoo.org/viewtopic-t-1071888-highlight-.html , that would explain the problems you see in the other thread, bad iptables rules definitely would cause that weird behavior.
I haven't experimented with this 10G setup yet. The problem in that thread has been ongoing now for at least 2 months. I only put in the 10G cards 2 days ago to confirm they both worked. I think the problem I have that thread might be related to bug# 609682. I've been getting the "error NTNETLINK" message on both the problem PCs. I'm revisiting that thread later today.

It looks like there are two reasonable solutions:
  1. Connect all nics to the switch and use subnets to separate traffic. Let the switch control traffic exposure to the rest of the LAN.
  2. Effectively "air gap" the 10G network by directly connecting the PCs 10G nics and use IP tables or subnets.
I agree that subnets are probably the simplest and easiest to do. Also, I haven't run across any mention of directly connecting two computer's SPF+ ports yet. So I don't know if I need a cross over cable or if the SPF+ port definition includes the ability to automatically connect either directly or to a switch.

The whole SPF+ thing is new to me. I haven't worked in a business network for awhile now. I see a lot of turn on the used equipment market and the pricing of SPF+ era equipment is finally within reasonable reach of the home network. The 10G Mellanox cards cost me less than $25 for each card and cable. The 48 port Quanta L4M switch was less than $100 including shipping.

While CAT 7 and 10G RJ11 ports are nice and something to look forward to, pricing of cards and switches is still obscene for the home office. I've absolutely zero interest in spending $3,000 or more for an 8 or 16 port 10G switch.

I'm marking this solved even though I haven't finished setting everything up yet. Once I was reminded of it, I know subnets will work. ;)

Thank you!!

edit: same day - Sat Nov 25 16:15:07 EST 2017

New 10G cards and private subnet configured. Works as expected. View from the workstation: I've been monitoring network traffic. For the most part, the 1G connection has been pretty steady state with 80Mbps down and 40Mbps up with some long term bittorrent based activity happening. Traffic on the 10G connection has been much spikier with spikes directly related to personal activity.

Highest spike seen so far: 1600 Mbps.

Everything is working as expected. There is no unnecessary traffic on the rest of the LAN and the 10G private network segment is invisible everywhere else on my local network. My 48 port Gigabit Ethernet 2 port 10 Gigabit SPF+ switch is obviously not strained with 2 SPF+ and 6 mixed Fast/Gigabit connections.

For anyone interested in a real world working example - includes all configuration changes and steps after the cards were installed and confirmed working. This is the text file I save for myself so I can recall what I did.

blaze = server
pyrogyro = workstation client
configuration notes cut & paste:
blaze

blaze /etc/conf.d # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
3: enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:d1:82:a6 brd ff:ff:ff:ff:ff:ff
4: enp10s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:d1:82:a7 brd ff:ff:ff:ff:ff:ff
5: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:02:c9:54:94:c0 brd ff:ff:ff:ff:ff:ff

--0--

blaze /etc/conf.d # cat net
config_enp9s0="dhcp"
config_enp10s0=null
config_enp3s0="192.168.2.102/24"

--0--

blaze /etc # cd init.d
blaze init.d # ln -s net.lo net.enp3s0
blaze init.d # rc-update add net.enp3s0
 * service net.enp3s0 added to runlevel default
 
--0--

blaze /etc # cat exports
# /etc/exports: NFS file systems being exported.  See exports(5).

#       exports for the 1G nic {LAN wide availabilty}
/Silo01                 192.168.1.0/24(rw,no_root_squash,sync,subtree_check)
/Silo02                 192.168.1.0/24(rw,no_root_squash,sync,subtree_check)
/Silo03                 192.168.1.0/24(rw,no_root_squash,sync,subtree_check)
#/Video01               192.168.1.0/24(rw,no_root_squash,sync,subtree_check)
/MainArray1             192.168.1.0/24(rw,no_root_squash,sync,subtree_check)
/Private                192.168.1.0/24(rw,no_root_squash,sync,subtree_check)

#       exports for the 10G nic {local switch only availabilty}
/home                   192.168.2.0/24(rw,no_root_squash,sync,subtree_check)
/Silo01                 192.168.2.0/24(rw,no_root_squash,sync,subtree_check)
/Silo02                 192.168.2.0/24(rw,no_root_squash,sync,subtree_check)
/Silo03                 192.168.2.0/24(rw,no_root_squash,sync,subtree_check)
#/Video01               192.168.2.0/24(rw,no_root_squash,sync,subtree_check)
/Private                192.168.2.0/24(rw,no_root_squash,sync,subtree_check)

--0--

blaze /etc/samba # cat smb.conf
...
##      interface options
#socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192

# permit only local network PCs/devices/users
bind interfaces only = yes
interfaces = lo enp9s0 enp3s0
hosts allow = 127.0.0.1 192.168.1.0/24 192.168.2.0/24
hosts deny = 0.0.0.0/0
...

== after rebooting ==

blaze /etc/conf.d # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
3: enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:d1:82:a6 brd ff:ff:ff:ff:ff:ff
4: enp10s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:d1:82:a7 brd ff:ff:ff:ff:ff:ff
5: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:02:c9:54:94:c0 brd ff:ff:ff:ff:ff:ff
blaze /etc/conf.d # ip route
default via 192.168.1.1 dev enp9s0 src 192.168.1.202 metric 3
127.0.0.0/8 via 127.0.0.1 dev lo
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.202 metric 3
192.168.2.0/24 dev enp3s0 proto kernel scope link src 192.168.2.102

--0--
==O==
--0--

pyrogyro

pyrogyro conf.d # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 1000
    link/ether fc:aa:14:5a:13:c9 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:02:c9:54:bc:48 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fc:aa:14:5a:13:c9 brd ff:ff:ff:ff:ff:ff
   
--0--

pyrogyro conf.d # cat net
rc_net_lo_provide="!net"
rc_net_eth0_provide="!net"
rc_net_eth1_provide="!net"

config_eth0=null
config_br0="192.168.1.200/24"
bridge_br0="eth0"
rc_net_br0_need="net.eth0"
dns_servers_br0="8.8.4.4 8.8.8.8 208.67.222.222"
routes_br0="default via 192.168.1.1"

config_eth1="192.168.2.100/24"

--0--

pyrogyro init.d # ln -s net.lo net.eth1
pyrogyro init.d # rc-update add net.eth1
 * service net.eth1 added to runlevel default


--0--

pyrogyro ~ # grep 168.2 /etc/fstab
#       blaze   subnet: 192.168.2.0/24
192.168.2.102:/home             /zmnts/blaze/home       nfs     nfsvers=4,rsize=1048576,wsize=1048576,noauto,timeo=20,acregmin=10,acregmax=120,acdirmax=120
192.168.2.102:/Private          /Private                nfs     nfsvers=4,rsize=1048576,wsize=1048576,noauto,timeo=20,acregmin=10,acregmax=120,acdirmax=120
192.168.2.102:/Silo01           /MainArray1             nfs     nfsvers=4,rsize=1048576,wsize=1048576,noauto,timeo=20,acregmin=10,acregmax=120,acdirmax=120
192.168.2.102:/Silo02           /Silo02                 nfs     nfsvers=4,rsize=1048576,wsize=1048576,noauto,timeo=20,acregmin=10,acregmax=120,acdirmax=120
192.168.2.102:/Silo03           /Silo03                 nfs     nfsvers=4,rsize=1048576,wsize=1048576,noauto,timeo=20,acregmin=10,acregmax=120,acdirmax=120
192.168.2.102:/Silo04           /Silo04                 nfs     nfsvers=4,rsize=1048576,wsize=1048576,noauto,timeo=20,acregmin=10,acregmax=120,acdirmax=120
192.168.2.102:/Video01          /Video01                nfs     nfsvers=4,rsize=1048576,wsize=1048576,noauto,timeo=20,acregmin=10,acregmax=120,acdirmax=120

--0--

pyrogyro samba # cat smb.conf
...
##      interface options
#socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192

# permit only local network PCs/devices/users
interfaces = lo eth0 br0 eth1
bind interfaces only = yes
hosts allow = 127.0.0.1 192.168.1.0/24 192.168.2.0/24
hosts deny = 0.0.0.0/0
...

--0--

== after rebooting ==

pyrogyro ~ # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 1000
    link/ether fc:aa:14:5a:13:c9 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:02:c9:54:bc:48 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fc:aa:14:5a:13:c9 brd ff:ff:ff:ff:ff:ff
pyrogyro ~ # ip route
default via 192.168.1.1 dev br0 metric 5
127.0.0.0/8 via 127.0.0.1 dev lo
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.200
192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.100

_________________
People whom think M$ is mediocre, don't know the half of it.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum