Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
FWMARK directed routing for localhost originated traffic
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
dbishop
Tux's lil' helper
Tux's lil' helper


Joined: 08 Dec 2007
Posts: 101

PostPosted: Sun Aug 21, 2011 1:17 am    Post subject: FWMARK directed routing for localhost originated traffic Reply with quote

This seems to be a difficult problem. I have a single host with three NICs, each connected to a different ISP . This host is *not* routing/forwarding for any other hosts.

I am trying to get all traffic for certain ports (22, 25, and 123) to route via eth0. All other traffic should go via eth1 unless I set something to change to eth0 or eth2 (like changing an iptables rule or a routing table or a routing rule, etc.)

eth0's IP is 172.16.1.100 and its default gateway is 172.16.1.129 and its network is 172.16.1.128/25 (eth1 is 192.168.0.2, 192.168.0.0/24, 192.168.0.1 ; eth2 is 192.168.100.2, 192.168.100.0/24, 192.168.100.1)

I have tried all sorts of combinations of fwmark, ip route tables/rules, and snat'ing. All traffic stubbornly obeys the main table's default gateway (surprise, surprise...)

Is there a way to do this? It seems that fwmark won't work with localhost because the routing decision is made before marking. I've been thinking about the 'route' target in iptables, but that doesn't seem like it will work either.

I have searched this forum (and countless others) and none address this issue (or worked it to a successful resolution. the one workaround I found won't work for me.

Anyone?
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.

PostPosted: Sun Aug 21, 2011 5:31 am    Post subject: Reply with quote

You should be marking in the "prerouting" chain of the "mangle" table, and then using SNAT in the "postrouting" chain of the "nat" table.

Edit: correction, per below: you should using the "output" chain of the "mangle" table.

I don't know what sources you have consulted, but this is a good one:
http://linux-ip.net/html/adv-multi-internet.html
_________________
patrix_neo wrote:
The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it.


Last edited by Bones McCracker on Mon Aug 22, 2011 4:32 am; edited 2 times in total
Back to top
View user's profile Send private message
truc
Advocate
Advocate


Joined: 25 Jul 2005
Posts: 3199

PostPosted: Sun Aug 21, 2011 7:42 am    Post subject: Reply with quote

Quote:
mangle:
This table is used for specialized packet alteration. [...] and OUTPUT (for altering locally-generated packets before routing)

_________________
The End of the Internet!
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.

PostPosted: Sun Aug 21, 2011 7:46 am    Post subject: Reply with quote

truc wrote:
Quote:
mangle:
This table is used for specialized packet alteration. [...] and OUTPUT (for altering locally-generated packets before routing)

Thank you, truc.

My mistake. The "prerouting" chain is for inbound only. Use the "output" chain of the mangle table (and the "postrouting" chain of the "nat" table, for the corresponding SNAT rules).
_________________
patrix_neo wrote:
The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it.
Back to top
View user's profile Send private message
dbishop
Tux's lil' helper
Tux's lil' helper


Joined: 08 Dec 2007
Posts: 101

PostPosted: Mon Aug 22, 2011 1:33 am    Post subject: Reply with quote

This is now working, thank you all for the help. For those that may follow this discussion, the the issue that's different is that this is written for fwmark'ing traffice generated by the marking host itself -- most of the howto's written with the expectation that the router doing the firewall marking will be doing so for the packets it is forwarding, that is, not for packets locally generated by the host itself. The few that discuss the mangle table's OUTPUT chain are a little confusing (and somewhat contradictory at times).

For clarity, I have a single host that has three NICs, each connecting to a different ISP. I desired some routes to be chosen by UID owner, some by port, and everything else by default route.

There are several kernel options that need to be included. I have probably switched on a few too many trying to make this work, but there are several you'll need. My best advice would be to use the ncurses interface and then search for MARK. You'll see a bunch of tags and their locations and dependencies. The point is that for Gentoo and 2.6.39 gentoo-sources kernel this definitely works, so if you're having trouble like I did, keep going. Here are the details of what I ultimately got working.

This snippet sets up the fwmarks in the mangle table's OUTPUT chain, followed by the nat table's POSTROUTING rules:

Code:

ETH1_IP="192.168.0.223"
ETH2_IP="192.168.20.6"
ETH3_IP="192.168.100.56"
$ISP1_ETH="eth0"
$ISP2_ETH="eth1"
$ISP3_ETH="eth2"

# FWMARK Port 22 (tcp), 25 (tcp) and Port 123 (udp)
iptables -t mangle -A OUTPUT -p tcp --dport 22  -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -p udp --dport 123 -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -p tcp --dport 25 -j MARK --set-mark 3

# FWMARK base on UID
iptables -t mangle -A OUTPUT -m owner --uid-owner userone -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -m owner --uid-owner usertwo -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -m owner --uid-owner userthree -j MARK --set-mark 3

# SNAT for advanced routing redirects
iptables -t nat -A POSTROUTING -o $ISP1_ETH  -j SNAT --to-source $ETH1_IP
iptables -t nat -A POSTROUTING -o $ISP2_ETH  -j SNAT --to-source $ETH2_IP
iptables -t nat -A POSTROUTING -o $ISP3_ETH  -j SNAT --to-source $ETH3_IP


These rules should mark all NTP, SSH and SMTP traffic with 0x1, 0x2, and 0x3, respectively. The SNAT rules make sure that the source IP is sensible for the interface being used.

There are a few other things that need to get done, and these are pretty close to what the myriad other howto's say. You'll need some tables defined in /etc/iproute2/rt_tables. Mine looks like this:
Code:
255 local
254 main
253 default
102 isp3
101 isp2
100 isp1
0   unspec


Then there are the routes and rules. I am doing this from a script after all the interfaces are up. I have found, for my purposes anyway, that this is the simplest way to make sure everything is right, without having to second-guess what the Gentoo net scripts may decide do (i.e., I have everything configured correctly in /etc/conf.d/net but this makes everything just as I like it :roll:
Code:
# Flush the routing tables and rules
ip route flush table main
ip route flush table isp1
ip route flush table isp2
ip route flush table isp3
ip rule flush

# fwmark rules entries
ip rule add lookup default priority 32767
ip rule add lookup main priority 32766
ip rule add fwmark 1 priority 101 table isp1
ip rule add fwmark 2 priority 102 table isp2
ip rule add fwmark 3 priority 103 table isp3

# Add the main routing table entries
ip route add 127.0.0.0/8 via 127.0.0.1 dev lo
ip route add 192.168.0.128/25 dev eth0 src 192.168.0.223
ip route add 192.168.20.0/24 dev eth1 src 192.168.20.6
ip route add 192.168.100.0/24 dev eth2 src 192.168.100.56
ip route add default via 192.168.20.1

# isp1 table entries
ip route add 192.168.0.128/25 dev eth0 table isp1
ip route add default via 192.168.0.129 dev eth0 table isp1

# isp2 table entries
ip route add 192.168.20.0/24 dev eth1 table isp2
ip route add default via 192.168.20.1 dev eth1 table isp2

# isp3 table entries
ip route add 192.168.100.0/24 dev eth2 table isp3
ip route add default via 192.168.100.1 dev eth2 table isp3

# Flush the route cache
ip route flush cache


And you'll need to do this somewhere convenient too:
Code:
echo 1 > /proc/sys/net/ipv4/ip_forward
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 1 > $f ; done


Hopefully someone will find this useful, and equally hopefully, I haven't missed anything or mistyped anything either. If I have, please correct it.

Thanks again.
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.

PostPosted: Mon Aug 22, 2011 4:57 am    Post subject: Reply with quote

Nice write-up. After a while, when you've let your configuration stabilize, you might want to copy a final version of this over to the "tips and tricks" forum or to the Gentoo Wiki.

Quote:
And you'll need to do this somewhere convenient too:
Code:
echo 1 > /proc/sys/net/ipv4/ip_forward
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 1 > $f ; done


One good place to set those is in /etc/sysctl.conf. Another option, if you are scripting it, is to use the sysctl command. The code you have above is more useful than these two options if your interfaces may change.

On your machine, ip_forward is quite possibly already defaulting to 1, since you may have the kernel config option "IP: Advanced Router" enabled, which I believe changes the default value for that sysctl, along with one or two others.

I'm not sure you actually need or want to enable forwarding, if you are only routing packets originating on the local host, although it's conceivable that this is required for other routing functions.
_________________
patrix_neo wrote:
The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it.
Back to top
View user's profile Send private message
dbishop
Tux's lil' helper
Tux's lil' helper


Joined: 08 Dec 2007
Posts: 101

PostPosted: Mon Aug 22, 2011 11:22 am    Post subject: Reply with quote

Okay, will-do on the Gentoo Wiki, etc., good idea. I've written a few before. This was written partly to say let you know what I finally got working and partly to help others gain some insight, gentoo-y-gentoo. And because of that, the writing p.o.v. does seem to annoyingly flip between those two purposes.

Regarding the /etc/sysctrl.conf, that's where I have IP Forwarding set to 1. Not sure if it's needed for DNAT or SNAT either, but somewhere in the back of my mind I recall that for nat'ing to work this has to be a 1, and that the advanced routing kernel option seems to set it anyway (it is set for me). At some point I'll try it set to 0, but many readers of this will probably be using such a box as a LAN router too and will need it.

As for the rp_filter, once I get everything all trimmed up I will probably reduce this to only the interfaces involved, i.e., eth0, eth1 and eth2 in this case.

The /proc/sys/ipv4/... settings are mostly a mystery to me. they're not hard to understand, they're just hard to find out about, and even harder to figure out the side effects. I only ever look here when someone else tells me to. Some of these I know from experience, some not so much, and much of the information is hard to understand. For example, I read more than one site on this very subject that said to set rp_filter to "2", but when I looked up the docs, it reportedly is a boolean. Maybe "2" is a quantum state for this setting, maybe docs are out of sync, maybe it's a kernel version thing, who knows. Question: Do you know someplace reliable and straightforward that covers what all these do and their interactions and any collateral/consequential effects?

Thanks again for all your help. All you guys are great :D
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.

PostPosted: Mon Aug 22, 2011 12:59 pm    Post subject: Reply with quote

dbishop wrote:
The /proc/sys/ipv4/... settings are mostly a mystery to me. they're not hard to understand, they're just hard to find out about, and even harder to figure out the side effects. I only ever look here when someone else tells me to. Some of these I know from experience, some not so much, and much of the information is hard to understand. For example, I read more than one site on this very subject that said to set rp_filter to "2", but when I looked up the docs, it reportedly is a boolean. Maybe "2" is a quantum state for this setting, maybe docs are out of sync, maybe it's a kernel version thing, who knows. Question: Do you know someplace reliable and straightforward that covers what all these do and their interactions and any collateral/consequential effects?

I recall something of a dearth of documentation on these too. I think the best source is probably the kernel docs:
/usr/src/Documentation/networking/ (e.g. ip-sysctl.txt)
/usr/src/linux/Documentation/sysctl (e.g., net.txt)
_________________
patrix_neo wrote:
The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum