HELP - static nat not handling arp correctly - SOLVED

[Moriah]

I have a classic bastion firewall gateway feeding into a dmz with a choke firewall between the dmz and the secured lan. I am using iptables. I am masquerading the lan, and that works properly. I also have several servers on the dmz that need to be internet facing. I am using static nat for them, as the dmz uses non-routable ip addressing. The internet side of the gateway connects to a cable modem. The cable modm also provides the gateway router address as one greater than the ip address of the cidr block as a whole.

My problem is that none of the servers are working properly, and after studying the traffic on the intenet interface of the gateway, I see that none of the static natted servers is replying to requests from the gateway for arp packets.

Does anyone have any idea what might be causing this problem?

I realize there will likely be questions for more detail, but this can at least start the conversation.

Thanks!
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[krinn]

[Moriah]

You *TOTALLY* misunderstood me!

This is a network setup that has worked well for 14 years. Lately, I have had to make some changes in the way it connects to the internet. My cable modem provider supplies me with a CIDR block of ip addresses, *NOT JUST ONE* !

My gateway firewall does static nat to each of the dmz based servers, one server per static publicly routable ip address in my CIDR block.

It also does masquarade to hide the choke firewall, which handles the lan full of workstations. The choke masquerades the workstations, and the gateway hides the choke by means of masquerading the choke itself, so 2 levels of nat.

The servers on the dmz each have their own non-routable ip address, each of which gets separately static natted by the gateway to the public static ip address assigned to that server.

My problem is the static natted servers seem to not be getting ARP replies back to the cable modem's gateway ip address. Consequently, they cannot communicate with the outside world. They communicate with each other and with the lan workstations just fine. The workstations communicate with the internet just fine.

Now do you understand the problem better?
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Ant P.]

First question: are the routing tables on the servers and gateway sane? That they're not replying to ARP can mean one of two things, either they're set to ignore that via sysctl or they believe there's no way to reply (i.e. link-local route).

Also how are you determining the ARPs aren't working? It might be useful to know where exactly the communication breaks down.

Since you haven't mentioned it I assume the LAN can reach the servers OK, just not the internet side?

(Before we waste too much time on this, make sure it's not just a loose cable. It happens...)

[krinn]

Sorry, i read it once and twice, really hard to get you. You make it hard with things like "non-routable ip address", i don't get it.

At least i get something: you are mistaking ARP, arp is use to translate IP->mac address, for internet you will use dns but not arp.
And not having arp messages going thru internet is a good thing.
If they can communicate with each other, it's a proof arp is working fine ; because arp will be use by one of your local device to identify and locate one of this natted server to speak with it.

If your problem is then inability to translate name into IP, that's dns problem, something you can test with a natted server, as ping -c1 8.8.8.8 should work then.
If your problem is inability to ping internet (just with the upper example), it's more a routing problem. While it could be firewall, generally you don't care what is getting out, more about what is getting in.

From what i get of your problem, it seems you have a routing problem

[Moriah]

@Ant P.:

[Ant P.]

I think I understand what's happening here...

ARP works directly atop the link-layer protocol, not IP, and doesn't pass through IP routers at all, so what matters to it is what each end of the wire thinks its own address is. The servers may be accessible at those public addresses from the outside world, but as far as they're concerned they only live on a 192.168.2/24 network segment.

Thus it's correct that they don't respond to ARP pings at those public addresses - but a regular ICMP `ping` directed at them should work by traversing the NAT (moreover, it should work the same either side of the NAT). If that doesn't, your problem is very likely in the gateway's iptables rules somewhere.

(side note: that's a huge amount of collisions on the gateway's ethernet, something seems off about that...)

[krinn]

the one that always wish to know who is who thru arp request is the router itself, it's the one everyone should contact to ask him then.

if you want server to get arp answer, set its route to 96.11.110.97, always aim at final destination
* set 192.168.2.13 route to 96.11.110.97
* bridge 192.168.2.1 eth0 and eth1 to allow packet from network 192.168.2.* to jump to network 96.11.110.97

[Moriah]

@Ant P.

Those interfaces on the gateway have been up for weeks, so the number of collisions is not as bad as you might think.

Yes, I see now why the cable-modem gateway is not getting replies to its arp requests for the static natted servers, but if it doesn't get a reply, will it not send packates to the ip address it did not get a reply from? Should they get a reply from the gateway router when it asks for those addresses? I am thinking not, so I am still confused.

Would you rather see an excerpt of the shell script that set the iptables rules, or an iptables -L dump, or something else. This has been making me crazy for weeks. This script used to work using an openvpn tunnel to get to the internet, but not that it has a direct connection, its broken.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Ant P.]

ARP not working doesn't necessarily mean IP won't, they're two separate protocols at the same layer and have different functions. You shouldn't be able to get ARP replies from those addresses at all, since they're not bound to interfaces on the network.

iptables -L would be very useful, yes (the mangle table will have the interesting stuff, but you can include the filter table too if you want).

[Moriah]

I am not using the mangle table, only filter and nat,

[Ant P.]

Right, nat table, that's what I meant to say... (it's been years since I last used iptables).

That lack of matches for the DNAT rules is a pretty big hint. I thought it'd work without this but... can you try adding all of those public addresses to the gateway's eth1?

[Moriah]

I have already tried:

[Ant P.]

[Moriah]

Oh! Using the newer "ip" command. I've never used that before, only ifconfig. OK, I'll give the ip man page a read, then try your suggestion. I suspected that somehow the other 96.11.110.* addresses were not making it thru the interface, but I had no good way to test it, and I did not know how to the the eth1 interface to pass them.

Thanks for your help. I will not likely be able to try this out until tomorrow evening, so don't think I am ignoring you...
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Moriah]

Apparently, it makes no difference. After a fresh boot, which runs the firewall script, I get:

[Ant P.]

Yes, knowing what this looks like from outside might help.

Run nmap on it too, I forgot that existed... very useful for debugging connectivity issues like this. Recommend you use the GUI (USE=zenmap) if possible because it adds a "radar" view that shows the distance each host is at.

[Moriah]

I will try to run some experiments from an outside vantage point. I will connect a laptop to the DSL line, which used DHCP to give the wireless router an ip address, and the router nats the wireless connections, so the ip address of the laptop will be double-natted, but I have a web browser on the wireless router's web page, so I can determine the address the router faces the internet with. That would be the address we need to look for using tcpdump on the cable modem gateway router.

Hopefully, I will have some time tomorrow (2018_04_21) to try these experiments. I will post the results here.

Did you have any time to look at the iptables -L displays I posted?
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Ant P.]

I've looked at them, but I can't see any obvious mistakes. The DNAT rules aren't being matched, but that looks like a symptom, not a problem.

[Moriah]

I'm thinking they aren't being matched because the packet never makes it that far. I'm thinking we need to look closer at the input chain.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Moriah]

I did some tcpdump-ing and now I can see that the snat-ting is working ok. When I dump packets containing the web server's ip address, then ping 8.8.8.8 from the web server, I can see the pings going out, but the replies do not come back in:

[Hu]

First, apologies for neglecting your original private message. Your configuration is too complex for me to reason about in the small bits of time I can easily spend diagnosing network issues. You started this thread before I was able to clear enough time to dig through it, and seemed to be getting good advice.

DNAT should only matter for new incoming traffic. Responses to previous outbound traffic are handled separately, because remapping entries should already exist.

tcpdump sees traffic before netfilter can filter it, so even packets destined to be DROPped can be seen in tcpdump (but only on the host which performs the DROP; hosts deeper in never get the traffic, so they never have a chance to drop it). If a tcpdump on your public interface shows no pongs arriving, then your problem is that the peer's pongs never arrive, not that they are mishandled on arrival. Since the same peer pongs for one of your other tests, we can assume that it usually answers ping requests. I suggest you cease using grep to filter this output interactively, and instead rely on some combination of tcpdump filters and post-processing of a raw capture. As is, you may be hiding important clues because they fail to match your grep pattern. If I were to guess, I would say that the source address used on the unanswered pings causes upstream to be able to route the pongs back to you.

[Moriah]

Thank you for the response. I will look into your idea this evening, as I am busy until then. I just used grep because I wa too lazy to look up the pcap filter command I really needed.

Also, if I go outside my own 96.11.110.96/29 nework and try to connect to 96.11.110.101 (my web server on my dmz) I fail to see anything with tcpdump.

So outbound originating packets use -m state --state ESTABLISHED,RELATED to get their reply, but dnat still has problems too.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Moriah]

This was finally solved. See:

https://forums.gentoo.org/viewtopic-p-8219104.html#8219104
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.