Network routing and redundancy question

[depontius]

My home network is built as a cable modem with a Netgear router attached, running NAT. Attached to the Netgear router I have a pair of dual-homed servers, and the other side of both servers goes to my main switch, where the rest of my LAN exists. So I essentially have what looks like a DMZ, though I don't offer outside services. (Comcast TOS) My servers route between the LAN and the DMZ - no NAT there.

The idea behind the dual servers is to have redundant fail-over, or at least the ability to come back up quickly. It doesn't really work that way, because my Netgear has a static route to reach my LAN, and it can only point to one of the servers. If that servers goes down, I'll have to get into the Netgear to point the LAN static route at the other server. Plus I have to do this from the server itself, or directly plugged into the Netgear, because at this point the static route won't be working, and my LAN won't be able to get there.

I would like to use an aliased IP as the static route target for the Netgear. That way each server has its own path everywhere, but routing for my LAN all occurs on aliased IPs. I can bring the aliased IPs up and down as needed. I might take an ARP flush, or perhaps just reset the router.

So here's the question... From what I can tell, all I can do is route packets from my LAN on eth0 out to my DMZ on eth1. I can tell the Netgear to send all LAN packets to my aliased IP, which will be eth1 on one of the servers. This looks like it might all work, because it's all traveling over the same ethernet cable, no matter what the IP is. But it doesn't feel right. It feels like I might have packets from the LAN coming out on eth1, from my server's native IP, going to the Netgear and out to Comcast. The return packet comes from Comcast and the Netgear sends it back to the aliased IP on my server, same device.

I guess when routing packets, the IPs of the router don't really matter, as long as you got there and as long as it knows the next hop?

Comments?

[your_WooDness]

Hi,

normally it shouldn't matter, but it might depend on your iptable configuration and if it binds to IP addresses. For automatic failover you could use ucarp or corosync/pacemaker whereas ucarp should be sufficient for your needs. I haven't configured ucarp yet, but a friend told me about it and it sounds really cool solution for smaller HA solutions.
For the services, some bind to addresses and need to be restartet after the IP alias has been set, otherwise the alias won't pick up the service. TCP connections normally get dropped after a failover...

w00d

[depontius]

I've been having a tough time getting my head around this, but I believe I'm getting there. Fortunately whenever I've coded firewalls, I've tended to code to the interface rather than to the IP. So I should be in good shape. Though when I adopt this framework I'm going to start coding to IPs for my services, I guess.

I guess the way to look at this is that, absent policy - like NAT, routing is really a layer-2 function. The router receives a packet on some interface, peeks into its header to see where it's headed, decrements its TTL, and then passes it out on the interface that leads to the next hop, headed to the MAC for that next hop. That last bit is important, because nothing about passing it on to the next hop rewrites the header, other than the TTL. There's nothing added to the packet about the router's identity, unless it's doing NAT. There's also nothing added to the packet about the next hop's identity, either. That also means that the return packet can take a completely different path than the outbound packet, and all will be well.

Nothing revolutionary there, really. It's just some plumbing aspects we normally don't think about, and take for granted. I never really thought about layer 2, ARP, and the relations between layer 2 and layer 3 until years ago, when a kernel upgrade flipped the eth0 and eth1 assignments to my network cards. THAT was a new learning curve.

Thanks for the assist.

[depontius]

Does "ucarp" automagically do the ARP flood necessary for HA failover? The web documentation was a bit sparse on the details, and the online man page, while better, doesn't say, either.

I also see a "-M" flag, saying to use broadcast instead of multicast. From what I understand, multicast is not in any sort of general usage. I guess since this is direct-connect for my use, it doesn't matter. But I'm surprised that none of the examples I see use the "-M" flag, since I've seldom seen information on setting up multicast. Seems a little odd that it would "just work (TM)", but I guess in this restricted case it's possible?

[AngelKnight]

Does "ucarp" automagically do the ARP flood necessary for HA failover? The web documentation was a bit sparse on the details, and the online man page, while better, doesn't say, either.

Guessing that you're looking up www.ucarp.org's uCARP implementation... According to the README, yes, ucarp specifically gratuitous-ARPs on a failover.

I also see a "-M" flag, saying to use broadcast instead of multicast. From what I understand, multicast is not in any sort of general usage. I guess since this is direct-connect for my use, it doesn't matter. But I'm surprised that none of the examples I see use the "-M" flag, since I've seldom seen information on setting up multicast. Seems a little odd that it would "just work (TM)", but I guess in this restricted case it's possible?

You're correct, your setup is direct-connect so the fact that broadcasts don't traverse routers and multicast only does under certain circumstances is not relevant for your use case.

[AngelKnight]

aliased IP, which will be eth1 on one of the servers. This looks like it might all work, because it's all traveling over the same ethernet cable, no matter what the IP is. But it doesn't feel right. It feels like I might have packets from the LAN coming out on eth1, from my server's native IP, going to the Netgear and out to Comcast. The return packet comes from Comcast and the Netgear sends it back to the aliased IP on my server, same device.

I guess when routing packets, the IPs of the router don't really matter, as long as you got there and as long as it knows the next hop?

Comments?

Only comment of note is that your guess above is one of the fundamental pieces of knowledge everyone has to get to about IP packets before good guesses about how a packet flows can be made.

For everyone else, a summary:

• With normal IP packet forwarding, the destination IP address is the root of the forwarding decision. The forwarding decision is the choice of how the given packet will forward. Note: the forwarding decision determines how the packet will forward, not *if* it should forward.
  
  Nothing else matters.
  
  Nothing else.
  
  Really.
  
  Your instincts might be yelling at you as you read this. Ignore them, they're wrong.
• When any other information is used as the start of the decision on how to forward, this is known as "policy routing". Examples: source IP address, ingress interface, payload protocol, etc.

[depontius]

Guessing that you're looking up www.ucarp.org's uCARP implementation... According to the README, yes, ucarp specifically gratuitous-ARPs on a failover.

I looked at the link, and it appears that ucarp only furnishes a hook for me to generate the ARP flood. I know I've seen this information before, but I can't find it now. I also know that I've got to send an ARP flood to all clients, so they'll start shooting packets to the new MAC. I also know that ARP flooding is generally frowned on, but this is my own LAN and this is a legitimate reason.

I guess I'm finding links on "gratuitous ARP" now, and can puzzle something out. On the other hand, if you have something that works now with in-portage packages, I'd be curious to see.

[AngelKnight]

Check the source: src/garp.c

[depontius]

I see now that it has the gratuitous arp code built-in, and see more about it in the README. The documentation talked about diddling the arp cache in the up/down scripts, but I guess that's the local arp cache, not the clients' arp caches.