Xen networking woes [SOLVED]

[KShots]

I've successfully set up a pfsense firewall under xen. The firewall routes properly, connects to the internet, and the hosts on my network route through it just fine... but not dom0. As far as dom0 is concerned, all IPv4 TCP traffic is failing to route outside of the local subnet. IPv6 TCP traffic appears to pass unimpeded, as does ICMP traffic. The firewall is not blocking (or even seeing) any traffic from dom0. My routing table looks right... I don't get it

[KShots]

Interesting development... I managed to get a packet capture from the firewall showing that traffic is indeed getting that far:

[Atom2]

Kshots,
if you take a closer look at the packet capture you will notice that many packets have an incorrect checksum - and that's the source of your problem.

I am also running pfsense under XEN and have had the same issue (starting from pfsense 2.2 back in January 2015). At that time I did quiet a bit of investigation and have also come up with a solution to rectify the issue. I'll explain what I found out then and als try to provide a reasoning why I think it works that way for you to understand what I think is going on:

1. It appears that any TCP traffic from a domU or the dom0 arriving at the pfSense domU through the XEN virtual netback/netfron vif interface xn0 displays an incorrect checksum. I also figured this through tcpdump within the pfSense domU listening on the (paravirtualized) xn0 interface.
   
2. Interestingly enough, ICMP and UDP (e.g. DNS resolution) traffic was not affected and has always worked in my case.
   
3. It appeared to me that the firewall system (probably the pf packet filter) silently dropped all TCP traffic with an incorrect checksum
   
4. The reason why those packets arrive without a proper checksum lies within the paravirtualized XEN vif interfaces: Those are using shared memory between dom0 and the domUs (respectively between different domUs). As this is considered to be safe (there's no transfer over a probably faulty wire) the checksum calculation functionality is advertised by the driver, but in essence is just a null-function.
   
5. This means that a normal domU / the dom0 does neither calculate the checksum when sending packets nor checks the checksum when receiving traffic through a paravirtualized vif interface. The pfSense system / the pf packet filter however seems to care about those seemingly incorrect checksum values and decides to silently drop packets without a proper checksum.
   
6. Given the above I came to the conclusion that changing anything at the pfSense domU level would not make any sense as there was no option to just ignore the faulty checksum for received packets and accept the packet despite that error. In particular (and contrary to some advice found elsewhere) enabling "Disable hardware checksum offload" under System/Advanced/Networking on the pfSense menu can stay off (i.e. remain with offloading hardware checksum calculation to the NIC). This offloading (in terms of XEN) is only relevant for sending traffic from the pfSense firewall to any domU / the dom0 through the paravirtualized vif xn0 and neither of these recipients cares about the checksum arriving on a paravirtualized vif interface.
   
7. The only issue remaining was to ensure that traffic arriving at the pfSense firewall through the xn0 interface had correct checksums. In the standard bridged setup this clearly could only be sorted in the dom0 backend vif connected to the frontend for the pfSense domU.
   
8. Once I arrived at that conclusion I simply disabled the tx offload function for the backend connected to the pfSense domU within the dom0 machine:

[KShots]

[KShots]

Ok, I've resolved the checksum issue as indicated by this new packet capture from pfsense:

[Atom2]

It seems I have missed you previous reply, but I am currently online, so maybe we can try to resolve this together. Could you please describe your setup. Mine is as follows:

I do have a HVM pfSense setup with a PV vif interface that is connected to the bridge and, through the dom0 via the bridged NIC, to the internal LAN. Furthermore there is a PCI network card passed through to the pfSense domU which connects to the WAN and is used to route the internet traffic. This means there's no issue with checksums on the internet connection from pfSense outwards as this goes through a real NIC interface which does the calculation. All traffic destined to the internet from the LAN (including dom0 and all other domUs) is routed through the pfSense domU.

How about your setup?

Atom2

[KShots]

[Atom2]

[KShots]

To clarify, here's my pfsense config:

[KShots]

[Atom2]

[Atom2]

[Atom2]

Before I forget: In case you use my script, you should revert the standard bridge script to its original state (i.e. delete any changes you have made). That approach also guarantees that you always use that latest script that ships with xen.

And you obviously need to change the script name in the pfSense configuration file to match my name.

[KShots]

[Atom2]

[KShots]

[KShots]

[Atom2]

[KShots]

[KShots]

[Atom2]

[Atom2]

[Atom2]

To be honest, I am more concerned about that PoD thing.

[KShots]

[Atom2]