iptables creating malformed log messages

[katachi]

I am getting malformed syslog messages originating from iptables which are getting dumped into /var/log/messages, such as the following:

Code: Select all

Oct 11 12:54:13 server 4[71.993 palsALWD N U=t0SC13232924DT13274.3LN13 O=x0PE=x0TL6 D116D RT=C P=9 P=00 E=33419AK98998WNO=8 E=x0AKUG= P 000002ECFF6B <>217933]itbe LOE:I=OTeh R=7.0.0.3 S=2.2.49 E=48TS00 RC00 T=4I=61 FPOOTPST93DT591SQ34459 C=6401 IDW22RS00 C RP0OT(118A08B7FF6B <>217939]itbe LOE:I=OTeh R=7.0.0.3 S=2.2.49 E=48TS00 RC00 T=4I=61 FPOOTPST93DT591SQ34468 C=6401 IDW22RS00 C RP0OT(118A08B7FF6B <>217936]itbe LOE:I=OTeh R=7.0.0.3 S=2.2.49 E=48TS00 RC00 T=4I=61 FPOOTPST93DT591SQ34486 C=6401 IDW22RS00 C RP0OT(118A08B7FF6B <>217935]itbe LOE:I=OTeh R=7.0.0.3 S=2.2.49 E=48TS00 RC00 T=4I=62 FPOOTPST93DT591SQ34495 C=6401 IDW22RS00 C RP0OT(118A08B7FF6B <>217931]itbe LOE:I=OTeh R=7.0.0.3 S=2.2.49 E=48TS00 RC00 T=4I=62 FPOOTPST93DT591SQ34413 C=6401 IDW22RS00 C RP0OT(118A08B7FF6B <>217931]itbe LOE:I=OTeh R=7.0.0.3 S=2.2.49 E=48TS00 RC00 T=4I=62 FPOOTPST93DT591SQ34422 C=6401 IDW22RS00 C RP0OT(118A08B7FF6B <>217937]itbe LOE:I=OTeh R=7.0.0.3 S=2.2.49 E=48TS00 RC00 T=4I=62 FPOOTPST93DT591SQ34431 C=6401 IDW22RS00 C RP0OT(101080A0028EBC7FFFF869B)

This problem cropped up about two months ago, but I'm only now getting around to troubleshooting it. iptables is logging all network traffic. rsyslog is using a regex filter to send iptables log messages to a separate file. The malformed messages are obviously not parsed by the regex, and are thus sent to /var/log/messages. The length of the malformed message can be large, as above, or can be short (50-100 characters). The format of the malformed message is not consistent. Sometimes part of the iptables message will appear in various locations, but the rest will be malformed. The problem occurrence is also not consistent. About 95% of the iptables log messages are properly-formatted, get parsed correctly by rsyslog, and are put into the proper separate log file. The rest is the garbage similar to the above.

This machine is a Gentoo x86_64 VPS running on Xen. The kernel is a 2.6.35.4 custom-build kernel built by the VPS provider (problem occurs when using all kernels -- even the last-known-good one). rsyslog 5.8.5 and iptables 1.4.12.1 -- both most recent stable versions for the platform. I have tried using various kernel versions, replacing rsyslog with syslog-ng, and have cleared configs and reinstalled syslog daemons and iptables, but to no avail. These malformed log messages originate from only iptables. All other logging appears correctly.

Any ideas?

[katachi]

It's now dawning on me that the malformed text is iptables messages with only every other character being displayed.

For example:

palsALWD N U=t0

should be

iptables ALLOWED: IN= OUT=eth0

The malformed blob I posted above is actually multiple log messages clumped together.

The question is why is this happening, and why is the syslog header perfectly intact on the first message only? Anyone have any theories?