spamdyke on hardened - RLIMIT_AS

[NeddySeagoon]

Team,

I am trying to run spamdyke in front of qmail on a gentoo-hardened system. Its a KVM but that probably doesn't matter.

dmesg tells me

Code: Select all

grsec: From 212.23.1.5: denied resource overstep by requesting 16228352 for RLIMIT_AS against limit 16000000 for /usr/bin/spamdyke[spamdyke:1626] 

and /var/log/qmail/qmail-smtpd/current tells

Code: Select all

@40000000553d65a02edc836c /usr/bin/spamdyke: error while loading shared libraries: libz.so.1: failed to map segment from shared object.

So it looks like spamdyke needs more that a 16Mb address space.
spamdyke is made up of

Code: Select all

# lddtree /usr/bin/spamdyke
spamdyke => /usr/bin/spamdyke (interpreter => /lib64/ld-linux-x86-64.so.2)
    libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0
    libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0
        libdl.so.2 => /lib64/libdl.so.2
        libz.so.1 => /lib64/libz.so.1
    libc.so.6 => /lib64/libc.so.6

RLIMIT_AS against limit 16000000 says its allowed 16Mb of address space.
Having tried to change it in /etc/security/limits.conf and with ulimit -v in a wrapper script, nothing will change the RLIMIT_AS - not even downwards.

Where is RLIMIT_AS set and how can i change it?

[boozo]

Sir Neddy,

just an idea according to this : it seem that you should search directly from the spamdyke source-code

nb. I precise that I've never had to do something with any "AS" setting (noob inside) but there are an example to define this $vars in the RSBAC handbook ( §Ressources restrinctions) too ...

[NeddySeagoon]

boozo,

Thank you for the pointer. I'll look later this evening if I don't run out of time.

[NeddySeagoon]

qmail runs under softlimit=16000000 and it seems as if its not enough.
That's not all of the issue. I havu to soften the hardening a little by turning off address space randomisation. under the PAX settings in the kernel too.