going back a while there were some patch sets that offered a serious gain in responciveness and speed.
When lovechild was providing love-sources (c. 2.6.4 IIRC) they offered great improvements. That's history now but I just insert that in his honour , he descerves a medal.
around 2.6.11 I had the fastest and most responsive system ever. I could build KDE , ftp a large file and listen to mp3 without even a glitch in the sound. Sadly I have never been able to reproduce this since (close, but always some glitches in sound).
Since that time gentoo-sources seem to have picked up on some of this pinoneering work and the std gentoo kernel is a lot faster than it used to be, negating some of the advantages of custom kernels.
Like iphitus says , features is probably the main reason now.
OP wrote:I can live without that Reiser garbage
Well I'm sure Reiser4 can live without you too , so no harm done. I've been using it for about 3yrs for the majority of my partitions and swear by it. YMMV.
R4 is my main reason to use a patchset because I cant be arsed to patch it myself, plus there are generally a couple of features like ck patches that I find useful.
If ck included R4 I would have been using it for a long time.
Currently on no-sources but since Conrad has had to put his family above messing with patchsets it looks like time for a new choice. (Many thanks to Conrad for his excellent work in the past good health to his mom).
Finally most recent instability and breakages seem to come the state of the kernel releases more than the patchsets. There has been some clumbsy hacking in inode structures and fundemental changes in pata/sata drivers so anything beyond early 2.6.18* and you are likely to have several things not working , depending on what you need and use.
vanilla is stable
no funky patches
problems can be solved by the experts. You are not dependent on some hobbyist.
No need to be disparaging about maintainers of patchsets. Looking at the recent flux in the inode-diet fiacso, stable is not the first adjective that springs to mind for vanilla. Most of your "hobbyists" are pretty competant and you'll have a good chance of getting a quick fix if you highlight an issue.
Just look at the support threads to see likely issues and to judge the competance of the maintainers.

Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86