I use XFCE4, and I have -gnome -kde -qt4 -arts set. I dont think I use gtk at all either, as I think its only for gtk2 support, which I dont use.
Some apps seem to *need* qt4, which they'll install anyway AFAIK even if you have -qt4 set, so I wouldnt worry about it.
XFree doesnt have any of those as USE flags anyway.
You can see what USE flags apps do have by emerging gentoolkit, and doing etcat -u <package>. Its a great little tool.
Example: etcat -u xfree
