Thank you charles17! Didn't know that.
Avoid having too many settings there. This is from my https://wiki.gentoo.org/wiki/Dell_Latitude_E6430_%28CYRTRY1%29 :
CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j8"
CPU_FLAGS_X86="aes avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ...