So this really is a broader problem then I thought.
Any long compilation fails with "internal compiler error". Gcc fails consistently but in random locations.
Ran a memtest but no errors were found.
Took down optimization level, but that didn't help.
Any ideas?