Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Help compiling system with -flto
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Christian99
Veteran
Veteran


Joined: 28 May 2009
Posts: 1668

PostPosted: Sat Jun 14, 2014 11:28 pm    Post subject: Help compiling system with -flto Reply with quote

Hi, I am using lto quite a while, and also had some packages failing to build or showing bad runtime behaviour.
I recently found a blog post here, where he describes what all needs to be done. In the beginning, I only had -flto in my c(xx)flags, not in ldflags. After reading this blog, i added "-flto=2 -fuse-linker-plugin" to c/cxx/ldflags and also did otherthings described there: linking /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.so.0.0.0 to /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins.
and also applying a patch to binutils (in the blog, he uses binutils-9999, i instead applied the linked patch to 2.24 version).

but still many packages fail, with gcc-4.9 even more than with 4.8. Most errors are undefined references. But not the same with every packages, some packages complain about a few hundred missing symbols others just report a couple.
And also packages fail, which are reported in this blog to work.

What I want to ask, does anybody know/has an idea, what i am missing? I don't think that it depends on the system if a package fails with lto, so it must be some error on my side. Hope somebody can bring a little light into this.
Back to top
View user's profile Send private message
mackal
Tux's lil' helper
Tux's lil' helper


Joined: 04 Aug 2011
Posts: 87

PostPosted: Sun Jun 15, 2014 4:39 am    Post subject: Reply with quote

If a package fails to build with LTO enabled, you need to build it without LTO enabled.

http://realnc.blogspot.com/2012/06/building-gentoo-linux-with-gcc-47-and.html That should help you out.
Back to top
View user's profile Send private message
Christian99
Veteran
Veteran


Joined: 28 May 2009
Posts: 1668

PostPosted: Sun Jun 15, 2014 11:05 am    Post subject: Reply with quote

yeah, I'm totally aware of that. It's just, that I have more packages failing than other people. And I wanted to know the cause for that.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Jun 15, 2014 6:34 pm    Post subject: Reply with quote

gcc-4.9 fails with more package, because -fno-fat-lto-objects is the default, i.e. anything which uses libraries and for some reason or another does not honour lto in some linking step will fail. Often, it helps to pass -ffat-lto-objects, but it may mean that just the library contains redundant information and flto was not really used...
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Mon Jun 16, 2014 3:47 pm    Post subject: Reply with quote

You should be really careful with -lto imo; it changes the semantic of what the compiler does, albeit in line with the Standard. Trouble is, the newer semantic has many deeper implications around type-aliasing, that weren't around pre-C99, and gcc does some nutty stuff sometimes around type-aliasing. With -lto that gets exposed an awful lot more.

At work it's not even an option til the next major series of gcc, when it'll be reviewed again. We just don't have the time to explore every option while it's still being changed so rapidly.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Jun 16, 2014 4:07 pm    Post subject: Reply with quote

steveL wrote:
You should be really careful with -lto imo; it changes the semantic of what the compiler does

Unless there is a bug, it should not change any semantic (except for possibly triggering some bugs, of course). But maybe this is what you meant:
Quote:
With -lto that gets exposed an awful lot more.

One reason more to use -flto on everything related with your own code: you get a better testing of your code.
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 137
Location: Berlin

PostPosted: Tue Jun 17, 2014 12:33 pm    Post subject: Re: Help compiling system with -flto Reply with quote

Christian99 wrote:
Hi, I am using lto quite a while, and also had some packages failing to build or showing bad runtime behaviour.
I recently found a blog post here, where he describes what all needs to be done. In the beginning, I only had -flto in my c(xx)flags, not in ldflags. After reading this blog, i added "-flto=2 -fuse-linker-plugin" to c/cxx/ldflags and also did otherthings described there: linking /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.so.0.0.0 to /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins.
and also applying a patch to binutils (in the blog, he uses binutils-9999, i instead applied the linked patch to 2.24 version).

but still many packages fail, with gcc-4.9 even more than with 4.8. Most errors are undefined references. But not the same with every packages, some packages complain about a few hundred missing symbols others just report a couple.
And also packages fail, which are reported in this blog to work.

What I want to ask, does anybody know/has an idea, what i am missing? I don't think that it depends on the system if a package fails with lto, so it must be some error on my side. Hope somebody can bring a little light into this.


Make sure you have got the basics right. Try the example from https://gcc.gnu.org/wiki/LinkTimeOptimizationFAQ#ar.2C_nm_and_ranlib .
This should work without any issue.

For some ebuilds it helps if you export CC and CXX in your /etc/make.conf (instead of LDFLAGS and CFLAGS), e.g.:
Code:

CC="gcc -flto=4 -march=amdfam10 -O2"
CXX="g++ -flto=4 -march=amdfam10 -O2"

With this you should get a very low one digit failure rate.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Tue Jun 17, 2014 6:15 pm    Post subject: Reply with quote

mv wrote:
steveL wrote:
You should be really careful with -lto imo; it changes the semantic of what the compiler does

Unless there is a bug, it should not change any semantic (except for possibly triggering some bugs, of course). But maybe this is what you meant:
Quote:
With -lto that gets exposed an awful lot more.

OFC it is: what I said ;), with the bit you missed out:
Quote:
the newer semantic has many deeper implications around type-aliasing, that weren't around pre-C99, and gcc does some nutty stuff sometimes around type-aliasing.

It does change the semantic, and weasel words about how that's not a change, and those are just bugs, which I've heard before, are in fact incorrect (see below.) I'm sure you don't really believe it doesn't change anything, or you wouldn't use it.
Quote:
One reason more to use -flto on everything related with your own code: you get a better testing of your code.

Not in our shop; my boss is pretty clear on the matter, and I have to concur: gcc quite often has experimental features that take a while to bed down. Once it has, we'll deal with the settled-down variant.

That's not to say we don't deal with the new type-aliasing implications of C99; we talk about them far too often for my liking. However those are restricted to the translation-unit according to the Standard; unless you use -lto. And gcc will do weird stuff (reordering for one) if it decides two pointers can't possibly point to the same thing, even when you've just statically derived one from the other. We can cope with the latter within a TU by using gcc-specific code, but we don't have time to lose Standard C; at least not until gcc settles down a bit, and we can work it through with a stabler variant.

You'll know when that happens as there won't have been any new flags in the area for a year, and everyone will be talking about some other new-fangled option. ;-)

As for testing of our code, as said this isn't Standard C. Most of the newer changes in the Standard are to stay in line with C++, and so is much of the motivation for -lto. C++ really does need this kind of thing, since it's a pig to optimise (and in many cases can't be optimised) without a whole-program approach.

Many of these make C into a much higher-level language than it was before. There's a limit to how far we want to go with that, since we're not using it as a VHLL, but as a portable assembler. The Standard allows us to do that reasonably: -lto makes it a bit of a crap-fest, and like I said, we're not doing C++ here.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Jun 17, 2014 6:29 pm    Post subject: Reply with quote

steveL wrote:
I'm sure you don't really believe it doesn't change anything, or you wouldn't use it.

Quite the opposite: I wouldn't use it if it would really change the semantic.
Quote:
We can cope with the latter within a TU by using gcc-specific code, but we don't have time to lose Standard C

Do I get this right: You do know that you do incorrect things according to standard but you rely on some accidental misbehavior of gcc?
I would not call this "changing the semantics" but "fixing bugs".
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Jun 19, 2014 4:21 am    Post subject: Reply with quote

mv wrote:
steveL wrote:
I'm sure you don't really believe it doesn't change anything, or you wouldn't use it.

Quite the opposite: I wouldn't use it if it would really change the semantic.

That's not the point I made there, so I feel you're putting words in my mouth: I said you must believe it makes a difference, or you wouldn't use it.

And ofc with C++ there is the fact that the language effectively requires a whole-program approach.
Quote:
Quote:
We can cope with the latter within a TU by using gcc-specific code, but we don't have time to lose Standard C

Do I get this right: You do know that you do incorrect things according to standard but you rely on some accidental misbehavior of gcc?

No you get it wrong: gcc changes the semantic across the whole program, and a guarantee that is in the Standard (about type-compatibility of structs and struct pointers) no longer applies, as stated.

How you can possibly read "don't have time to lose Standard C" and infer that I'm doing incorrect things according to the standard, and that I'm reliant on gcc misbehaviour is truly beyond me.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Thu Jun 19, 2014 8:35 am    Post subject: Reply with quote

Sorry, I am not a native English speaker, and it seems for that reason there were some misunderstandings.
steveL wrote:
and a guarantee that is in the Standard (about type-compatibility of structs and struct pointers) no longer applies, as stated.

This seems to be the non-linguistic part of the misunderstanding: I am not aware that -flto violates the C standard. It is not clear to me to which assertion (of the "standard") you refer. Can you give a simple example/reference? (If this is rather cumbersome, let us stop discussion here.)
Back to top
View user's profile Send private message
Christian99
Veteran
Veteran


Joined: 28 May 2009
Posts: 1668

PostPosted: Tue Jun 24, 2014 10:25 am    Post subject: Reply with quote

thank you everybody for the input. especially trippels' link helped, because when I tested the example there, it turned out, that my "ar" is the fault. When using "gcc-ar", the example is fine. with plain "ar" it fails like stated there. So I set
Code:
AR="gcc-ar"
NM=gcc-nm
RANLIB=gcc-ranlib
in make.conf and now many more packages compile.
The world recompile isn't finished yet and I haven't checked all logs, but I found that at least ocaml isn't honoring the AR Var and is using plain "ar" instead of "gcc-ar". Perl is also failing with undefined reference errors, maybe it's the same problem, but I couldn't find a direct use of a plain ar/nm/ranlib in the log. But I guess it's the same problem.

So, what I want to know now:
1) Is it expected to work with plain "ar/nm/ranlib" or not? And if yes, what am I doing wrong, that it doesn't work as expected?

2) Is it ok to change ar/nm/ranlib via make.conf or could this lead to more problems? (emerge seems to be mostly fine with it, but since I don't really know what those programs do, I better ask)
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 137
Location: Berlin

PostPosted: Tue Jun 24, 2014 1:56 pm    Post subject: Reply with quote

Christian99 wrote:
thank you everybody for the input. especially trippels' link helped, because when I tested the example there, it turned out, that my "ar" is the fault. When using "gcc-ar", the example is fine. with plain "ar" it fails like stated there. So I set
Code:
AR="gcc-ar"
NM=gcc-nm
RANLIB=gcc-ranlib
in make.conf and now many more packages compile.
The world recompile isn't finished yet and I haven't checked all logs, but I found that at least ocaml isn't honoring the AR Var and is using plain "ar" instead of "gcc-ar". Perl is also failing with undefined reference errors, maybe it's the same problem, but I couldn't find a direct use of a plain ar/nm/ranlib in the log. But I guess it's the same problem.

So, what I want to know now:
1) Is it expected to work with plain "ar/nm/ranlib" or not? And if yes, what am I doing wrong, that it doesn't work as expected?

2) Is it ok to change ar/nm/ranlib via make.conf or could this lead to more problems? (emerge seems to be mostly fine with it, but since I don't really know what those programs do, I better ask)


1) The next version of binutils will allow you to use ar/nm/ranlib directly by placing gcc's liblto_plugin.so.0.0.0 in /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins.

2) As you've already found out, there are ebuilds that just ignore the exports.

So I would recommend using binutils-9999 for now.
Back to top
View user's profile Send private message
Christian99
Veteran
Veteran


Joined: 28 May 2009
Posts: 1668

PostPosted: Tue Jun 24, 2014 6:17 pm    Post subject: Reply with quote

Ok, I applied the patch mentioned in the blog from my first post to my binutils. I thought, this would be enough. But anyway, I just tried to install binutils-9999, but they fail, and guess what: it's an undefined reference error :) (oh and yes, I disabled lto for the package).
Maybe I just got a bad binutils revision? This is the error:
Code:
x86_64-pc-linux-gnu-g++ -W -Wall    -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -frandom-seed=ld-new -O2 -pipe -march=native -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu  -static-libstdc++ -static-libgcc -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu -O2 -pipe -march=native -o ld-new main.o i386.o x86_64.o sparc.o powerpc.o arm.o arm-reloc-property.o tilegx.o mips.o libgold.a ../libiberty/libiberty.a   -lpthread -ldl -ldl -lz
mips.o: In function `(anonymous namespace)::Target_mips<32, true>::do_finalize_sections(gold::Layout*, gold::Input_objects const*, gold::Symbol_table*)':
mips.cc:(.text+0x285f8): undefined reference to `gold::Symbol::should_add_dynsym_entry(gold::Symbol_table*) const'


the "undefined reference" is repeated several times. Do you know what I could do, or do you know a revision taht works?
Back to top
View user's profile Send private message
Christian99
Veteran
Veteran


Joined: 28 May 2009
Posts: 1668

PostPosted: Tue Jun 24, 2014 6:50 pm    Post subject: Reply with quote

OK, forget it, going back just one commit builds fine (even with lto enabled :) ). But ar still doesn't seem to pick up the plugin automatically. I placed it via symlink in /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins
Code:
ll /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins
insgesamt 8,0K
lrwxrwxrwx 1 root root 29 23. Jun 14:22 LLVMgold.so -> ../../../../lib64/LLVMgold.so
lrwxrwxrwx 1 root root 65 24. Jun 00:02 liblto_plugin.so.0.0.0 -> /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.so.0.0.0

if I try the example from your link it still fails. but if i specify the plugin manually ( like this:
Code:
ar cr --plugin /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.so.0.0.0 a.a a.o
), it works. So it seems that I'm still missing one point to make ar use the plugin automatically....
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 137
Location: Berlin

PostPosted: Wed Jun 25, 2014 7:35 am    Post subject: Reply with quote

Christian99 wrote:
OK, forget it, going back just one commit builds fine (even with lto enabled :) ). But ar still doesn't seem to pick up the plugin automatically. I placed it via symlink in /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins
Code:
ll /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins
insgesamt 8,0K
lrwxrwxrwx 1 root root 29 23. Jun 14:22 LLVMgold.so -> ../../../../lib64/LLVMgold.so
lrwxrwxrwx 1 root root 65 24. Jun 00:02 liblto_plugin.so.0.0.0 -> /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.0/liblto_plugin.so.0.0.0



Ah yes. I forgot to mention that you must only place one plugin in this directory.

I use two simple scripts to switch between LLVM and gcc LTO plugin:
Code:

markus@x4 ~ % cat =plugin_gcc
sudo rm /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/*
sudo ln -s /usr/libexec/gcc/x86_64-pc-linux-gnu/4.9.1/liblto_plugin.so.0.0.0 /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins
markus@x4 ~ % cat =plugin_clang
sudo rm /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/*
sudo ln -s /usr/local/lib/LLVMgold.so /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins
Back to top
View user's profile Send private message
Christian99
Veteran
Veteran


Joined: 28 May 2009
Posts: 1668

PostPosted: Wed Jun 25, 2014 11:26 am    Post subject: Reply with quote

Thx, this, indeed was the problem. the example now works, and now i'll try and see how emerging goes.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sat Jun 28, 2014 8:12 pm    Post subject: Reply with quote

mv wrote:
Sorry, I am not a native English speaker, and it seems for that reason there were some misunderstandings.

OK, fair enough. Turns out I got my versions of the Standard mixed up, though that's not what I reacted to, but the conception that I'm deliberately trying to go against the standard and use gcc-specific code instead. As you said though, a misunderstanding.

Points of fact and information are much less emotive, and much more interesting, so thanks for bringing it back to that.
Quote:
steveL wrote:
and a guarantee that is in the Standard (about type-compatibility of structs and struct pointers) no longer applies, as stated.

This seems to be the non-linguistic part of the misunderstanding: I am not aware that -flto violates the C standard. It is not clear to me to which assertion (of the "standard") you refer. Can you give a simple example/reference?

It doesn't: the C99 standard is what changed to mentioning the tag specifier, whereas I thought that was coming in C11. See 6.2.7.1 of C99, where it mentions:
C99 wrote:
Moreover, two structure, union, or enumerated types declared in separate translation units are compatible if their tags and members satisfy the following requirements: If one is declared with a tag, the other shall be declared with the same tag.
If both are complete types, then the following additional requirements apply: there shall be a one-to-one correspondence between their members such that each pair of corresponding members are declared with compatible types, and such that if one member of a corresponding pair is declared with a name, the other member is declared with the same name.
For two structures, corresponding members shall be declared in the same order. For two structures or unions, corresponding bit-fields shall have the same widths. For two enumerations, corresponding members shall have the same values.

Prior to C99, (ie up to C95, aka Amendment 1) the requirement is looser. The structs do not need to be named the same to be compatible:
Plauger & Brodie (1995) wrote:
Both are structure, union, or enumerated types declared in different translation units with the same member names. Structure members are declared in the same order. Structure and union members whose names match are declared with compatible types. Enumeration constants whose names match have the same values.

Note: I'm well aware we need to use unions.

The above explains how ABI guarantees are/were normally preserved by only allowing additional members at the end of structs (inside some reserved space at the end). Somewhere else, I know it's guaranteed that there's no padding at the beginning of a struct.

So it's more of a problem with the changing standard (and yes, I was wrong :-); it's turning C into a VHLL instead of a HLL, and that makes our life a little trickier. That continues in C11, from what I hear, though I haven't spent any time reading up on it yet, just browsed.

Code using that behaviour is not "buggy" though, as I often hear stated. It's just not run-of-the-mill application-level code.
And sure, we don't have to compile with C99, and we can (and already do) add attribute tags to get round odd behaviour.

It just feels like much of the importation from C++ isn't actually that beneficial to C. An example would be declarations after statements. I like being able to declare: for (int i = .. but there is no utility to adding C++ style declarations anywhere you like in the block. They're needed for C++, since it merges allocation and initialisation in most code, certainly in the preferred approach. But then C++ is more of a scripting language in that sense, ie a VHLL not a HLL. They simply clutter C code, and annoy some more advanced users that I know.

I'm not saying type-aliasing rules aren't useful: i know they are, which is why restrict is a useful keyword (if you know what you're doing.) It's a total pita having to declare union types so you can be assured the compiler isn't going to suddenly throw away your code, though. To my mind the need for an explicit_bzero function indicates a failure somewhere; and I don't mean at the programmer side. Over-zealous interpretation of type-aliasing to mean the compiler can ignore lexically-derived alias-information in the same scope (or function) is simply stupid, however convoluted the reasons for it in terms of information available to a pass, or whatever other excuse is given.

As is flagging UB in the compiler so that you can skip optimisation steps and emit nonsense, but not telling the programmer about it.
That simply has to be one of the dumbest ideas I've ever heard of, perhaps topped only by deliberately inducing inverted vertical-coupling. (don't ask me for a reference, it came up in ##c.)

I don't buy that the later phase doesn't know the types or aliasing info; if that's so it cannot be the phase deciding that two types aren't the same, and therefore some object is never really used, even though another pointer was derived from it (or it was derived from another), in the same scope, so let's just chuck away most of the code in this function. My point is if you're doing that in a lexical scope where you have all the information you possibly could[1], I don't want to go near what you're doing when you think you know the whole program, or more cogently library module.

Not till it's just another boring optimisation no one talks about any more, and the flags for it aren't changing with every new version.

Regards,
steveL.

[1] Yes, we can and do code round it, but ffs why should we have to add compiler-specifics (or a call to a less efficient function) to give it information that is right in front of it? Doesn't that mean that the info can in fact be used by a "later" pass? So just add it automatically, since you have it and it affects your code generation, and move on. You wouldn't be able to chuck the code when compiling to an earlier standard, so there really is no excuse, imnsho.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Jun 29, 2014 8:02 am    Post subject: Reply with quote

Thanks for the explanation. However, I still don't quite get it: The problem seems to be that optimizing the local unit might cancel some code which would perhaps not be canceled if the compilation unit were larger. So it seems to me quite the opposite that flto might prevent some undesired cancellation.

Independent of that, AFAIK, flto does not have any information about types: It only optimizes the intermediate code emitted during compilation, and since this intermediate code is used for all gcc languages (all c and c++ dialects, java, fortran, ...) I doubt that this phase can really have that influence which you claim. But I guess that you had examples where this problem really occurs, so I must admit that there is something which I obviously do not understand.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum