LDFLAG --as-needed / Speeding up of GNOME or KDE

Message

infirit · Post by **infirit** » Wed Mar 30, 2005 9:53 pm

It is ok now to file bugs for packages failing with --as-needed. File the bug and provide all the ussual needed info (emerge --info and error message for example). And make it depend on bug# 129413

I was reading this article on osnews and wondered what one would gain from this and what the problems could when using it in gentoo?

How to set LDFLAGS
There are a couple of ways to set the LDFLAGS.

1: export LDFLAGS="-Wl,--as-needed"
2: Set it globally in /etc/make.conf by adding LDFLAGS="-Wl,--as-needed"
3: Setting them per package, see below section named: Per package {LD,C}FLAGS

It should be set as LDFLAGS="-Wl,--as-needed" not as the article says as CFLAG!!

The story

Today I thought about how I can make my very own application to only link against those libraries that it really requires and not those it get provided by PKG-CONFIG.

Problem-case:

When we use pkgconfig within our programs which of course makes sense, we also see that other libraries from other packages are being processed to our *_LIBS variables. So we end up in linking a lot of libraries to our application which we really do not need and which also causes (so I assume) a lot of overhead. After thinking for a while I looked some programs up by using:

readelf -d /usr/local/bin/gnome-terminal |grep NEEDED | wc -l

And saw that it was requiring over 52 libraries. I then recompiled gnome-terminal with

export CFLAGS = "-Os -s -Wl,--as-needed"

Pay attention to the --as-needed and the resulting binary required just 21 libraries afterwards instead of the 52 before because it only linked those really required to the final executable. I continued this with vte and saw that it required 29 libraries initially but after the --as-needed flag it went down to 7 libraries.

Now imagine this for the entire GNOME desktop or the entire KDE desktop and also package management systems which will reduce dependency tracking a lot. The problem with this method is that you surely don't keep track of API compatibility but a good package management system will certainly make sure that the packages packed belong to a certain desktop version.

How to filter the flags in ebuilds
There is a way to have portage filter this flag at compile time:

Add the following to the top of the ebuild below IUSE

Code: Select all

inherit flag-o-matic

and put the below in src_compile at the top.

Code: Select all

# This LDFLAG breaks "package name"
filter-ldflags "-Wl,--as-needed"
filter-cflags "-Wl,--as-needed"

New LDFLAG --enable-new_ldflags
There is another LDFLAG which is more safe then plain --as-needed. The flag is --enable-new_ldflags and devsk eplains it best

defsk wrote:when passed enable-new_ldflags, configure just checks if --as-needed works and if it does then it just assigns the variable LDFLAGS_AS_NEEDED="-Wl,--as-needed", which Makefiles have the liberty of using. Currently most kde Makefile's do. If a component finds that it doesn't work for them, they don't put it on the specific link line. Its manual in that sense.

one more thing is configure arg is --enable-new_ldflags and NOT --enable-new-ldflags as mentioned by the ./configure --help.

I could not work out if this works or not, tested on konsole

LDFLAG Central
Some interesting ldflags in this post

Per package {LD,C}FLAGS
There is a way to have a per package ldflag. It involves copying an bashrc file into /etc/portage and creating a file package.ldflags also in /etc/portage. Get the script here and full explanation here. Thanks to thebigslide for creating the original bashrc and SoTired for adding the LDFLAGS support

i92guboj · Post by **i92guboj** » Wed Mar 30, 2005 10:08 pm

Mmmh! This would be awesome, but I dont think that this is really a so easy thing.
The differences in numbers are just too large.

Flameeyes · Post by **Flameeyes** » Thu Mar 31, 2005 9:48 am

Well, actually KDE already uses --as-needed by default, if not disabled (for example on *BSD where it doesn't work too well).

This approach generally works well.. but still there ae cases in which packages got broken by --as-needed.. see for example x11-libs/gtk+-2.6.4-r1 which fails under amd64 for some conflicts between pic, amd64 relocations and --as-needed.

lanius · Post by **lanius** » Thu Mar 31, 2005 11:31 am

I think it only breaks if you set CFLAGS="-Wl,--as-needed", but the correct way to do it is to set LDFLAGS="-Wl,--as-needed"

fuoco · Post by **fuoco** » Thu Mar 31, 2005 3:02 pm

anyone tried that on his gentoo box, and can share his experience?

nxsty · Post by **nxsty** » Thu Mar 31, 2005 3:21 pm

fuoco wrote:anyone tried that on his gentoo box, and can share his experience?

No, but I will try it ASAP.

Rapsey · Post by **Rapsey** » Thu Mar 31, 2005 6:01 pm

I think there is a speed improvement...

badgers · Post by **badgers** » Thu Mar 31, 2005 6:28 pm

is this supposed to be a speed improvment in the time to compile or will it result in a faster binary when done?

Rapsey · Post by **Rapsey** » Thu Mar 31, 2005 6:41 pm

reduced startup time.
edit: some more playing arround and it does work great. It cut the number of libraries loaded with gedit by half, startup time is much faster. Same thing for gnome-terminal.

aethyr · Post by **aethyr** » Thu Mar 31, 2005 7:38 pm

Judging by this post, I'd be careful of what you use it with. I wouldn't put it in make.conf and rebuild my system.

http://thread.gmane.org/gmane.linux.red ... ters/24698

badgers · Post by **badgers** » Thu Mar 31, 2005 8:22 pm

so if I wanted to recompile just one app with this option does it get added to the cflags variable or the ldflags variable? I have never changed the ldflags in make.conf

hielvc · Post by **hielvc** » Thu Mar 31, 2005 10:09 pm

Dumb responces . Removed

rhill · Post by **rhill** » Fri Apr 01, 2005 12:23 am

there are reasons that --as-needed isn't used by default. it breaks many things in vague and creative ways.

http://sources.redhat.com/ml/binutils/2 ... 00103.html

badgers · Post by **badgers** » Fri Apr 01, 2005 2:46 am

if this is to speed up the launch of a program, how does this differ from pre-linking?

foosh · Post by **foosh** » Fri Apr 01, 2005 11:58 am

Maybe the title of this thread could have been "compiling '--as-needed' / speeding up Gnome". such a title would almost immediately tell me we're doing specific thing X in hopes of specific result Y

"story on OSnews.com" tells me absolutely nothing worthwhile. sorry to be so anal...but I feel it's the duty of a poster to make a concise, useful thread title to aid in the de-obfuscation of these forums, allowing people to search them more quickly--there is a TON of information to go through.
How many times have you seen threads with completely vague or ambiguous titles like "problem with X.org" or "stupid question..." or "help me please" ? those do a less than ideal job of directing people to a.) problems they could provide answers for, or b.) problems they need help with.
</soapbox> (infirit, no offense to you)

~~~
flameeyes said KDE compiles with --as-needed by default. I'm just curious why Gnome, for instance, couldn't?

Rapsey · Post by **Rapsey** » Fri Apr 01, 2005 12:11 pm

badgers wrote:if this is to speed up the launch of a program, how does this differ from pre-linking?

With prelinking the program still loads the same amount of libraries when you run it, but it loads them faster. This eliminates loading of unnecessary libraries.
But it can cause problems for some programs apparently. I used it on programs that I find realy annoyingly slow to load: gedit, gnome-console, gaim, totem, rhythmbox. None of them have any problems.
So I guess we should not put it into make.conf. Can someone tell me how to set it, so that portage will only use it on specific programs?

haven · Post by **haven** » Fri Apr 01, 2005 1:42 pm

So I guess we should not put it into make.conf. Can someone tell me how to set it, so that portage will only use it on specific programs?

A simple wrapper script in bash might work - not nearly as automated as you would like I'm sure but its an option.

Code: Select all

#!/bin/bash

source /etc/make.conf

CFLAGS="${CFLAGS} -Wl,--as-needed" /usr/bin/emerge $1

Call it what you like and run it instead of emerge when you want to run "-Wl,--as-needed". I suspect you were looking at some sort of ebuild fix though

Post by **suka** » Fri Apr 01, 2005 1:45 pm

Rapsey wrote: But it can cause problems for some programs apparently. I used it on programs that I find realy annoyingly slow to load: gedit, gnome-console, gaim, totem, rhythmbox. None of them have any problems.
So I guess we should not put it into make.conf. Can someone tell me how to set it, so that portage will only use it on specific programs?

I got into problems with quite a few programs: Totem, Sound-Juicer, basically all the GStreamer stuff, also gnome-system-tools and evince. On the other hand it seems to really speed up startup of some other stuff...

Rapsey · Post by **Rapsey** » Fri Apr 01, 2005 2:52 pm

I use xine library for totem and not gstreamer. Works fine.

infirit · Post by **infirit** » Fri Apr 01, 2005 3:54 pm

foosh wrote:Maybe the title of this thread could have been "compiling '--as-needed' / speeding up Gnome". such a title would almost immediately tell me we're doing specific thing X in hopes of specific result Y
</soapbox> (infirit, no offense to you)

None taken

i have change the topic title to LDFLAGS --as-needed / speeding up Gnome"

I am going to take some measurements this wekeend for a couple of large program (large = bigger than gnome-terminal). I'll report back with some results. Any suggestions?

Rapsey · Post by **Rapsey** » Fri Apr 01, 2005 5:08 pm

Well anjuta is bigger, runs fine if you compile it with that flag. Also starts up quite a bit fater.

Maedhros · Post by **Maedhros** » Fri Apr 01, 2005 5:56 pm

I tried it, and things stopped being able to find gtk+ to compile against. I'm going to wait a while before trying again, I think.

fridrik · Post by **fridrik** » Fri Apr 01, 2005 10:07 pm

i tried and recompiled terminal and panel
can't "see" any improvements
by reading this thread

aethyr wrote:http://thread.gmane.org/gmane.linux.red ... ters/24698

i think i won't use it anymore

jdgill0 · Post by **jdgill0** » Fri Apr 01, 2005 10:32 pm

I would like to point out that --as-needed is not only good for GNOME. Although it was said that KDE already uses --as-needed, I saw a difference in the number of libs linked against Konsole as shown below.

I have also built gimp with --as-needed and it seems to be working fine. I rebuilt KDE's konsole (I am using the splitted ebuilds). Using -Wl,--as-needed as a compiler flag did NOT work for Konsole, I had to use the following as was suggested above:

Code: Select all

LDFLAGS="$LDFLAGS -Wl,--as-needed"

Using the below code to compare the number of libs linked, I got the following results for Konsole:
Konsole WITHOUT --as-needed links to 63 libs.
Konsole WITH --as-needed links to 26 libs.

Code: Select all

readelf -d `which konsole` | wc -l

I haven't compared the number of libs linked for gimp.

haven · Post by **haven** » Sat Apr 02, 2005 9:02 am

Is it worth compiling a list of packages that benefit from this vs packages that break with this ? And where would be the best place to do this ?

A simple script to exploit this list could be (excuse the simplicity as I'm not a coder):

Code: Select all

#!/bin/bash

source /etc/make.conf
match=$(grep "^$1$" "/etc/portage/package_as_needed" )

if [[ "$match" = "$1" ]]
   then
      echo "emerging using -Wl,--as-needed"
      LDFLAGS="${LDFLAGS} -Wl,-O1 -Wl,--as-needed" /usr/bin/emerge $1
   else
       echo "emerging using defaults"
       /usr/bin/emerge $1
    fi
exit 0

This just uses a flat file "/etc/portage/package_as_needed" to check for packages that work when compiled with "-Wl,--as-needed" and chooses the appropriate method.

From what I read, those applications that have a plugin architecture are doomed to fail with this optimisation (i.e. apache) whilst many desktop apps that are fairly self contained can be optimised quite heavily.

NB the other LDFLAG optimisations came from http://forums.gentoo.org/viewtopic.php?t=67777