Experiment: Lightning-fast up your KDE on a 64 Bit Gentoo!

[Guenther Brunthaler]

Hi all,

While studying the Gentoo prelink guide, I noticed that it is clearly written with 32 bit processors in mind and does not take full advantage of a 64 bit processor running a 64 bit operating system (like Gentoo can be used).

So, if you are using KDE and have an AMD-64 or one of those new INTEL 64 bit processors and are using a 64 bit Gentoo, then read on. (If you are still using a 32 bit Gentoo running on a 64 bit processor, this article may convince you to switch to a pure 64 bit Gentoo.)

The cause why KDE is so goddamn slow on most machines is (besides machines having too little RAM) called "RELOCATIONS".

Relocations occur when an executable or shared library is compiled for the same memory addresses as another one.

When two (or more) such executables/libraries with conflicting address requirements have to be loaded, all of them except for the first have to be relocated. Relocation means:

• It is GODDAMN slow (can take up to several seconds in environments like KDE where many shared libraries are loaded)
• Cannot be shared between processes (or only to a much lesser extent). Which means relocations waste precious memory!

Both disadvantages together make KDE so very slow on most 32 bit machines. (At least slower than it were without those relocations.) And it also requires much more RAM than necessary.

The basic solution to this problem already exists: It's called PRELINKING.

The basic idea of prelinking: Assign a different address range to each shared library in the system, then there will never be an address conflict and thus relocations will never be necessary.

A good plan, but it cannot be executed on a 32 bit processor to its full extent: On 32 bit processors, at most 3 gigabytes of memory are usually available to a process.

And unfortunately, if you add the address space requirements of all shared libraries in your system together, you will typically need more than those 3 GB of addressing space in order to assign a unique address range to each library.

In other words: prelinking will not work on 32 bit systems - at least not to its full extent.

That's why prelinking seems not to gain anything (or at least not too much) when prelinking KDE under a 32 bit operating system.

But on 64 bit systems it can work to its full extent (if you are following the right guide...)!

A full, flat 64 bit address space has the enormous size of 16 exabytes. Even if part of that address space is reserved by the system, there will be plenty of room left in order to assign a different, unique address range to each and every shared library in the system.

Which means your KDE should be lightning fast then - compared to how fast it is now (if you did no prelinking or only that from the official prelinking guide).

What to do:

1. Follow the Gentoo prelink guide http://www.gentoo.org/doc/en/prelink-howto.xml except for the Code listing 3.1 in section "Prelink usage".

2. Instead of the command "prelink -amR" which is suggested in Code listing 3.1 of the guide, use the following command:

Code: Select all

# prelink -afR

This should do the magic!

And every time you have installed or updated a package which may contain shared libraries, be sure to run the following command after the emerge operation is complete:

Code: Select all

# prelink -aqR

This ensures the new/updated libraries will be prelinked as well.

(Hint: Why not making it a cron to be run daily or weekly? However, be sure to check in your cron script that it is not coincidentally run at the same time while an installation is in progress - it could interfere with the installation "merge" or "qmerge" phase of an ebuild.)

But what is the problem with the Gentoo prelink guide?

It is the "-m" switch. This switch is necessary on 32 bit systems for prelinking to work at all (unless you have a minimal installation where all libraries taken together actually fit into a 32 bit addressing space). But it also disables the raw power of prelinking when using a true 64 bit operating system, severely crippling it. So just away with the "-m", and the power is back!

Important notice: Why is this guide tagged as "experimental"? Because I have not tried to do it myself yet! The simple reason: I just haven't got a 64 bit processor...

So ... volunteers wanted!

Who is daring enough to risk the health of his life and his system to prove (or disprove) my fine little theory and to get NOTHING in return (except for some speed gain, perhaps)?

Comments welcome!

[irondog]

Well, that sounds good. I'm not convinced to use 64bit Linux yet, but this definitely is a good reason to do it in the future.

[Phenax]

Unfortunately, I don't use KDE. Although, I have a fully functional AMD64 system. I figured I'd follow the pre-link guide with your modified Section 3.1 command. It didn't mess up my system or anything, nothings noticeably slower or faster (No extensive testing, no 'real' proof).

It would be interesting for someone who uses KDE to try this, just to see if the increase is really 'that' great.

I use Openbox, and it obviously doesn't have a lot to prelink to, neither does almost anything in my system, so the results were as-expected. Not much of an increase.

[Archangel1]

A full, flat 64 bit address space has the enormous size of 16 exabytes. Even if part of that address space is reserved by the system, there will be plenty of room left in order to assign a different, unique address range to each and every shared library in the system.

That would be nice, but the address space of amd64 at present is only 2^40 bytes (about a terabyte). I think em64t is similar, no idea about ppc or anything else.
Might still be enough for what you've got in mind though.

Who is daring enough to risk the health of his life and his system to prove (or disprove) my fine little theory and to get NOTHING in return (except for some speed gain, perhaps)?

I might consider it when I get home tonight. We'll see

[bombcar]

I though that the usable external address space was 2^40, are you saying that it doesn't have 64 bits available for virtual addressees? That sounds stupid.

[Guenther Brunthaler]

Hi Archangel & Bombcar,

Thanks for your comments. I wasn't aware of that limitation.

So let's hope that there is at least a 64 bit addressing space, because otherwise the term "64 bit operating system" seems to be sneering at us...

But even in the worst case that only 2 ** 40 bytes are available, I hope that will be enough: My whole Gentoo installation has no more than 30 gigs - that's 35 bits. Including (non-video-) data! So there is still a factor of 32 left for safety, assuming the address space reqirements of all shared objects cannot be much larger than the sum of all files in the installation.

Unless .. huge amounts of BSS data are declared by the shared objects, like huge uninitialized static arrays.

But let's make a worst-case analysis: Assume every .so in the system was so mad to declare a 4 GB static array or somehow else wastes all its addressing space.

Code: Select all

ldconfig -p | wc -l

reports about 1600 shared libraries on my system (some of them may be symlinks, but we'll be conservative anyway). So let's assume a "full blown" system has no more than 3000 SOs installed.

If each SO required the full 4 gigs of addressing space, this calculates to a total address space requirement of 44 bits... WHICH IS BAD. We are short of addressing space by a factor of 16!

On the other hand, are 3000 SOs, each using 4 gb realistic? I think not.

So I conclude 40 bit will still be enough to prelink all SOs in the near future for all practical considerations.

[Guenther Brunthaler]

Seems I've been a bit too optimistic for a "full blown" system having only 3000 SOs... A quick strace of firefox revealed that this app alone loads 1'680 SOs!

A full

Code: Select all

find /usr /lib /bin /opt -type f -name '*.so*' | wc -l

reported even horrible 4'000+ SOs! Urrrgh. (I'm starting to wonder how all those SO's can actually fit on my disk... my installation is not that large. Bless ReiserFS for making lots of files a rather cheap thing!)

So I can only hope there is really more than 40 Bits virtual addressing space provided by the kernel!

[Phenax]

1603 here.

[richard_ablitt]

I've just tried it. KDE now loads much faster, and aMSN seems to open in about half the time that it was before. No problems so far... (:

[Guenther Brunthaler]

I've just tried it. KDE now loads much faster ... No problems so far... (:

Cool! It seems to work then.

Thanks very much for trying!

[pteppic]

I'd followed the guide previously (about 2 weeks ago) and did notice a nice speed increase while loading apps, to the point I use konqueror for web browsing now as it leaves firefox in the dust.
I allready have prelink running as a cron job, using the switches in the guide. To use your instructions should I prelink -ua first and re-run with -afR or just change the cron job and let it slowly propagate?

[Elfan]

Is there anything specific to KDE in your guide or were you just using that as an example?

[Phenax]

Is there anything specific to KDE in your guide or were you just using that as an example?

Nope, you'll just see best results using KDE because it has so many shared libraries.

[ChickensDontFly]

I'm gonna give this a try tomorrow. It's too late to mess with it tonight.

Guenther, this is the second post of yours I have come across today that will save me a lot of time.

Thanks for contributing and keep up the good posts!

[Master_Of_Disaster]

LDFLAGS="-Wl,-Bdirect" breaks this, right?

*edited* pic in USE is evil!

[Guenther Brunthaler]

I'd followed the guide previously (about 2 weeks ago)

Must have been another guide then, because my guide is just a couple of days old!

and did notice a nice speed increase while loading apps, to the point I use konqueror for web browsing now as it leaves firefox in the dust.

I wonder if they are sleeping at INTEL and AMD?

Everyone in their marketing departments seems to be searching for "Killer Apps" for their new 64 bit processors, but I haven't ever heard them talking about KDE, where the 64 bit have the most obvious positive impact.

Perhaps one should send them an e-mail?

I allready have prelink running as a cron job, using the switches in the guide. To use your instructions should I prelink -ua first and re-run with -afR or just change the cron job and let it slowly propagate?

I wouldn't use -u at all. Its for "undoing" prelinking, which won't help.

The -afR unconditionally prelinks all shared objects (shared libraries). I would only do this once, such as after upgrading the compiler and recompiling the entire system.

But in the cron job, I'd only use "prelink -aqR". This will only prelink libraries which have been updated, leaving the ones already prelinked alone. It uses the modification time of the SOs to check whether they have been updated.

If you don't want to trust the modification time of your SOs, use "prelink -aR" instread. This examines all SOs
whether they require prelinking, but prelinks only those which need prelinking. It takes longer than -aqR, but is still much faster than -afR.

In my opinion, -aqR should be enough for a cron job unless you have very serious timing problems with your system clock.[/b]

[zxy]

Here is the output of my system.

Code: Select all

localhost / # find /usr /lib /bin /opt -type f -name '*.so*' | wc -l
3666

I just prelinked the whole system. But I will have to restart, to see the true difference. I mostly use fluxbox, but konqueror, firefox, thunderbird and some other kde apps are on my useage list too, so It should help.

I wonder what would happen with oppenoffice if it could be compiled for amd64, and prelinked then.

[Guenther Brunthaler]

Guenther, this is the second post of yours I have come across today that will save me a lot of time.

Thanks! But the Gentoo project and its community has given me so much already, that I felt it was highest time to give some of it back.

Especially as I'm still a newbie. Not only to Gentoo, but also to Linux as a whole.

A couple of months ago, I still was one of those gameloader-OS users. I never really liked it, but I used it anyway because "you can't live without it". Which was what that pre-alpha-software selling company made most of us ****-users believe.

Well, I already had a Gentoo installed on one of my machines for playing around, but I'd never thought of really using it for serious work. You know, "because it's all so complicated, while under **** it's all so simple."

However, then came that glorious day when that company decided to stop their update support for users which refused to reveal their personal data to their registration wizard. (Thinking of it in retrospective, this decision was the best they have ever done to me.)

And although that OS was already installed when I purchased my computer, I'd never even dreamt of registrating with that highly dubious "registration wizard". Because I never trusted that company, or their browsers, or their messengers, or their single-sign-on accounts.

But now they actually insisted on registration.

There was no easy way of ignoring their hunger for my personal data any more.

So the only viable alternative for me was to say goodbye to that big-brother OS and to look for something else.

I have never been a fan of that church from Cupertino. That OS 7 was pure horrors to me. Although their hardware was usually quite OK. Well, now the have a UNIX-like OS (they know why). But why only using some crippled-down UNIX-clone? Why not using the real thing instead?

So I started to take a closer look at Linux.

I had set up a SuSE and a Red Hat a couple of years ago, but I felt they were too complicated for me: So-called "user friendly" configuration tools tried to do their best to hide what they were doing and why. I never had a clue why something was happening. It (mostly) worked, but was otherwise black magic to me.

I eventually started to think that a Linux-From-Scratch-System might be the only way to learn what was going on behind the scenes.

But then: Always hunting the web for different updates, permanently figuring out what patches to apply, etc etc. That was a bit too hefty for me. (Especially still being a **** user.)

And then I saw Gentoo.

It seemed to combine the best aspects of an anarchy-style LFS system with those of the usual managed distro.

And the best was: No precompiled binary packages! I always hated the idea that vendors usually compile their binaries with the most "compatible" (= least effective) compiler settings they could find. I only wanted the best settings for my CPU!

Plus it's somewhat harder to ship trojans, viruses etc when only distributing source code packages. Not impossible, but undoubtedly it's a gain in security. (Especially compared to what that company was shipping all the years.)

And what about all that "it's all so complicated?"

I was all a lie!

I've never seen such a simple and transparent distro like Gentoo!

OK, it took me days to do the initial setup. (Stage 1 install of course.)

But then, how many days of my life had I wasted already for fiddling around in the **** Registry for fixing some problem, only to find out that the only way to fix it is a complete reinstall of that "OS"?

And it was so much fun installing Gentoo!

Being forced to do nearly everything by hand (I have never used that new graphical installer), the installation taught me much about how to actually do and setup things, which helped me very much later.

And - there was a community!

Under that other OS, when you had a really hard problem, you could go to some web forum only to learn that others had the same problem and could not fix it either. Very helpful, indeed.

But of course, there is that paid "vendor support" which you could use.

And I actually had to use at in the time I was developing solutions under ACCESS (Version 1.0).

There was some problem, so I called the support and they told me: "We know that. It's a ... well ... you have to wait for one of the next releases. We have already filed it; it will eventually be fixed."

What a help!

I'm much happier with the help I have got from the Gentoo community (and Linux community in general), than anything I've ever got from that paid support.

To be more specfic: Not the fact that the support was paid was the problem, but that it never provided me with anything useful at all:

If there was a solution for some problem, I found it myself in the web.

It I couldn't help myself, neither the support could.

No. I'm not missing that support even a bit!

And then - it's all so easy under Gentoo. OK, the initial setup usually takes longer as you have to learn how to configure things first. But then it works like a charm.

In that other OS it was exactly the other way around: Installation was a breeze.

But the farther one attempts to deviate from the factory defaults, the more problems were approaching for sure.

So, since a couple of months now I'm actually using Gentoo for serious work, and it gets better and better each day.

And I'm also learning and discovering a lot.

It's really fun!

And that other OS?

Well, it's still being used: As a gameloader.

There is simply nothing else it's good for.

[Guenther Brunthaler]

LDFLAGS="-Wl,-Bdirect" breaks this, right?

I was unable to find a -Bdirect option in the "ld" manual, so I'm not sure what it actually does.

But generally prelink helps only in cases where dynamic libraries (shared objects) are involved.

And it helps the more, the more relocations are contained within each shared library, because relocations require relatively much time to be performed.

BTW: For prelink, you should include 'pic' in your USE...

Yes, but I guess the "prelink" in "$FEATURES" will take care of this.

However, -fpic or -Fpic is necessary only if prelinking is not effective for a shared library, i. e. if it must be relocated (= moved to a different address in memory than that for which it was compiled for or prelinked to) at runtime.

I'm not completely certain about that, but it may even be more efficient if a shared library was linked without -fpic or -fPIC, as long as it can be guaranteed that the library will never require relocation to be performed on it.

This is because neither -fpic nor -fPIC come "for free".

Code compiled using these option will contain less efficient code, but in exchange the code will only contain a mimimum of potential relocations.

So, if one can say for sure a shared library will never need relocation (because prelinking will assign a different, unique memory area to each and every SO in the system), then pic/PIC will not provide any advantage, but still causes suboptimal code to be generated.

[piercey]

I wonder what would happen with oppenoffice if it could be compiled for amd64, and prelinked then.

Well no need to wonder, OpenOffice 2.0.4 compiles perfectly on amd64 and is already lightning fast compared to the -bin.

[vipernicus]

Even better: prelink + --hash-styles=gnu

[zxy]

Yup, new hash-style makes wonders.

[ChickensDontFly]

Does anyone know how to record application load times?

Point me in the right direction and I'll give us some test data to look at.

Also, list any apps you might want tested and I'll do my best to accomodate.

[Master_Of_Disaster]

Günther (and anyone else who is interested...), see this and this about -Bdirect.

[energyman76b]

BTW: For prelink, you should include 'pic' in your USE...

no, you should not.

on amd64 pic is set everywhere where it is supported anyway. And forcing it, where it is not supported breaks things.

Don't do it.