improving emerge performance

chutz · n00b Joined: 03 Feb 2003 Posts: 25 Location: Sapporo

I don't know if this has ever been covered here (tried searching, but had trouble finding), so I decided I'd just share a tip others may find useful.

The idea is that emerge -s or even emerge -p take quite some time on my system. emerge -s is ridiculously slow and emerge -p is just not as fast as I'd like it to be.

The short story is: I mounted /usr/portage on a loopback device. I am not sure if the fact that I formatted the loopback as reiserfs and the rest of the system is ext3 has anything to do with it, but performance is much better than before. I.e.:

Genone · Posted: Fri Jul 18, 2003 1:49 pm Post subject:

reiserfs is significantly faster for small files (ebuilds are small in most cases) than ext2/3. You could create a second loopback file with ext3 for comparison.

eckythump · n00b Joined: 24 Jul 2003 Posts: 1

I had the thought that by using a [i]loop backed filesystem in a file[/i] you could rsync that file instead of rsyncing the portage directory tree. This could make "[B]emerge rsync[/B]" significantly faster or perhaps much more [I]scaleable[/I] particularly solving the problem of people over frequently hitting emerge rsync and requiring at the least a ~1meg dir listing... as opposed to a single checksum... then split rechecksum (we all know how rsync works) .. any way ... it would certainly [i]improve the worst case[/i]and potentially could be done without changing too much other stuff (clearly importaint). Although i will admit it could have the blocking problem that the loop back might need to be unmounted while the rsync happens but this might be ok.. after all should you be using emerge foo whilst emerge rsync?

not sure where to put this idea but i did a quick search for "portage loop rsync" and this was at the top... now to go back and find out that someone else has already had the idea :oops:

or not

Kalin · Last edited by Kalin on Mon Apr 12, 2004 2:36 pm; edited 1 time in total

I was just following chutz' suggestion, so I made some slight (say cosmetic) improvements and preparations. Here it is:

tomaw · Guru Joined: 26 Mar 2003 Posts: 429 Location: UK

Kalin's suggestions seem to work just fine, although I forgot to emerge reiserfsprogs before starting, so had to reset /etc/make.conf.

Also, I presume something that is a sideeffect of using udev is I've had to add loop to my modules autoload file, or the mount command fails. It doesn't seem bothered if the reiserfs module is loaded or not though, it gets auto-loaded by mount.

Seems a little quicker too.
_________________
Tom Wesley

Kalin · Posted: Mon Apr 12, 2004 2:35 pm Post subject:

Since a lot of time passed, now I am confident in the ReiserFS, so most partitions are such.

Just keeping /boot in ext3 for some unknown reason...

And I am thinking to meve portage tree to /var/portage or something, and mount /usr ro. We'll see.

BTW /usr/portage has grown a bit (290MB at present), so adjust the loop file accordingly.

chutz · n00b Joined: 03 Feb 2003 Posts: 25 Location: Sapporo

If you need more space on the loopback:

chutz · n00b Joined: 03 Feb 2003 Posts: 25 Location: Sapporo

It is also good to sync frequently or we may run out of space when we finally sync, because portage does "rsync --delete-after". Doing a "rsync -av --delete rsync.gentoo.org::gentoo-portage/ /usr/portage/" manually is a workaround, but one would need to interrupt the sync after the files are deleted.