Storing portage-tree in one file: big speed gain

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

I recently found this tip in the gentoo-wiki and since it seems to be rarely used (or at least I know of nobody who does) but really gives big speed-ups when browsing the tree, I thought I'd point people to it:

Portage in Sparse File

Here's an example of the speed gain, portage is the new tree and portage.old the ... well, old one:

NeddySeagoon · Posted: Mon Jul 31, 2006 1:35 pm Post subject:

Moved from Portage & Programming to Documentation, Tips & Tricks.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

irondog · Posted: Mon Jul 31, 2006 10:57 pm Post subject:

Okay, Reiser is the better choice for small files then. The trick is to think well about which filesystem to use on which files.

Really, loop mounting a file carrying a filesystem is nothing special. It should also be clear that portage uses a lot of files.
_________________
Alle dingen moeten onzin zijn.

xanas3712 · Guru Joined: 15 Oct 2004 Posts: 455

Well yes, but if it's so obvious why isn't everyone doing it? I think it was worth posting..

BTW, what is meant by

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

irondog,
yes, it might be obvious, but I think it isn't widely used. And so I thought it was worth posting to point people to that idea.

xanas,
I wondered about that as well and don't see how it could be any different from /usr/portage. As long as one keeps a backup of the dirs trying shouldn't do any harm, so maybe I'll give it a try tomorrow.

synss · Posted: Tue Aug 01, 2006 9:21 am Post subject:

The nice thing with doing it with the tree is that if ever anything would go wrong, you can just download a new one. Recovery is nothing that easy with other dirs, however, if you keep backups, then it sure is fine.

And you can also compress the tree since you are at it

https://forums.gentoo.org/viewtopic-t-465367-highlight-.html
_________________
Compress portage tree
Elog viewer
Autodetect swap

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

Compressing the tree sounded interesting at first, but if I got it right updateing the tree still requires the full 500-600MB. So the gain of space will only be temporarily, meaning I cannot use it for anything I want to keep for some longer time.

xanas3712 · Guru Joined: 15 Oct 2004 Posts: 455

Yeah, I like this way, and it is very very fast by comparison to how it was. I went ahead and backed up the other dirs and did them as well, not as much of a speed boost as doing it for the tree but still pretty nice.

yoshi314 · l33t Joined: 30 Dec 2004 Posts: 850 Location: PL

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

yoshi314,
I see, I'll try that method when I find some time.

Regarding placing portage directories in a sparse file, I tried this with /var/db/pkg and /var/cache/edb as suggested by the wiki-article but actually experienced a decrease in speed. Both "du" and "find" took about 20-30% more time. It seems like the structure of these dirs differs from /usr/portage so they don't benefit from reiserfs but rather the opposite.

netmaster · n00b Joined: 04 Aug 2006 Posts: 4

Dude! Excellent tip man. Wow. Thanks!
_________________
Helping Others Helps You
Career
Resources

c0r0 · Posted: Sat Aug 05, 2006 3:40 pm Post subject:

Ohh the tips works well and i like it

thanks for this
_________________
- Laughter, The Best Medicine -

xming · Guru Joined: 02 Jul 2002 Posts: 441

Instead of sparse file or squashfs you can use jffs2, which only use +/- 120 MB and it's fully writable, see http://wojia.be for more details
_________________
http://wojia.be

yoshi314 · l33t Joined: 30 Dec 2004 Posts: 850 Location: PL

xming · Guru Joined: 02 Jul 2002 Posts: 441

120MB is stll a lot beter tahn 600MB + jffs2 is in the vanilla kernel and you never have to rebuild anything. HOw big is you unionfs overlay after few syncs?
_________________
http://wojia.be

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

I tried making /usr/portage a jffs2 file but it didn't work.

Here's what I did:

xming · Guru Joined: 02 Jul 2002 Posts: 441

Is jffs2 compiled in the kernel or as a module? Did you modprobe jffs2? Is your /root/portage.jffs2 formatted as jffs2?

Can you post the output of dmesg|tail?
_________________
http://wojia.be

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

Strangely now another error popped up, when trying to load block2mtd I get this:

xming · Guru Joined: 02 Jul 2002 Posts: 441

replace

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

Thanks, it works now and takes much less space then the sparse file and is much faster to access. What do I need to do to have it mounted automatically each boot?

I think I found out I have to run losetup each boot and after that load block2mtd (because it needs the /dev/loop). How could this be done most easily? And when adding the jffs2 file to fstab, what parameters are needed? For example the sparse method used loop,noatime and some more.

Also where exactly is the portage tree stored? Is it like that:
/tmp/portage.jffs2 -> /dev/loop0 -> /dev/mtdblock0 -> /usr/portage ? But if the actual data is in portage.jffs2, why is the file placed in /tmp? The data is not only needed temporarily, if I got it right.

xming · Guru Joined: 02 Jul 2002 Posts: 441

bur · Apprentice Joined: 20 Feb 2004 Posts: 229

Everything works fine now but I ran out of space on /dev/mtdblock0 making emerge --sync impossible. How can I resize the partition besides creating a new one from scratch? And is there a way to make it a little bigger than needed so that I won't run into that problem again?

xming · Guru Joined: 02 Jul 2002 Posts: 441

are you using a partition or a loopback file? If it's a loopback, you just need to recreate the file.
_________________
http://wojia.be

xentric · Guru Joined: 16 Mar 2003 Posts: 410 Location: Netherlands

I'm running Gentoo on a Pentium3-850Mhz with 256Mb RAM...
I use ReiserFS on my partitions so I had to use ext2 as filesystem in the file.
Before putting Portage in the file, an "emerge --sync" would take about 20 minutes.
After putting Portage in the file it only takes 10 minutes

Thanks!
_________________
When all else fails, read the manual...
Registered Linux User #340626

zxy · Posted: Tue Aug 22, 2006 1:37 pm Post subject:

Did anybody try to copy the squashfs file to ram and mount it there.
It's just 35 or 40 Megs.

What are the speeds/timings then?
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu