View previous topic :: View next topic |
Author |
Message |
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Sun Jul 30, 2006 10:56 pm Post subject: Storing portage-tree in one file: big speed gain |
|
|
I recently found this tip in the gentoo-wiki and since it seems to be rarely used (or at least I know of nobody who does) but really gives big speed-ups when browsing the tree, I thought I'd point people to it:
Portage in Sparse File
Here's an example of the speed gain, portage is the new tree and portage.old the ... well, old one:
Code: |
buren portage # time du -sh .
424M .
real 0m16.825s
user 0m0.248s
sys 0m2.134s
buren portage # time find . > /dev/null
real 0m1.075s
user 0m0.260s
sys 0m0.802s
buren portage # cd ../portage.old/
buren portage.old # time du -sh .
401M .
real 0m56.485s
user 0m0.309s
sys 0m1.925s
buren portage.old # time find . > /dev/null
real 0m0.840s
user 0m0.230s
sys 0m0.600s
buren portage.old #
|
Also any other operations like emerge -s or equery work much faster. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54237 Location: 56N 3W
|
Posted: Mon Jul 31, 2006 1:35 pm Post subject: |
|
|
Moved from Portage & Programming to Documentation, Tips & Tricks. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
irondog l33t
Joined: 07 Jul 2003 Posts: 715 Location: Voor mijn TV. Achter mijn pc.
|
Posted: Mon Jul 31, 2006 10:57 pm Post subject: |
|
|
Okay, Reiser is the better choice for small files then. The trick is to think well about which filesystem to use on which files.
Really, loop mounting a file carrying a filesystem is nothing special. It should also be clear that portage uses a lot of files. _________________ Alle dingen moeten onzin zijn. |
|
Back to top |
|
|
xanas3712 Guru
Joined: 15 Oct 2004 Posts: 455
|
Posted: Mon Jul 31, 2006 11:57 pm Post subject: |
|
|
Well yes, but if it's so obvious why isn't everyone doing it? I think it was worth posting..
BTW, what is meant by
Quote: |
and if you want some real speed, you could do the same with /var/db/pkg and /var/cache/edb ( faster dependency checks etc ) , but doing that is bound to end in tears
|
I was thinking about going ahead and doing this too, but I'm not sure why it would end in tears? |
|
Back to top |
|
|
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Tue Aug 01, 2006 2:20 am Post subject: |
|
|
irondog,
yes, it might be obvious, but I think it isn't widely used. And so I thought it was worth posting to point people to that idea.
xanas,
I wondered about that as well and don't see how it could be any different from /usr/portage. As long as one keeps a backup of the dirs trying shouldn't do any harm, so maybe I'll give it a try tomorrow. |
|
Back to top |
|
|
synss Apprentice
Joined: 08 Mar 2006 Posts: 282 Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
|
Back to top |
|
|
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Tue Aug 01, 2006 11:16 am Post subject: |
|
|
Compressing the tree sounded interesting at first, but if I got it right updateing the tree still requires the full 500-600MB. So the gain of space will only be temporarily, meaning I cannot use it for anything I want to keep for some longer time. |
|
Back to top |
|
|
xanas3712 Guru
Joined: 15 Oct 2004 Posts: 455
|
Posted: Tue Aug 01, 2006 11:31 am Post subject: |
|
|
Yeah, I like this way, and it is very very fast by comparison to how it was. I went ahead and backed up the other dirs and did them as well, not as much of a speed boost as doing it for the tree but still pretty nice. |
|
Back to top |
|
|
yoshi314 l33t
Joined: 30 Dec 2004 Posts: 850 Location: PL
|
Posted: Tue Aug 01, 2006 12:37 pm Post subject: |
|
|
Quote: | Compressing the tree sounded interesting at first, but if I got it right updateing the tree still requires the full 500-600MB. | nope, you missed a thing
the new squashfs image is created directly from mounted squashfs image and unionfs branch which contains the changes made by e.g recent emerge --sync. so there is no need for unpacking the squashfs image.
at least the squashfs-initscript method does that. _________________ ~amd64
shrink your /usr/portage with squashfs+aufs
Last edited by yoshi314 on Tue Aug 01, 2006 12:39 pm; edited 1 time in total |
|
Back to top |
|
|
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Fri Aug 04, 2006 2:06 pm Post subject: |
|
|
yoshi314,
I see, I'll try that method when I find some time.
Regarding placing portage directories in a sparse file, I tried this with /var/db/pkg and /var/cache/edb as suggested by the wiki-article but actually experienced a decrease in speed. Both "du" and "find" took about 20-30% more time. It seems like the structure of these dirs differs from /usr/portage so they don't benefit from reiserfs but rather the opposite. |
|
Back to top |
|
|
netmaster n00b
Joined: 04 Aug 2006 Posts: 4
|
Posted: Fri Aug 04, 2006 8:14 pm Post subject: |
|
|
Dude! Excellent tip man. Wow. Thanks! _________________ Helping Others Helps You
Career
Resources |
|
Back to top |
|
|
c0r0 Tux's lil' helper
Joined: 25 Jun 2006 Posts: 81 Location: Jakarta -Indonesia
|
Posted: Sat Aug 05, 2006 3:40 pm Post subject: |
|
|
Ohh the tips works well and i like it thanks for this _________________ - Laughter, The Best Medicine - |
|
Back to top |
|
|
xming Guru
Joined: 02 Jul 2002 Posts: 441
|
Posted: Sun Aug 06, 2006 12:29 pm Post subject: |
|
|
Instead of sparse file or squashfs you can use jffs2, which only use +/- 120 MB and it's fully writable, see http://wojia.be for more details _________________ http://wojia.be |
|
Back to top |
|
|
yoshi314 l33t
Joined: 30 Dec 2004 Posts: 850 Location: PL
|
Posted: Mon Aug 07, 2006 6:45 pm Post subject: |
|
|
xming wrote: | Instead of sparse file or squashfs you can use jffs2, which only use +/- 120 MB and it's fully writable, see http://wojia.be for more details | 120mb is a lot, compared to squashfs image, which takes ~35mb. (at least with squashfs 3.0)
and if you just did emerge --sync, unionfs takes a couple of megs of RAM to track the changes, until you rebuild the squashfs image. _________________ ~amd64
shrink your /usr/portage with squashfs+aufs |
|
Back to top |
|
|
xming Guru
Joined: 02 Jul 2002 Posts: 441
|
Posted: Tue Aug 08, 2006 12:20 pm Post subject: |
|
|
120MB is stll a lot beter tahn 600MB + jffs2 is in the vanilla kernel and you never have to rebuild anything. HOw big is you unionfs overlay after few syncs? _________________ http://wojia.be |
|
Back to top |
|
|
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Thu Aug 10, 2006 12:53 am Post subject: |
|
|
I tried making /usr/portage a jffs2 file but it didn't work.
Here's what I did:
Code: | # I use loop1 as loop0 is already in use
# I substituted blkmtd for block2mtd, I think that's what it's called in my kernel (ck-2.6.17)
# dmesg | tail or grepping for mtd didn't yield anything
03:53:16 [/usr/src/linux] # mknod /dev/mtdblock0 b 31 0
03:53:46 [/usr/src/linux] # losetup /dev/loop1 /root/portage.jffs2
03:53:58 [/usr/src/linux] # modprobe block2mtd device=/dev/loop1
03:54:11 [/usr/src/linux] # mount -t jffs2 /dev/mtdblock0 /usr/portage.new/
mount: wrong fs type, bad option, bad superblock on /dev/mtdblock0,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
|
Any idea what's wrong? |
|
Back to top |
|
|
xming Guru
Joined: 02 Jul 2002 Posts: 441
|
Posted: Thu Aug 10, 2006 10:34 am Post subject: |
|
|
Is jffs2 compiled in the kernel or as a module? Did you modprobe jffs2? Is your /root/portage.jffs2 formatted as jffs2?
Can you post the output of dmesg|tail? _________________ http://wojia.be |
|
Back to top |
|
|
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Thu Aug 10, 2006 11:41 am Post subject: |
|
|
Strangely now another error popped up, when trying to load block2mtd I get this: Code: | FATAL: Error inserting block2mtd (/lib/modules/2.6.17-ck1-r2/kernel/drivers/mtd/devices/block2mtd.ko): Unknown symbol in module, or unknown parameter (see dmesg) | And dmesg|tail says "block2mtd: Unknown parameter `device'".
I don't know why this happens now as yesterday it didn't and I entered the same comand as before: Code: | 13:38:53 [/home/bur] # modprobe jffs2
13:39:09 [/home/bur] # mknod /dev/mtdblock0 b 31 0
13:39:17 [/home/bur] # losetup /dev/loop1 /root/portage.jffs2
13:39:24 [/home/bur] # modprobe block2mtd device=/dev/loop1
FATAL: Error inserting block2mtd (/lib/modules/2.6.17-ck1-r2/kernel/drivers/mtd/devices/block2mtd.ko): Unknown symbol in module, or unknown parameter (see dmesg) |
Is there any way to find out which parameters a module supports? Maybe it isn't called device anymore. Or maybe block2mtd isn't the right module at all, though I doubt that. |
|
Back to top |
|
|
xming Guru
Joined: 02 Jul 2002 Posts: 441
|
Posted: Thu Aug 10, 2006 11:51 am Post subject: |
|
|
replace
Quote: | modprobe block2mtd device=/dev/loop1 |
with
Code: | modprobe block2mtd block2mtd=/dev/loop1 |
this is my script for 2.6.17
Code: |
modprobe loop
mknod /dev/mtdblock0 b 31 0
losetup /dev/loop0 /tmp/portage.jffs2
modprobe block2mtd block2mtd=/dev/loop0
mount -t jffs2 /dev/mtdblock0 /usr/portage
|
For info about the modules try
Code: |
modinfo block2mtd
filename: /lib/modules/2.6.17-beyond3/kernel/drivers/mtd/devices/block2mtd.ko
license: GPL
author: Simon Evans <spse@secret.org.uk> and others
description: Emulate an MTD using a block device
vermagic: 2.6.17-beyond3 preempt mod_unload PENTIUMIII gcc-3.4
depends: mtdcore
srcversion: EA21219D30D40143B1D006C
parm: block2mtd:Device to use. "block2mtd=<dev>[,<erasesize>]"
|
_________________ http://wojia.be |
|
Back to top |
|
|
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Thu Aug 10, 2006 2:15 pm Post subject: |
|
|
Thanks, it works now and takes much less space then the sparse file and is much faster to access. What do I need to do to have it mounted automatically each boot?
I think I found out I have to run losetup each boot and after that load block2mtd (because it needs the /dev/loop). How could this be done most easily? And when adding the jffs2 file to fstab, what parameters are needed? For example the sparse method used loop,noatime and some more.
Also where exactly is the portage tree stored? Is it like that:
/tmp/portage.jffs2 -> /dev/loop0 -> /dev/mtdblock0 -> /usr/portage ? But if the actual data is in portage.jffs2, why is the file placed in /tmp? The data is not only needed temporarily, if I got it right. |
|
Back to top |
|
|
xming Guru
Joined: 02 Jul 2002 Posts: 441
|
Posted: Thu Aug 10, 2006 7:38 pm Post subject: |
|
|
bur wrote: | Thanks, it works now and takes much less space then the sparse file and is much faster to access. What do I need to do to have it mounted automatically each boot? |
I just put the script in /etc/conf.d/local.start
Quote: |
Also where exactly is the portage tree stored? Is it like that:
/tmp/portage.jffs2 -> /dev/loop0 -> /dev/mtdblock0 -> /usr/portage ? But if the actual data is in portage.jffs2, why is the file placed in /tmp? The data is not only needed temporarily, if I got it right.
|
data is in /tmp/portage.jff2, for me it's tmp, which it is data that can be deleted, screwed, .... If you prefer to place it elsewhere (I think /var/spool would be the perfect place) you can do that. _________________ http://wojia.be |
|
Back to top |
|
|
bur Apprentice
Joined: 20 Feb 2004 Posts: 229
|
Posted: Wed Aug 16, 2006 12:46 am Post subject: |
|
|
Everything works fine now but I ran out of space on /dev/mtdblock0 making emerge --sync impossible. How can I resize the partition besides creating a new one from scratch? And is there a way to make it a little bigger than needed so that I won't run into that problem again? |
|
Back to top |
|
|
xming Guru
Joined: 02 Jul 2002 Posts: 441
|
Posted: Sun Aug 20, 2006 8:42 am Post subject: |
|
|
are you using a partition or a loopback file? If it's a loopback, you just need to recreate the file. _________________ http://wojia.be |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Mon Aug 21, 2006 6:34 pm Post subject: |
|
|
I'm running Gentoo on a Pentium3-850Mhz with 256Mb RAM...
I use ReiserFS on my partitions so I had to use ext2 as filesystem in the file.
Before putting Portage in the file, an "emerge --sync" would take about 20 minutes.
After putting Portage in the file it only takes 10 minutes
Thanks! _________________ When all else fails, read the manual...
Registered Linux User #340626
Last edited by xentric on Wed Aug 23, 2006 12:08 am; edited 1 time in total |
|
Back to top |
|
|
zxy Veteran
Joined: 06 Jan 2006 Posts: 1160 Location: in bed in front of the computer
|
Posted: Tue Aug 22, 2006 1:37 pm Post subject: |
|
|
Did anybody try to copy the squashfs file to ram and mount it there.
It's just 35 or 40 Megs.
What are the speeds/timings then? _________________ Nature does not hurry, yet everything is accomplished.
Lao Tzu |
|
Back to top |
|
|
|