Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Storing portage-tree in one file: big speed gain
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Sun Jul 30, 2006 10:56 pm    Post subject: Storing portage-tree in one file: big speed gain Reply with quote

I recently found this tip in the gentoo-wiki and since it seems to be rarely used (or at least I know of nobody who does) but really gives big speed-ups when browsing the tree, I thought I'd point people to it:

Portage in Sparse File

Here's an example of the speed gain, portage is the new tree and portage.old the ... well, old one:
Code:

buren portage # time du -sh .
424M    .

real    0m16.825s
user    0m0.248s
sys     0m2.134s
buren portage # time find . > /dev/null

real    0m1.075s
user    0m0.260s
sys     0m0.802s
buren portage # cd ../portage.old/
buren portage.old # time du -sh .
401M    .

real    0m56.485s
user    0m0.309s
sys     0m1.925s
buren portage.old # time find . > /dev/null

real    0m0.840s
user    0m0.230s
sys     0m0.600s
buren portage.old #


Also any other operations like emerge -s or equery work much faster.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 31341
Location: 56N 3W

PostPosted: Mon Jul 31, 2006 1:35 pm    Post subject: Reply with quote

Moved from Portage & Programming to Documentation, Tips & Tricks.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
irondog
l33t
l33t


Joined: 07 Jul 2003
Posts: 715
Location: Voor mijn TV. Achter mijn pc.

PostPosted: Mon Jul 31, 2006 10:57 pm    Post subject: Reply with quote

Okay, Reiser is the better choice for small files then. The trick is to think well about which filesystem to use on which files.

Really, loop mounting a file carrying a filesystem is nothing special. It should also be clear that portage uses a lot of files.
_________________
Alle dingen moeten onzin zijn.
Back to top
View user's profile Send private message
xanas3712
Guru
Guru


Joined: 15 Oct 2004
Posts: 455

PostPosted: Mon Jul 31, 2006 11:57 pm    Post subject: Reply with quote

Well yes, but if it's so obvious why isn't everyone doing it? I think it was worth posting..

BTW, what is meant by

Quote:

and if you want some real speed, you could do the same with /var/db/pkg and /var/cache/edb ( faster dependency checks etc ) , but doing that is bound to end in tears ;)


I was thinking about going ahead and doing this too, but I'm not sure why it would end in tears?
Back to top
View user's profile Send private message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Tue Aug 01, 2006 2:20 am    Post subject: Reply with quote

irondog,
yes, it might be obvious, but I think it isn't widely used. And so I thought it was worth posting to point people to that idea.

xanas,
I wondered about that as well and don't see how it could be any different from /usr/portage. As long as one keeps a backup of the dirs trying shouldn't do any harm, so maybe I'll give it a try tomorrow.
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 280
Location: Dijon (F) > Berlin > Tokyo > Nürnberg (D)

PostPosted: Tue Aug 01, 2006 9:21 am    Post subject: Reply with quote

The nice thing with doing it with the tree is that if ever anything would go wrong, you can just download a new one. Recovery is nothing that easy with other dirs, however, if you keep backups, then it sure is fine.

And you can also compress the tree since you are at it

https://forums.gentoo.org/viewtopic-t-465367-highlight-.html
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Tue Aug 01, 2006 11:16 am    Post subject: Reply with quote

Compressing the tree sounded interesting at first, but if I got it right updateing the tree still requires the full 500-600MB. So the gain of space will only be temporarily, meaning I cannot use it for anything I want to keep for some longer time.
Back to top
View user's profile Send private message
xanas3712
Guru
Guru


Joined: 15 Oct 2004
Posts: 455

PostPosted: Tue Aug 01, 2006 11:31 am    Post subject: Reply with quote

Yeah, I like this way, and it is very very fast by comparison to how it was. I went ahead and backed up the other dirs and did them as well, not as much of a speed boost as doing it for the tree but still pretty nice.
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 817
Location: PL

PostPosted: Tue Aug 01, 2006 12:37 pm    Post subject: Reply with quote

Quote:
Compressing the tree sounded interesting at first, but if I got it right updateing the tree still requires the full 500-600MB.
nope, you missed a thing ;-)

the new squashfs image is created directly from mounted squashfs image and unionfs branch which contains the changes made by e.g recent emerge --sync. so there is no need for unpacking the squashfs image.

at least the squashfs-initscript method does that.
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs


Last edited by yoshi314 on Tue Aug 01, 2006 12:39 pm; edited 1 time in total
Back to top
View user's profile Send private message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Fri Aug 04, 2006 2:06 pm    Post subject: Reply with quote

yoshi314,
I see, I'll try that method when I find some time.


Regarding placing portage directories in a sparse file, I tried this with /var/db/pkg and /var/cache/edb as suggested by the wiki-article but actually experienced a decrease in speed. Both "du" and "find" took about 20-30% more time. It seems like the structure of these dirs differs from /usr/portage so they don't benefit from reiserfs but rather the opposite.
Back to top
View user's profile Send private message
netmaster
n00b
n00b


Joined: 04 Aug 2006
Posts: 4

PostPosted: Fri Aug 04, 2006 8:14 pm    Post subject: Reply with quote

Dude! Excellent tip man. Wow. Thanks!
_________________
Helping Others Helps You
Career
Resources
Back to top
View user's profile Send private message
c0r0
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jun 2006
Posts: 81
Location: Jakarta -Indonesia

PostPosted: Sat Aug 05, 2006 3:40 pm    Post subject: Reply with quote

Ohh the tips works well and i like it :D thanks for this
_________________
- Laughter, The Best Medicine -
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Sun Aug 06, 2006 12:29 pm    Post subject: Reply with quote

Instead of sparse file or squashfs you can use jffs2, which only use +/- 120 MB and it's fully writable, see http://wojia.be for more details
_________________
http://wojia.be
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 817
Location: PL

PostPosted: Mon Aug 07, 2006 6:45 pm    Post subject: Reply with quote

xming wrote:
Instead of sparse file or squashfs you can use jffs2, which only use +/- 120 MB and it's fully writable, see http://wojia.be for more details
120mb is a lot, compared to squashfs image, which takes ~35mb. (at least with squashfs 3.0)

and if you just did emerge --sync, unionfs takes a couple of megs of RAM to track the changes, until you rebuild the squashfs image.
_________________
~amd64, ~x86
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Tue Aug 08, 2006 12:20 pm    Post subject: Reply with quote

120MB is stll a lot beter tahn 600MB + jffs2 is in the vanilla kernel and you never have to rebuild anything. HOw big is you unionfs overlay after few syncs?
_________________
http://wojia.be
Back to top
View user's profile Send private message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Thu Aug 10, 2006 12:53 am    Post subject: Reply with quote

I tried making /usr/portage a jffs2 file but it didn't work.

Here's what I did:
Code:
# I use loop1 as loop0 is already in use
# I substituted blkmtd for block2mtd, I think that's what it's called in my kernel (ck-2.6.17)
# dmesg | tail or grepping for mtd didn't yield anything

03:53:16 [/usr/src/linux] # mknod /dev/mtdblock0 b 31 0
03:53:46 [/usr/src/linux] # losetup /dev/loop1 /root/portage.jffs2
03:53:58 [/usr/src/linux] # modprobe block2mtd device=/dev/loop1
03:54:11 [/usr/src/linux] # mount -t jffs2 /dev/mtdblock0 /usr/portage.new/
mount: wrong fs type, bad option, bad superblock on /dev/mtdblock0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so


Any idea what's wrong?
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Thu Aug 10, 2006 10:34 am    Post subject: Reply with quote

Is jffs2 compiled in the kernel or as a module? Did you modprobe jffs2? Is your /root/portage.jffs2 formatted as jffs2?

Can you post the output of dmesg|tail?
_________________
http://wojia.be
Back to top
View user's profile Send private message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Thu Aug 10, 2006 11:41 am    Post subject: Reply with quote

Strangely now another error popped up, when trying to load block2mtd I get this:
Code:
FATAL: Error inserting block2mtd (/lib/modules/2.6.17-ck1-r2/kernel/drivers/mtd/devices/block2mtd.ko): Unknown symbol in module, or unknown parameter (see dmesg)
And dmesg|tail says "block2mtd: Unknown parameter `device'".

I don't know why this happens now as yesterday it didn't and I entered the same comand as before:
Code:
13:38:53 [/home/bur] # modprobe jffs2
13:39:09 [/home/bur] # mknod /dev/mtdblock0 b 31 0
13:39:17 [/home/bur] # losetup /dev/loop1 /root/portage.jffs2
13:39:24 [/home/bur] # modprobe block2mtd device=/dev/loop1
FATAL: Error inserting block2mtd (/lib/modules/2.6.17-ck1-r2/kernel/drivers/mtd/devices/block2mtd.ko): Unknown symbol in module, or unknown parameter (see dmesg)


Is there any way to find out which parameters a module supports? Maybe it isn't called device anymore. Or maybe block2mtd isn't the right module at all, though I doubt that.
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Thu Aug 10, 2006 11:51 am    Post subject: Reply with quote

replace

Quote:
modprobe block2mtd device=/dev/loop1


with

Code:
modprobe block2mtd block2mtd=/dev/loop1


this is my script for 2.6.17

Code:

modprobe loop
mknod /dev/mtdblock0 b 31 0
losetup /dev/loop0 /tmp/portage.jffs2
modprobe block2mtd block2mtd=/dev/loop0
mount -t jffs2 /dev/mtdblock0 /usr/portage


For info about the modules try

Code:

modinfo block2mtd
filename:       /lib/modules/2.6.17-beyond3/kernel/drivers/mtd/devices/block2mtd.ko
license:        GPL
author:         Simon Evans <spse@secret.org.uk> and others
description:    Emulate an MTD using a block device
vermagic:       2.6.17-beyond3 preempt mod_unload PENTIUMIII gcc-3.4
depends:        mtdcore
srcversion:     EA21219D30D40143B1D006C
parm:           block2mtd:Device to use. "block2mtd=<dev>[,<erasesize>]"

_________________
http://wojia.be
Back to top
View user's profile Send private message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Thu Aug 10, 2006 2:15 pm    Post subject: Reply with quote

Thanks, it works now and takes much less space then the sparse file and is much faster to access. What do I need to do to have it mounted automatically each boot?

I think I found out I have to run losetup each boot and after that load block2mtd (because it needs the /dev/loop). How could this be done most easily? And when adding the jffs2 file to fstab, what parameters are needed? For example the sparse method used loop,noatime and some more.

Also where exactly is the portage tree stored? Is it like that:
/tmp/portage.jffs2 -> /dev/loop0 -> /dev/mtdblock0 -> /usr/portage ? But if the actual data is in portage.jffs2, why is the file placed in /tmp? The data is not only needed temporarily, if I got it right.
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Thu Aug 10, 2006 7:38 pm    Post subject: Reply with quote

bur wrote:
Thanks, it works now and takes much less space then the sparse file and is much faster to access. What do I need to do to have it mounted automatically each boot?


I just put the script in /etc/conf.d/local.start

Quote:

Also where exactly is the portage tree stored? Is it like that:
/tmp/portage.jffs2 -> /dev/loop0 -> /dev/mtdblock0 -> /usr/portage ? But if the actual data is in portage.jffs2, why is the file placed in /tmp? The data is not only needed temporarily, if I got it right.


data is in /tmp/portage.jff2, for me it's tmp, which it is data that can be deleted, screwed, .... If you prefer to place it elsewhere (I think /var/spool would be the perfect place) you can do that.
_________________
http://wojia.be
Back to top
View user's profile Send private message
bur
Apprentice
Apprentice


Joined: 20 Feb 2004
Posts: 229

PostPosted: Wed Aug 16, 2006 12:46 am    Post subject: Reply with quote

Everything works fine now but I ran out of space on /dev/mtdblock0 making emerge --sync impossible. How can I resize the partition besides creating a new one from scratch? And is there a way to make it a little bigger than needed so that I won't run into that problem again?
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Sun Aug 20, 2006 8:42 am    Post subject: Reply with quote

are you using a partition or a loopback file? If it's a loopback, you just need to recreate the file.
_________________
http://wojia.be
Back to top
View user's profile Send private message
xentric
Guru
Guru


Joined: 16 Mar 2003
Posts: 410
Location: Netherlands

PostPosted: Mon Aug 21, 2006 6:34 pm    Post subject: Reply with quote

I'm running Gentoo on a Pentium3-850Mhz with 256Mb RAM...
I use ReiserFS on my partitions so I had to use ext2 as filesystem in the file.
Before putting Portage in the file, an "emerge --sync" would take about 20 minutes.
After putting Portage in the file it only takes 10 minutes :D

Thanks!
_________________
When all else fails, read the manual...
Registered Linux User #340626


Last edited by xentric on Wed Aug 23, 2006 12:08 am; edited 1 time in total
Back to top
View user's profile Send private message
zxy
Veteran
Veteran


Joined: 06 Jan 2006
Posts: 1160
Location: in bed in front of the computer

PostPosted: Tue Aug 22, 2006 1:37 pm    Post subject: Reply with quote

Did anybody try to copy the squashfs file to ram and mount it there.
It's just 35 or 40 Megs.

What are the speeds/timings then?
_________________
Nature does not hurry, yet everything is accomplished.
Lao Tzu
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum