Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage in a ReiserFS Loop Device -- Faster & Save Space
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
yottabit
Guru
Guru


Joined: 11 Nov 2002
Posts: 313
Location: Columbus, Ohio, US

PostPosted: Fri Jan 26, 2007 5:16 am    Post subject: Portage in a ReiserFS Loop Device -- Faster & Save Space Reply with quote

Hi Fellow Party People,

I used ReiserFS for years, and it bit me hard twice over the 7 years or so by corrupting twice so badly that it took me about 12 hours each time of extensive recovery and renaming of each and every file handle that I vowed never to use it again. I went back to the venerable ext3 for a rock-solid, easily recoverable filesystem, and hey, it works great, as always. However...

ext2/3 is slow as dirt when it comes to lots of small files. (Just to keep a perspective, vfat & ntfs are slow as molasses, which is considerably slower than dirt. ;) ) And what is Portage but tons of small files? Portage is basically a database-on-filesystem. And that, my friends, is what ReiserFS was designed for! So, how to get the advantages of ReiserFS without living every day in constant fear of corruption?

One method would be to give yourself a 2 to 4 GB partition dedicated to ReiserFS and mounted on /usr/portage, but of course if you're like me and am already running a drive with full partitions and don't want to mess around with resizing to make room, you'll need a better solution. Enter the loop device...

I created a sparse (means zero-bytes inside the file don't take up any real room on the underlying filesystem; ext2/3, XFS, and other modern filesystems support this) 2 GB file, created a loop device, associated the 2 GB sparse file with the loop device, formatted the loop device with ReiserFS, mounted the loop device, moved all of Portage into it, and then moved the mount point to /usr/portage.

Be sure you have the loop device and ReiserFS both compiled either into your kernel or as a modules (and loaded), and then:

Create a sparse file by using the 'dd' utility. We choose our 'infile' as the 'zero' device, and our 'outfile' as the file we want to contain our filesystem. I set 'blocksize' to 32k to speed things up, and by using 'seek' in combination with 'count' we're telling 'dd' to seek to (32kbytes * 62500 = 2,000,000,000 bytes) = 2 GB mark without actually writing all of the zeros in between (which effectively makes our file sparse. You can see the effect the sparse file has by using the 'ls' and 'du' commands as shown in the below example. 'ls' shows us how large the "container" file is, and 'du' shows us how much space the file is actually using. Cool, eh?
Code:
# dd if=/dev/zero of=/mnt/portage.loopfs bs=32k seek=62500 count=0
# ls -lh /mnt/portage.loopfs
# du -h /mnt/portage.loopfs


Next we setup the loop device. (If you are already using loops, be sure to use losetup -f first to find your next available device.)
Code:
# losetup /dev/loop/0 /mnt/portage.loopfs


Now that our loop device is associated with our sparse file, we format the device just as we would any other device.
Code:
# mkfs.reiserfs /dev/loop/0


Finally, we'll get a speed baseline by scanning for disk usage on your existing filesystem, mount the new filesystem to a temporary mount point, move Portage into the new filesystem, unmount, remount where Portage should be, and finally scan the new filesystem for a speed comparison. Also note the difference in space being used in ReiserFS!
Code:
# mkdir /mnt/portage
# mount /dev/loop/0 /mnt/portage
# time du -ch /usr/portage | tail
# mv /usr/portage/* /mnt/portage/.
# umount /mnt/portage
# rmdir /mnt/portage
# mount /dev/loop/0 /usr/portage
# time du -ch /usr/portage | tail


Amazing, eh? I think the losetup will maintain the association with your sparse file between reboots, but don't forget to add the new filesystem to your /etc/fstab with a line like this:
Code:
/dev/loop/0 /usr/portage reiserfs noatime 0 0


This setup takes up approximately 700 MB less space on my disk, and Portage operations such as searching, sync, and other miscellaneous tasks done in the filesystem are in the order of 30x faster! Yes, that's 3,000%!

Enjoy.
_________________
Play The Hitchhiker's Guide to the Galaxy!
Back to top
View user's profile Send private message
dentharg
Guru
Guru


Joined: 10 Aug 2004
Posts: 438
Location: /poland/wroclaw

PostPosted: Fri Jan 26, 2007 9:24 am    Post subject: Reply with quote

Instead of slicing disk into partitions I use LVM2 and usually I have plenty of unused space.
That is I assign only as much space I need to a logical volume and extend it when needed.
EXT3 and ReiserFS both have the capability to grow (I have a ReiserFS volume for portage)

How would you grow a file and a filesystem inside?
Back to top
View user's profile Send private message
yottabit
Guru
Guru


Joined: 11 Nov 2002
Posts: 313
Location: Columbus, Ohio, US

PostPosted: Fri Jan 26, 2007 4:34 pm    Post subject: Reply with quote

Well, to grow the filesystem from within the loop device, I imagine you would umount, perhaps turn off the loop?, then append more zeros to the file, something like dd if=/dev/zero of=/mnt/portage.loopfs bs=32k seek=93750 count=0 which would add 1 GB to the 2 GB file I used in the example. (Don't try this at home unless you're prepared to completely lose that loop filesystem because I haven't tested this, and I can see just as logically how it would clear the file instead of adding onto it.)

Another method, perhaps safer, would be to dd if=/dev/zero of=/mnt/temp.zero bs=32k seek=31250 count=0; cat /mnt/temp.zero >> /mnt/portage.loopfs; rm /mnt/temp.zero and then setup the loop device again, mount it again, and then use your normal filesystem utilities to grow the filesystem to the new container. I think using this method would potentially add 1 GB of real space to the file instead of making it sparse, but again, I haven't tested it. (This creates a new, 1 GB sparse file, then we use cat to append it to the end of our loop file.)
_________________
Play The Hitchhiker's Guide to the Galaxy!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum