Forums

Skip to content

Advanced search
  • Quick links
    • Unanswered topics
    • Active topics
    • Search
  • FAQ
  • Login
  • Register
  • Board index Discussion & Documentation Documentation, Tips & Tricks
  • Search

Script:Measuring fragmentation on Reiserfs (and other fs)

Unofficial documentation for various parts of Gentoo Linux. Note: This is not a support forum.
Post Reply
Advanced search
52 posts
  • 1
  • 2
  • 3
  • Next
Author
Message
_droop_
l33t
l33t
Posts: 957
Joined: Sun May 30, 2004 12:15 pm

Script:Measuring fragmentation on Reiserfs (and other fs)

  • Quote

Post by _droop_ » Fri Feb 03, 2006 6:10 pm

Hi,

Reiserfs is one fs which has no tools to measure fragmentation.

I asked myself how can we measure fragmentation on a reiserfs partition. I wrote a script that is capable to measure this fragmentation.

This script requires e2fsprogs package (for filefrag command), but this package is installed on almost all gentoo. It is written in perl.

Since filefrag works on various fs type (including reiserfs), my script should work too on various fs type.

The script :

Code: Select all

#!/usr/bin/perl -w

#this script search for frag on a fs
use strict;

#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;

#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f |");

while (defined (my $file = <FILES>)) {
        #quote some chars in filename
        $file =~ s/!/\\!/g;
        $file =~ s/#/\\#/g;
        $file =~ s/&/\\&/g;
        $file =~ s/>/\\>/g;
        $file =~ s/</\\</g;
        $file =~ s/\$/\\\$/g;
        $file =~ s/\(/\\\(/g;
        $file =~ s/\)/\\\)/g;
        $file =~ s/\|/\\\|/g;
        $file =~ s/'/\\'/g;
        $file =~ s/ /\\ /g;
        #nb of fragment for the file
        open (FRAG, "filefrag $file |");
        my $res = <FRAG>;
        if ($res =~ m/.*:\s+(\d+) extents? found/) {
                my $fragment = $1;
                $fragments+=$fragment;
                if ($fragment > 1) {
                        $fragfiles++;
                }
                $files++;

        } else {
                print ("$res : not understand for $file.\n");
        }
        close (FRAG);
}
close (FILES);

print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");
To execute it : write it somewere with your beloved(!?) editor, for example in /root/fragck.pl. Change permission :

Code: Select all

chmod u+x /root/fragck.pl
It's ready.

You must be root to execute the script. It takes one argument : the mount point of the fs to analyze. It will report the percentage of fragmented files and the average number of fragment (see explanation below).

It has to scan all regular files of the examinated partition, so this script is quite slow (depends on the number of files). It takes about 5 mins on my root partition /.

Some examples from my personnal computer:

Code: Select all

/root/fragck.pl /
5.10010105507148% non contiguous files, 1.11087084031179 average fragments.
/root/fragck.pl /divers
79.5620437956204% non contiguous files, 7060.22627737226 average fragments.
Ideally the script should report "0% non contiguous files, 1 average fragments.". The second fs is highly fragmented... "7060 average fragments" means that all files on the fs are splited into 7060 pieces (in average)...

I hope this script will help some people :D

It is possible that you will have problem with file that contains special characters. Please report it, I will try to correct the problem.

Enjoy !

PS : the initial thread where I post the script is http://forums.gentoo.org/viewtopic-t-42 ... art-0.html.
PPS : English is not my motherlanguage, feel free to correct me...
PPPS : I put no licence on the code, you are free to do what you want with it. But I'm interested in any enhancement :D
Top
dhave
Apprentice
Apprentice
User avatar
Posts: 298
Joined: Fri Oct 28, 2005 4:03 pm
Location: Still outside the Matrix ...

  • Quote

Post by dhave » Fri Feb 03, 2006 6:20 pm

Thanks, _droop_. I've tested your script on all the directories on my reiserfs partition, and it worked as advertised. It's a very useful tool. I posted a link to this thread on www.linuxquestions.org, by the way.
This space available!
Top
mdeininger
Veteran
Veteran
User avatar
Posts: 1740
Joined: Wed Jun 15, 2005 11:45 am
Location: Emerald Isles, observing Dublin's docklands
Contact:
Contact mdeininger
Website

  • Quote

Post by mdeininger » Fri Feb 03, 2006 8:20 pm

very useful script, I will give it a shot tomorrow
:thumbs up:
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Top
batistuta
Veteran
Veteran
User avatar
Posts: 1384
Joined: Fri Jul 29, 2005 3:54 pm
Location: Aachen

  • Quote

Post by batistuta » Sat Feb 04, 2006 10:51 am

I've been waiting for this for years!! OK, I've started using linux only one year ago :D
Thanks
Top
batistuta
Veteran
Veteran
User avatar
Posts: 1384
Joined: Fri Jul 29, 2005 3:54 pm
Location: Aachen

  • Quote

Post by batistuta » Sat Feb 04, 2006 11:31 am

could you give us some metrics of what the output means? I.e., when should we defragment? I've found for example that my portage tree rebuild was MUCH faster fater copying back and forth the portage tree. How would such a badly defragmented partition show up?

It would be cool if there was some cron job or something that would monitor partitions for fragmentation and do this automatically. What seems to you like a sensible threashold for defragmenting (if there is such a thing)?
Top
fangorn
Veteran
Veteran
User avatar
Posts: 1886
Joined: Sat Jul 31, 2004 1:31 pm
Contact:
Contact fangorn
Website

  • Quote

Post by fangorn » Mon Feb 06, 2006 2:18 pm

Out of curiosity:

Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back :wink:
Video Encoding scripts collection | Project page
Top
dhave
Apprentice
Apprentice
User avatar
Posts: 298
Joined: Fri Oct 28, 2005 4:03 pm
Location: Still outside the Matrix ...

  • Quote

Post by dhave » Mon Feb 06, 2006 4:11 pm

fangorn wrote:Out of curiosity:

Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back :wink:
Well, you could look into Con Kolivas's defrag script, which is supposed to work on any filesystem. I don't think he claims that it's the solution to all your defrag problems, but it should reduce fragmentation some, since it rearranges all your data according to file size.

It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition.
This space available!
Top
fangorn
Veteran
Veteran
User avatar
Posts: 1886
Joined: Sat Jul 31, 2004 1:31 pm
Contact:
Contact fangorn
Website

  • Quote

Post by fangorn » Mon Feb 06, 2006 4:40 pm

Will surely test this before my data partition is entirely "cleaned" :twisted:
Video Encoding scripts collection | Project page
Top
Kateikyoushi
n00b
n00b
Posts: 39
Joined: Fri Jan 06, 2006 11:38 am

  • Quote

Post by Kateikyoushi » Wed Feb 08, 2006 2:18 pm

dhave wrote: It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition.
I let it loose in a half a year old reiserfs which was aroud 30% fragmented, pushed it down to 5%.
This will do till reiser4 hits mainstream.
Top
grenouille
Tux's lil' helper
Tux's lil' helper
User avatar
Posts: 97
Joined: Sat Jun 12, 2004 6:45 pm

  • Quote

Post by grenouille » Wed Feb 08, 2006 7:22 pm

nice! thanks :)
Top
wrc1944
Advocate
Advocate
Posts: 3467
Joined: Thu Aug 15, 2002 10:33 am
Location: Gainesville, Florida

  • Quote

Post by wrc1944 » Thu Feb 09, 2006 8:27 am

I assume this can be run on mounted partitions- correct?
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.41-r2, gcc-15.1.0
kernel-6.15.6 USE=experimental python3.13.3
Top
fangorn
Veteran
Veteran
User avatar
Posts: 1886
Joined: Sat Jul 31, 2004 1:31 pm
Contact:
Contact fangorn
Website

  • Quote

Post by fangorn » Thu Feb 09, 2006 9:37 am

This has to be run on mounted partitions, because it uses shell functions to find and categorize files and move them according to file size. that means for this script to run you will need at least a little bit more than the size of the biggest file on the filesystem as free space. The more continuous space you have free the less fragmented will the first worked on files be.
Video Encoding scripts collection | Project page
Top
gmichels
Guru
Guru
User avatar
Posts: 480
Joined: Fri Jun 20, 2003 8:02 pm
Location: Brazil

  • Quote

Post by gmichels » Thu Feb 09, 2006 12:03 pm

Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:

Code: Select all

./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3
Top
dhave
Apprentice
Apprentice
User avatar
Posts: 298
Joined: Fri Oct 28, 2005 4:03 pm
Location: Still outside the Matrix ...

  • Quote

Post by dhave » Thu Feb 09, 2006 6:40 pm

gmichels wrote:Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:

Code: Select all

./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3
You might want to send _droop_ a pm to let him know. He can fix this easily; I had a similar problem with special characters in file names.
This space available!
Top
dundas
Guru
Guru
User avatar
Posts: 317
Joined: Thu Dec 16, 2004 6:28 am
Location: China, Earth

  • Quote

Post by dundas » Fri Feb 10, 2006 2:04 am

testing, book marked for now, thx partners.
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer
Top
sirtalon42
Tux's lil' helper
Tux's lil' helper
Posts: 79
Joined: Tue Aug 09, 2005 10:04 pm

  • Quote

Post by sirtalon42 » Tue Feb 14, 2006 5:57 am

I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files):

Code: Select all

# ./fragchk.pl /usr/portage
0.121585826566503% non contiguous files, 1.00151982283208 average fragments.
/usr/portage isn't on its own partition, and thats the only thing I can think of that would be throwing off these results.

In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation).


EDIT: Oops for some reason I forgot that an ideal % non-contiguous files is 0% (and not 1%), though I am rather shocked that portage is that little fragmented (though I guess most likely portage itself's files are small enough to fit inbetween other files, but causing other files to fragment...).
Last edited by sirtalon42 on Tue Feb 14, 2006 3:00 pm, edited 1 time in total.
Top
TGL
Bodhisattva
Bodhisattva
Posts: 1978
Joined: Sun Jun 02, 2002 12:13 pm
Location: Rennes, France

  • Quote

Post by TGL » Tue Feb 14, 2006 10:39 am

sirtalon42 wrote:I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files)
Lot of very small (1 block) files, so yes, it's quite possible.
In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation).
Won't be a problem: it will only check your root partition ("find -xdev").
Top
as
n00b
n00b
Posts: 1
Joined: Tue Feb 14, 2006 6:40 pm

  • Quote

Post by as » Tue Feb 14, 2006 6:46 pm

With find -print0, putting \0 to record separator and running filefrag without shell interfering you can lose all filename mangling. A lot safer too.

Code: Select all

#!/usr/bin/perl -w

#this script search for frag on a fs
use strict;

#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;

#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f -print0 |");

$/ = "\0";

while (defined (my $file = <FILES>)) {
        open (FRAG, "-|", "filefrag", $file);
        my $res = <FRAG>;
        if ($res =~ m/.*:\s+(\d+) extents? found/) {
                my $fragment = $1;
                $fragments += $fragment;
                if ($fragment > 1) {
                        $fragfiles++;
                }
                $files++;
        } else {
                print ("$res : not understand for $file.\n");
        }
        close (FRAG);
}
close (FILES);

print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");
Top
revertex
l33t
l33t
User avatar
Posts: 806
Joined: Wed Apr 23, 2003 9:21 am

  • Quote

Post by revertex » Mon Mar 27, 2006 8:25 am

@ as,
Thank's for the fix, works fine here.

WTF,all partition was created 3 months ago, reiserfs3, noatime, notail flags.

Code: Select all

fragck.pl /home
2.5240078184754% non contiguous files, 3.15788221296847 average fragments.
Looks like the guys at nemesis are telling us a big lie about reiserfs fragmentation.
Reiser is only fast in the first week before format, a few weeks later the slowdown becomes irritating.

Code: Select all

fragck.pl /tmp     
34.8745364350046% non contiguous files, 1.89023744160285 average fragments.
After koliva's defrag script

Code: Select all

fragck.pl /tmp 
34.8793526946973% non contiguous files, 1.89038192939363 average fragments.
Look's like it does nothing to reiserfs.

My conclusion, if you reformat once a month then reiserfs should be a good choice.
Top
d4rkside
n00b
n00b
Posts: 5
Joined: Wed Jan 11, 2006 3:48 am

  • Quote

Post by d4rkside » Sat Apr 08, 2006 9:29 pm

Sweet Script for reiserfs users! Thanks

@as what does the script that you posted do differently?

Thanks.
Top
revertex
l33t
l33t
User avatar
Posts: 806
Joined: Wed Apr 23, 2003 9:21 am

  • Quote

Post by revertex » Sat Apr 08, 2006 11:27 pm

d4rkside wrote:Sweet Script for reiserfs users! Thanks

@as what does the script that you posted do differently?

Thanks.
d4rkside,

it handle long filenames with spaces.
Top
dundas
Guru
Guru
User avatar
Posts: 317
Joined: Thu Dec 16, 2004 6:28 am
Location: China, Earth

  • Quote

Post by dundas » Wed Apr 12, 2006 11:22 am

weird, I guess mine looks fine? but, my gentoo is like more than 1 yrs old

Code: Select all

# ./frag.sh /
3.24979630988634% non contiguous files, 1.13087017572245 average fragments.
but thx for the new script
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer
Top
G2k
l33t
l33t
User avatar
Posts: 672
Joined: Sat Mar 06, 2004 5:58 pm
Location: Rome, Italy

  • Quote

Post by G2k » Thu Apr 13, 2006 8:44 am

I didn't know that ReiserFS fragmented :cry: that makes me sad. Is there a file system that doesn't fragment available for Linux now? EXT3? JFS? Or is this something that Reiser4 promises to bring?
Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus
Top
PabOu
Veteran
Veteran
User avatar
Posts: 1088
Joined: Wed Feb 11, 2004 9:45 am
Location: Namur - Belgium
Contact:
Contact PabOu
Website

  • Quote

Post by PabOu » Thu Apr 13, 2006 10:06 am

all FS got fragmentation with time...
I have read somewhere that Reiser4 got an auto-defrag if you leave the computer ON without disk activity...
Mangez du poulet !
Top
G2k
l33t
l33t
User avatar
Posts: 672
Joined: Sat Mar 06, 2004 5:58 pm
Location: Rome, Italy

  • Quote

Post by G2k » Fri Apr 14, 2006 6:52 am

I don't know how much of a difference there is, but using' as's code instead of _droop_'s, these were the results:

Code: Select all

./fragchk.pl /
2.89009086202895% non contiguous files, 1.21397383030757 average fragments.
meh...I guess it holds out better than NTFS
Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus
Top
Post Reply

52 posts
  • 1
  • 2
  • 3
  • Next

Return to “Documentation, Tips & Tricks”

Jump to
  • Assistance
  • ↳   News & Announcements
  • ↳   Frequently Asked Questions
  • ↳   Installing Gentoo
  • ↳   Multimedia
  • ↳   Desktop Environments
  • ↳   Networking & Security
  • ↳   Kernel & Hardware
  • ↳   Portage & Programming
  • ↳   Gamers & Players
  • ↳   Other Things Gentoo
  • ↳   Unsupported Software
  • Discussion & Documentation
  • ↳   Documentation, Tips & Tricks
  • ↳   Gentoo Chat
  • ↳   Gentoo Forums Feedback
  • ↳   Duplicate Threads
  • International Gentoo Users
  • ↳   中文 (Chinese)
  • ↳   Dutch
  • ↳   Finnish
  • ↳   French
  • ↳   Deutsches Forum (German)
  • ↳   Diskussionsforum
  • ↳   Deutsche Dokumentation
  • ↳   Greek
  • ↳   Forum italiano (Italian)
  • ↳   Forum di discussione italiano
  • ↳   Risorse italiane (documentazione e tools)
  • ↳   Polskie forum (Polish)
  • ↳   Instalacja i sprzęt
  • ↳   Polish OTW
  • ↳   Portuguese
  • ↳   Documentação, Ferramentas e Dicas
  • ↳   Russian
  • ↳   Scandinavian
  • ↳   Spanish
  • ↳   Other Languages
  • Architectures & Platforms
  • ↳   Gentoo on ARM
  • ↳   Gentoo on PPC
  • ↳   Gentoo on Sparc
  • ↳   Gentoo on Alternative Architectures
  • ↳   Gentoo on AMD64
  • ↳   Gentoo for Mac OS X (Portage for Mac OS X)
  • Board index
  • All times are UTC
  • Delete cookies

© 2001–2026 Gentoo Foundation, Inc.

Powered by phpBB® Forum Software © phpBB Limited

Privacy Policy

 

 

magic