Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Script:Measuring fragmentation on Reiserfs (and other fs)
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
_droop_
l33t
l33t


Joined: 30 May 2004
Posts: 957

PostPosted: Fri Feb 03, 2006 6:10 pm    Post subject: Script:Measuring fragmentation on Reiserfs (and other fs) Reply with quote

Hi,

Reiserfs is one fs which has no tools to measure fragmentation.

I asked myself how can we measure fragmentation on a reiserfs partition. I wrote a script that is capable to measure this fragmentation.

This script requires e2fsprogs package (for filefrag command), but this package is installed on almost all gentoo. It is written in perl.

Since filefrag works on various fs type (including reiserfs), my script should work too on various fs type.

The script :
Code:
#!/usr/bin/perl -w

#this script search for frag on a fs
use strict;

#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;

#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f |");

while (defined (my $file = <FILES>)) {
        #quote some chars in filename
        $file =~ s/!/\\!/g;
        $file =~ s/#/\\#/g;
        $file =~ s/&/\\&/g;
        $file =~ s/>/\\>/g;
        $file =~ s/</\\</g;
        $file =~ s/\$/\\\$/g;
        $file =~ s/\(/\\\(/g;
        $file =~ s/\)/\\\)/g;
        $file =~ s/\|/\\\|/g;
        $file =~ s/'/\\'/g;
        $file =~ s/ /\\ /g;
        #nb of fragment for the file
        open (FRAG, "filefrag $file |");
        my $res = <FRAG>;
        if ($res =~ m/.*:\s+(\d+) extents? found/) {
                my $fragment = $1;
                $fragments+=$fragment;
                if ($fragment > 1) {
                        $fragfiles++;
                }
                $files++;

        } else {
                print ("$res : not understand for $file.\n");
        }
        close (FRAG);
}
close (FILES);

print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");


To execute it : write it somewere with your beloved(!?) editor, for example in /root/fragck.pl. Change permission :
Code:
chmod u+x /root/fragck.pl
It's ready.

You must be root to execute the script. It takes one argument : the mount point of the fs to analyze. It will report the percentage of fragmented files and the average number of fragment (see explanation below).

It has to scan all regular files of the examinated partition, so this script is quite slow (depends on the number of files). It takes about 5 mins on my root partition /.

Some examples from my personnal computer:
Code:

/root/fragck.pl /
5.10010105507148% non contiguous files, 1.11087084031179 average fragments.
/root/fragck.pl /divers
79.5620437956204% non contiguous files, 7060.22627737226 average fragments.


Ideally the script should report "0% non contiguous files, 1 average fragments.". The second fs is highly fragmented... "7060 average fragments" means that all files on the fs are splited into 7060 pieces (in average)...

I hope this script will help some people :D

It is possible that you will have problem with file that contains special characters. Please report it, I will try to correct the problem.

Enjoy !

PS : the initial thread where I post the script is http://forums.gentoo.org/viewtopic-t-429134-start-0.html.
PPS : English is not my motherlanguage, feel free to correct me...
PPPS : I put no licence on the code, you are free to do what you want with it. But I'm interested in any enhancement :D
Back to top
View user's profile Send private message
dhave
Apprentice
Apprentice


Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...

PostPosted: Fri Feb 03, 2006 6:20 pm    Post subject: Reply with quote

Thanks, _droop_. I've tested your script on all the directories on my reiserfs partition, and it worked as advertised. It's a very useful tool. I posted a link to this thread on www.linuxquestions.org, by the way.
_________________
This space available!
Back to top
View user's profile Send private message
mdeininger
Veteran
Veteran


Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, overlooking Dublin's docklands

PostPosted: Fri Feb 03, 2006 8:20 pm    Post subject: Reply with quote

very useful script, I will give it a shot tomorrow
:thumbs up:
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland

( Twitter | Blog | GitHub )
Back to top
View user's profile Send private message
batistuta
Veteran
Veteran


Joined: 29 Jul 2005
Posts: 1384
Location: Aachen

PostPosted: Sat Feb 04, 2006 10:51 am    Post subject: Reply with quote

I've been waiting for this for years!! OK, I've started using linux only one year ago :D
Thanks
Back to top
View user's profile Send private message
batistuta
Veteran
Veteran


Joined: 29 Jul 2005
Posts: 1384
Location: Aachen

PostPosted: Sat Feb 04, 2006 11:31 am    Post subject: Reply with quote

could you give us some metrics of what the output means? I.e., when should we defragment? I've found for example that my portage tree rebuild was MUCH faster fater copying back and forth the portage tree. How would such a badly defragmented partition show up?

It would be cool if there was some cron job or something that would monitor partitions for fragmentation and do this automatically. What seems to you like a sensible threashold for defragmenting (if there is such a thing)?
Back to top
View user's profile Send private message
fangorn
Veteran
Veteran


Joined: 31 Jul 2004
Posts: 1886

PostPosted: Mon Feb 06, 2006 2:18 pm    Post subject: Reply with quote

Out of curiosity:

Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back :wink:
_________________
Video Encoding scripts collection | Project page
Back to top
View user's profile Send private message
dhave
Apprentice
Apprentice


Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...

PostPosted: Mon Feb 06, 2006 4:11 pm    Post subject: Reply with quote

fangorn wrote:
Out of curiosity:

Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back :wink:

Well, you could look into Con Kolivas's defrag script, which is supposed to work on any filesystem. I don't think he claims that it's the solution to all your defrag problems, but it should reduce fragmentation some, since it rearranges all your data according to file size.

It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition.
_________________
This space available!
Back to top
View user's profile Send private message
fangorn
Veteran
Veteran


Joined: 31 Jul 2004
Posts: 1886

PostPosted: Mon Feb 06, 2006 4:40 pm    Post subject: Reply with quote

Will surely test this before my data partition is entirely "cleaned" :twisted:
_________________
Video Encoding scripts collection | Project page
Back to top
View user's profile Send private message
Kateikyoushi
n00b
n00b


Joined: 06 Jan 2006
Posts: 39

PostPosted: Wed Feb 08, 2006 2:18 pm    Post subject: Reply with quote

dhave wrote:

It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition.


I let it loose in a half a year old reiserfs which was aroud 30% fragmented, pushed it down to 5%.
This will do till reiser4 hits mainstream.
Back to top
View user's profile Send private message
grenouille
Tux's lil' helper
Tux's lil' helper


Joined: 12 Jun 2004
Posts: 97

PostPosted: Wed Feb 08, 2006 7:22 pm    Post subject: Reply with quote

nice! thanks :)
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 2760
Location: Gainesville, Florida

PostPosted: Thu Feb 09, 2006 8:27 am    Post subject: Reply with quote

I assume this can be run on mounted partitions- correct?
_________________
Main box- Gigabyte GIGABYTE GA-990FXA-UD3 AM3+ rev.-4.0
Amd FX 8320, 3.5 GHz, 16GB GSkill DDR3 1866mhz
Samsung SATA 1000GB, Radeon HD 6570 2GB DDR3
Gentoo ~x86, ~amd64, glibc-2.19, gcc-4.9.0, kernel 3.15.5-gentoo (USE=experimental "native")
Back to top
View user's profile Send private message
fangorn
Veteran
Veteran


Joined: 31 Jul 2004
Posts: 1886

PostPosted: Thu Feb 09, 2006 9:37 am    Post subject: Reply with quote

This has to be run on mounted partitions, because it uses shell functions to find and categorize files and move them according to file size. that means for this script to run you will need at least a little bit more than the size of the biggest file on the filesystem as free space. The more continuous space you have free the less fragmented will the first worked on files be.
_________________
Video Encoding scripts collection | Project page
Back to top
View user's profile Send private message
gmichels
Guru
Guru


Joined: 20 Jun 2003
Posts: 480
Location: Brazil

PostPosted: Thu Feb 09, 2006 12:03 pm    Post subject: Reply with quote

Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:
Code:
./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3
Back to top
View user's profile Send private message
dhave
Apprentice
Apprentice


Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...

PostPosted: Thu Feb 09, 2006 6:40 pm    Post subject: Reply with quote

gmichels wrote:
Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:
Code:
./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
 : not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3

You might want to send _droop_ a pm to let him know. He can fix this easily; I had a similar problem with special characters in file names.
_________________
This space available!
Back to top
View user's profile Send private message
dundas
Guru
Guru


Joined: 16 Dec 2004
Posts: 317
Location: China, Earth

PostPosted: Fri Feb 10, 2006 2:04 am    Post subject: Reply with quote

testing, book marked for now, thx partners.
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer
Back to top
View user's profile Send private message
sirtalon42
Tux's lil' helper
Tux's lil' helper


Joined: 09 Aug 2005
Posts: 79

PostPosted: Tue Feb 14, 2006 5:57 am    Post subject: Reply with quote

I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files):
Code:
# ./fragchk.pl /usr/portage
0.121585826566503% non contiguous files, 1.00151982283208 average fragments.


/usr/portage isn't on its own partition, and thats the only thing I can think of that would be throwing off these results.

In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation).


EDIT: Oops for some reason I forgot that an ideal % non-contiguous files is 0% (and not 1%), though I am rather shocked that portage is that little fragmented (though I guess most likely portage itself's files are small enough to fit inbetween other files, but causing other files to fragment...).


Last edited by sirtalon42 on Tue Feb 14, 2006 3:00 pm; edited 1 time in total
Back to top
View user's profile Send private message
TGL
Bodhisattva
Bodhisattva


Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France

PostPosted: Tue Feb 14, 2006 10:39 am    Post subject: Reply with quote

sirtalon42 wrote:
I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files)

Lot of very small (1 block) files, so yes, it's quite possible.
Quote:
In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation).

Won't be a problem: it will only check your root partition ("find -xdev").
Back to top
View user's profile Send private message
as
n00b
n00b


Joined: 14 Feb 2006
Posts: 1

PostPosted: Tue Feb 14, 2006 6:46 pm    Post subject: Reply with quote

With find -print0, putting \0 to record separator and running filefrag without shell interfering you can lose all filename mangling. A lot safer too.

Code:

#!/usr/bin/perl -w

#this script search for frag on a fs
use strict;

#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;

#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f -print0 |");

$/ = "\0";

while (defined (my $file = <FILES>)) {
        open (FRAG, "-|", "filefrag", $file);
        my $res = <FRAG>;
        if ($res =~ m/.*:\s+(\d+) extents? found/) {
                my $fragment = $1;
                $fragments += $fragment;
                if ($fragment > 1) {
                        $fragfiles++;
                }
                $files++;
        } else {
                print ("$res : not understand for $file.\n");
        }
        close (FRAG);
}
close (FILES);

print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");
Back to top
View user's profile Send private message
revertex
l33t
l33t


Joined: 23 Apr 2003
Posts: 806

PostPosted: Mon Mar 27, 2006 8:25 am    Post subject: Reply with quote

@ as,
Thank's for the fix, works fine here.

WTF,all partition was created 3 months ago, reiserfs3, noatime, notail flags.

Code:
fragck.pl /home
2.5240078184754% non contiguous files, 3.15788221296847 average fragments.


Looks like the guys at nemesis are telling us a big lie about reiserfs fragmentation.
Reiser is only fast in the first week before format, a few weeks later the slowdown becomes irritating.

Code:
fragck.pl /tmp     
34.8745364350046% non contiguous files, 1.89023744160285 average fragments.


After koliva's defrag script
Code:
fragck.pl /tmp
34.8793526946973% non contiguous files, 1.89038192939363 average fragments.


Look's like it does nothing to reiserfs.

My conclusion, if you reformat once a month then reiserfs should be a good choice.
Back to top
View user's profile Send private message
d4rkside
n00b
n00b


Joined: 11 Jan 2006
Posts: 5

PostPosted: Sat Apr 08, 2006 9:29 pm    Post subject: Reply with quote

Sweet Script for reiserfs users! Thanks

@as what does the script that you posted do differently?

Thanks.
Back to top
View user's profile Send private message
revertex
l33t
l33t


Joined: 23 Apr 2003
Posts: 806

PostPosted: Sat Apr 08, 2006 11:27 pm    Post subject: Reply with quote

d4rkside wrote:
Sweet Script for reiserfs users! Thanks

@as what does the script that you posted do differently?

Thanks.


d4rkside,

it handle long filenames with spaces.
Back to top
View user's profile Send private message
dundas
Guru
Guru


Joined: 16 Dec 2004
Posts: 317
Location: China, Earth

PostPosted: Wed Apr 12, 2006 11:22 am    Post subject: Reply with quote

weird, I guess mine looks fine? but, my gentoo is like more than 1 yrs old

Code:
# ./frag.sh /
3.24979630988634% non contiguous files, 1.13087017572245 average fragments.


but thx for the new script
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer
Back to top
View user's profile Send private message
G2k
l33t
l33t


Joined: 06 Mar 2004
Posts: 672
Location: Rome, Italy

PostPosted: Thu Apr 13, 2006 8:44 am    Post subject: Reply with quote

I didn't know that ReiserFS fragmented :cry: that makes me sad. Is there a file system that doesn't fragment available for Linux now? EXT3? JFS? Or is this something that Reiser4 promises to bring?
_________________
Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus
Back to top
View user's profile Send private message
PabOu
Veteran
Veteran


Joined: 11 Feb 2004
Posts: 1073
Location: Hélécine - Belgium

PostPosted: Thu Apr 13, 2006 10:06 am    Post subject: Reply with quote

all FS got fragmentation with time...
I have read somewhere that Reiser4 got an auto-defrag if you leave the computer ON without disk activity...
_________________
Mangez du poulet !
Back to top
View user's profile Send private message
G2k
l33t
l33t


Joined: 06 Mar 2004
Posts: 672
Location: Rome, Italy

PostPosted: Fri Apr 14, 2006 6:52 am    Post subject: Reply with quote

I don't know how much of a difference there is, but using' as's code instead of _droop_'s, these were the results:
Code:
./fragchk.pl /
2.89009086202895% non contiguous files, 1.21397383030757 average fragments.
meh...I guess it holds out better than NTFS
_________________
Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum