View previous topic :: View next topic |
Author |
Message |
_droop_ l33t

Joined: 30 May 2004 Posts: 957
|
Posted: Fri Feb 03, 2006 6:10 pm Post subject: Script:Measuring fragmentation on Reiserfs (and other fs) |
|
|
Hi,
Reiserfs is one fs which has no tools to measure fragmentation.
I asked myself how can we measure fragmentation on a reiserfs partition. I wrote a script that is capable to measure this fragmentation.
This script requires e2fsprogs package (for filefrag command), but this package is installed on almost all gentoo. It is written in perl.
Since filefrag works on various fs type (including reiserfs), my script should work too on various fs type.
The script :
Code: | #!/usr/bin/perl -w
#this script search for frag on a fs
use strict;
#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;
#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f |");
while (defined (my $file = <FILES>)) {
#quote some chars in filename
$file =~ s/!/\\!/g;
$file =~ s/#/\\#/g;
$file =~ s/&/\\&/g;
$file =~ s/>/\\>/g;
$file =~ s/</\\</g;
$file =~ s/\$/\\\$/g;
$file =~ s/\(/\\\(/g;
$file =~ s/\)/\\\)/g;
$file =~ s/\|/\\\|/g;
$file =~ s/'/\\'/g;
$file =~ s/ /\\ /g;
#nb of fragment for the file
open (FRAG, "filefrag $file |");
my $res = <FRAG>;
if ($res =~ m/.*:\s+(\d+) extents? found/) {
my $fragment = $1;
$fragments+=$fragment;
if ($fragment > 1) {
$fragfiles++;
}
$files++;
} else {
print ("$res : not understand for $file.\n");
}
close (FRAG);
}
close (FILES);
print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n"); |
To execute it : write it somewere with your beloved(!?) editor, for example in /root/fragck.pl. Change permission :
Code: | chmod u+x /root/fragck.pl | It's ready.
You must be root to execute the script. It takes one argument : the mount point of the fs to analyze. It will report the percentage of fragmented files and the average number of fragment (see explanation below).
It has to scan all regular files of the examinated partition, so this script is quite slow (depends on the number of files). It takes about 5 mins on my root partition /.
Some examples from my personnal computer:
Code: |
/root/fragck.pl /
5.10010105507148% non contiguous files, 1.11087084031179 average fragments.
/root/fragck.pl /divers
79.5620437956204% non contiguous files, 7060.22627737226 average fragments.
|
Ideally the script should report "0% non contiguous files, 1 average fragments.". The second fs is highly fragmented... "7060 average fragments" means that all files on the fs are splited into 7060 pieces (in average)...
I hope this script will help some people
It is possible that you will have problem with file that contains special characters. Please report it, I will try to correct the problem.
Enjoy !
PS : the initial thread where I post the script is https://forums.gentoo.org/viewtopic-t-429134-start-0.html.
PPS : English is not my motherlanguage, feel free to correct me...
PPPS : I put no licence on the code, you are free to do what you want with it. But I'm interested in any enhancement  |
|
Back to top |
|
 |
dhave Apprentice


Joined: 28 Oct 2005 Posts: 298 Location: Still outside the Matrix ...
|
Posted: Fri Feb 03, 2006 6:20 pm Post subject: |
|
|
Thanks, _droop_. I've tested your script on all the directories on my reiserfs partition, and it worked as advertised. It's a very useful tool. I posted a link to this thread on www.linuxquestions.org, by the way. _________________ This space available! |
|
Back to top |
|
 |
mdeininger Veteran


Joined: 15 Jun 2005 Posts: 1740 Location: Emerald Isles, observing Dublin's docklands
|
Posted: Fri Feb 03, 2006 8:20 pm Post subject: |
|
|
very useful script, I will give it a shot tomorrow
:thumbs up: _________________ "Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland
( Twitter | Blog | GitHub ) |
|
Back to top |
|
 |
batistuta Veteran


Joined: 29 Jul 2005 Posts: 1384 Location: Aachen
|
Posted: Sat Feb 04, 2006 10:51 am Post subject: |
|
|
I've been waiting for this for years!! OK, I've started using linux only one year ago
Thanks |
|
Back to top |
|
 |
batistuta Veteran


Joined: 29 Jul 2005 Posts: 1384 Location: Aachen
|
Posted: Sat Feb 04, 2006 11:31 am Post subject: |
|
|
could you give us some metrics of what the output means? I.e., when should we defragment? I've found for example that my portage tree rebuild was MUCH faster fater copying back and forth the portage tree. How would such a badly defragmented partition show up?
It would be cool if there was some cron job or something that would monitor partitions for fragmentation and do this automatically. What seems to you like a sensible threashold for defragmenting (if there is such a thing)? |
|
Back to top |
|
 |
fangorn Veteran


Joined: 31 Jul 2004 Posts: 1886
|
Posted: Mon Feb 06, 2006 2:18 pm Post subject: |
|
|
Out of curiosity:
Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back  _________________ Video Encoding scripts collection | Project page |
|
Back to top |
|
 |
dhave Apprentice


Joined: 28 Oct 2005 Posts: 298 Location: Still outside the Matrix ...
|
Posted: Mon Feb 06, 2006 4:11 pm Post subject: |
|
|
fangorn wrote: | Out of curiosity:
Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back  |
Well, you could look into Con Kolivas's defrag script, which is supposed to work on any filesystem. I don't think he claims that it's the solution to all your defrag problems, but it should reduce fragmentation some, since it rearranges all your data according to file size.
It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition. _________________ This space available! |
|
Back to top |
|
 |
fangorn Veteran


Joined: 31 Jul 2004 Posts: 1886
|
|
Back to top |
|
 |
Kateikyoushi n00b

Joined: 06 Jan 2006 Posts: 39
|
Posted: Wed Feb 08, 2006 2:18 pm Post subject: |
|
|
dhave wrote: |
It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition. |
I let it loose in a half a year old reiserfs which was aroud 30% fragmented, pushed it down to 5%.
This will do till reiser4 hits mainstream. |
|
Back to top |
|
 |
grenouille Tux's lil' helper


Joined: 12 Jun 2004 Posts: 97
|
Posted: Wed Feb 08, 2006 7:22 pm Post subject: |
|
|
nice! thanks  |
|
Back to top |
|
 |
wrc1944 Advocate

Joined: 15 Aug 2002 Posts: 3408 Location: Gainesville, Florida
|
Posted: Thu Feb 09, 2006 8:27 am Post subject: |
|
|
I assume this can be run on mounted partitions- correct? _________________ Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.36-r7, gcc-12.2.1_p20230304
kernel-6.2.2 USE=experimental python3_11 |
|
Back to top |
|
 |
fangorn Veteran


Joined: 31 Jul 2004 Posts: 1886
|
Posted: Thu Feb 09, 2006 9:37 am Post subject: |
|
|
This has to be run on mounted partitions, because it uses shell functions to find and categorize files and move them according to file size. that means for this script to run you will need at least a little bit more than the size of the biggest file on the filesystem as free space. The more continuous space you have free the less fragmented will the first worked on files be. _________________ Video Encoding scripts collection | Project page |
|
Back to top |
|
 |
gmichels Guru


Joined: 20 Jun 2003 Posts: 480 Location: Brazil
|
Posted: Thu Feb 09, 2006 12:03 pm Post subject: |
|
|
Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:
Code: | ./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3 |
|
|
Back to top |
|
 |
dhave Apprentice


Joined: 28 Oct 2005 Posts: 298 Location: Still outside the Matrix ...
|
Posted: Thu Feb 09, 2006 6:40 pm Post subject: |
|
|
gmichels wrote: | Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:
Code: | ./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3 |
|
You might want to send _droop_ a pm to let him know. He can fix this easily; I had a similar problem with special characters in file names. _________________ This space available! |
|
Back to top |
|
 |
dundas Guru


Joined: 16 Dec 2004 Posts: 317 Location: China, Earth
|
Posted: Fri Feb 10, 2006 2:04 am Post subject: |
|
|
testing, book marked for now, thx partners. _________________ Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer |
|
Back to top |
|
 |
sirtalon42 Tux's lil' helper

Joined: 09 Aug 2005 Posts: 79
|
Posted: Tue Feb 14, 2006 5:57 am Post subject: |
|
|
I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files):
Code: | # ./fragchk.pl /usr/portage
0.121585826566503% non contiguous files, 1.00151982283208 average fragments. |
/usr/portage isn't on its own partition, and thats the only thing I can think of that would be throwing off these results.
In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation).
EDIT: Oops for some reason I forgot that an ideal % non-contiguous files is 0% (and not 1%), though I am rather shocked that portage is that little fragmented (though I guess most likely portage itself's files are small enough to fit inbetween other files, but causing other files to fragment...).
Last edited by sirtalon42 on Tue Feb 14, 2006 3:00 pm; edited 1 time in total |
|
Back to top |
|
 |
TGL Bodhisattva

Joined: 02 Jun 2002 Posts: 1978 Location: Rennes, France
|
Posted: Tue Feb 14, 2006 10:39 am Post subject: |
|
|
sirtalon42 wrote: | I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files) |
Lot of very small (1 block) files, so yes, it's quite possible.
Quote: | In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation). |
Won't be a problem: it will only check your root partition ("find -xdev"). |
|
Back to top |
|
 |
as n00b

Joined: 14 Feb 2006 Posts: 1
|
Posted: Tue Feb 14, 2006 6:46 pm Post subject: |
|
|
With find -print0, putting \0 to record separator and running filefrag without shell interfering you can lose all filename mangling. A lot safer too.
Code: |
#!/usr/bin/perl -w
#this script search for frag on a fs
use strict;
#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;
#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f -print0 |");
$/ = "\0";
while (defined (my $file = <FILES>)) {
open (FRAG, "-|", "filefrag", $file);
my $res = <FRAG>;
if ($res =~ m/.*:\s+(\d+) extents? found/) {
my $fragment = $1;
$fragments += $fragment;
if ($fragment > 1) {
$fragfiles++;
}
$files++;
} else {
print ("$res : not understand for $file.\n");
}
close (FRAG);
}
close (FILES);
print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");
|
|
|
Back to top |
|
 |
revertex l33t


Joined: 23 Apr 2003 Posts: 806
|
Posted: Mon Mar 27, 2006 8:25 am Post subject: |
|
|
@ as,
Thank's for the fix, works fine here.
WTF,all partition was created 3 months ago, reiserfs3, noatime, notail flags.
Code: | fragck.pl /home
2.5240078184754% non contiguous files, 3.15788221296847 average fragments. |
Looks like the guys at nemesis are telling us a big lie about reiserfs fragmentation.
Reiser is only fast in the first week before format, a few weeks later the slowdown becomes irritating.
Code: | fragck.pl /tmp
34.8745364350046% non contiguous files, 1.89023744160285 average fragments. |
After koliva's defrag script
Code: | fragck.pl /tmp
34.8793526946973% non contiguous files, 1.89038192939363 average fragments. |
Look's like it does nothing to reiserfs.
My conclusion, if you reformat once a month then reiserfs should be a good choice. |
|
Back to top |
|
 |
d4rkside n00b

Joined: 11 Jan 2006 Posts: 5
|
Posted: Sat Apr 08, 2006 9:29 pm Post subject: |
|
|
Sweet Script for reiserfs users! Thanks
@as what does the script that you posted do differently?
Thanks. |
|
Back to top |
|
 |
revertex l33t


Joined: 23 Apr 2003 Posts: 806
|
Posted: Sat Apr 08, 2006 11:27 pm Post subject: |
|
|
d4rkside wrote: | Sweet Script for reiserfs users! Thanks
@as what does the script that you posted do differently?
Thanks. |
d4rkside,
it handle long filenames with spaces. |
|
Back to top |
|
 |
dundas Guru


Joined: 16 Dec 2004 Posts: 317 Location: China, Earth
|
Posted: Wed Apr 12, 2006 11:22 am Post subject: |
|
|
weird, I guess mine looks fine? but, my gentoo is like more than 1 yrs old
Code: | # ./frag.sh /
3.24979630988634% non contiguous files, 1.13087017572245 average fragments. |
but thx for the new script _________________ Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer |
|
Back to top |
|
 |
G2k l33t


Joined: 06 Mar 2004 Posts: 672 Location: Rome, Italy
|
Posted: Thu Apr 13, 2006 8:44 am Post subject: |
|
|
I didn't know that ReiserFS fragmented that makes me sad. Is there a file system that doesn't fragment available for Linux now? EXT3? JFS? Or is this something that Reiser4 promises to bring? _________________ Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus |
|
Back to top |
|
 |
PabOu Veteran


Joined: 11 Feb 2004 Posts: 1088 Location: Hélécine - Belgium
|
Posted: Thu Apr 13, 2006 10:06 am Post subject: |
|
|
all FS got fragmentation with time...
I have read somewhere that Reiser4 got an auto-defrag if you leave the computer ON without disk activity... _________________ Mangez du poulet ! |
|
Back to top |
|
 |
G2k l33t


Joined: 06 Mar 2004 Posts: 672 Location: Rome, Italy
|
Posted: Fri Apr 14, 2006 6:52 am Post subject: |
|
|
I don't know how much of a difference there is, but using' as's code instead of _droop_'s, these were the results:
Code: | ./fragchk.pl /
2.89009086202895% non contiguous files, 1.21397383030757 average fragments. | meh...I guess it holds out better than NTFS _________________ Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus |
|
Back to top |
|
 |
|