| View previous topic :: View next topic |
| Author |
Message |
_droop_
l33t
Joined: 30 May 2004
Posts: 957
|
 Posted: Fri Feb 03, 2006 6:10 pm Post subject: Script:Measuring fragmentation on Reiserfs (and other fs) |
 |
|
Hi,
Reiserfs is one fs which has no tools to measure fragmentation.
I asked myself how can we measure fragmentation on a reiserfs partition. I wrote a script that is capable to measure this fragmentation.
This script requires e2fsprogs package (for filefrag command), but this package is installed on almost all gentoo. It is written in perl.
Since filefrag works on various fs type (including reiserfs), my script should work too on various fs type.
The script :
| Code: | #!/usr/bin/perl -w
#this script search for frag on a fs
use strict;
#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;
#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f |");
while (defined (my $file = <FILES>)) {
#quote some chars in filename
$file =~ s/!/\\!/g;
$file =~ s/#/\\#/g;
$file =~ s/&/\\&/g;
$file =~ s/>/\\>/g;
$file =~ s/</\\</g;
$file =~ s/\$/\\\$/g;
$file =~ s/\(/\\\(/g;
$file =~ s/\)/\\\)/g;
$file =~ s/\|/\\\|/g;
$file =~ s/'/\\'/g;
$file =~ s/ /\\ /g;
#nb of fragment for the file
open (FRAG, "filefrag $file |");
my $res = <FRAG>;
if ($res =~ m/.*:\s+(\d+) extents? found/) {
my $fragment = $1;
$fragments+=$fragment;
if ($fragment > 1) {
$fragfiles++;
}
$files++;
} else {
print ("$res : not understand for $file.\n");
}
close (FRAG);
}
close (FILES);
print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n"); |
To execute it : write it somewere with your beloved(!?) editor, for example in /root/fragck.pl. Change permission :
| Code: | | chmod u+x /root/fragck.pl |
It's ready.
You must be root to execute the script. It takes one argument : the mount point of the fs to analyze. It will report the percentage of fragmented files and the average number of fragment (see explanation below).
It has to scan all regular files of the examinated partition, so this script is quite slow (depends on the number of files). It takes about 5 mins on my root partition /.
Some examples from my personnal computer:
| Code: |
/root/fragck.pl /
5.10010105507148% non contiguous files, 1.11087084031179 average fragments.
/root/fragck.pl /divers
79.5620437956204% non contiguous files, 7060.22627737226 average fragments.
|
Ideally the script should report "0% non contiguous files, 1 average fragments.". The second fs is highly fragmented... "7060 average fragments" means that all files on the fs are splited into 7060 pieces (in average)...
I hope this script will help some people 
It is possible that you will have problem with file that contains special characters. Please report it, I will try to correct the problem.
Enjoy !
PS : the initial thread where I post the script is https://forums.gentoo.org/viewtopic-t-429134-start-0.html.
PPS : English is not my motherlanguage, feel free to correct me...
PPPS : I put no licence on the code, you are free to do what you want with it. But I'm interested in any enhancement |
|
| Back to top |
|
 |
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
| Posted: Fri Feb 03, 2006 6:20 pm Post subject: |
|
|
Thanks, _droop_. I've tested your script on all the directories on my reiserfs partition, and it worked as advertised. It's a very useful tool. I posted a link to this thread on www.linuxquestions.org, by the way.
_________________
This space available! |
|
| Back to top |
|
|
mdeininger
Veteran
Joined: 15 Jun 2005
Posts: 1740
Location: Emerald Isles, observing Dublin's docklands
|
| Posted: Fri Feb 03, 2006 8:20 pm Post subject: |
|
|
very useful script, I will give it a shot tomorrow
:thumbs up:
_________________
"Confident, lazy, cocky, dead." -- Felix Jongleur, Otherland
( Twitter | Blog | GitHub ) |
|
| Back to top |
|
|
batistuta
Veteran
Joined: 29 Jul 2005
Posts: 1384
Location: Aachen
|
| Posted: Sat Feb 04, 2006 10:51 am Post subject: |
|
|
I've been waiting for this for years!! OK, I've started using linux only one year ago
Thanks |
|
| Back to top |
|
|
batistuta
Veteran
Joined: 29 Jul 2005
Posts: 1384
Location: Aachen
|
| Posted: Sat Feb 04, 2006 11:31 am Post subject: |
|
|
could you give us some metrics of what the output means? I.e., when should we defragment? I've found for example that my portage tree rebuild was MUCH faster fater copying back and forth the portage tree. How would such a badly defragmented partition show up?
It would be cool if there was some cron job or something that would monitor partitions for fragmentation and do this automatically. What seems to you like a sensible threashold for defragmenting (if there is such a thing)? |
|
| Back to top |
|
|
fangorn
Veteran
Joined: 31 Jul 2004
Posts: 1886
|
| Posted: Mon Feb 06, 2006 2:18 pm Post subject: |
|
|
Out of curiosity:
Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back 
_________________
Video Encoding scripts collection | Project page |
|
| Back to top |
|
|
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
| Posted: Mon Feb 06, 2006 4:11 pm Post subject: |
|
|
| fangorn wrote: | Out of curiosity:
Is there a actual application/script which can defragment a reiserfs partition? I just dont happen to have another 250 GB at hand to move the files and move them back |
Well, you could look into Con Kolivas's defrag script, which is supposed to work on any filesystem. I don't think he claims that it's the solution to all your defrag problems, but it should reduce fragmentation some, since it rearranges all your data according to file size.
It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition.
_________________
This space available! |
|
| Back to top |
|
|
fangorn
Veteran
Joined: 31 Jul 2004
Posts: 1886
|
|
| Back to top |
|
|
Kateikyoushi
n00b
Joined: 06 Jan 2006
Posts: 39
|
| Posted: Wed Feb 08, 2006 2:18 pm Post subject: |
|
|
| dhave wrote: |
It worked well for me, though I used it on just a few select directories. You should probably try it on a few smallish directories before turning it loose on an entire partition. |
I let it loose in a half a year old reiserfs which was aroud 30% fragmented, pushed it down to 5%.
This will do till reiser4 hits mainstream. |
|
| Back to top |
|
|
grenouille
Tux's lil' helper
Joined: 12 Jun 2004
Posts: 97
|
| Posted: Wed Feb 08, 2006 7:22 pm Post subject: |
|
|
nice! thanks  |
|
| Back to top |
|
|
wrc1944
Advocate
Joined: 15 Aug 2002
Posts: 3408
Location: Gainesville, Florida
|
| Posted: Thu Feb 09, 2006 8:27 am Post subject: |
|
|
I assume this can be run on mounted partitions- correct?
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.36-r7, gcc-12.2.1_p20230304
kernel-6.2.2 USE=experimental python3_11 |
|
| Back to top |
|
|
fangorn
Veteran
Joined: 31 Jul 2004
Posts: 1886
|
| Posted: Thu Feb 09, 2006 9:37 am Post subject: |
|
|
This has to be run on mounted partitions, because it uses shell functions to find and categorize files and move them according to file size. that means for this script to run you will need at least a little bit more than the size of the biggest file on the filesystem as free space. The more continuous space you have free the less fragmented will the first worked on files be.
_________________
Video Encoding scripts collection | Project page |
|
| Back to top |
|
|
gmichels
Guru
Joined: 20 Jun 2003
Posts: 480
Location: Brazil
|
| Posted: Thu Feb 09, 2006 12:03 pm Post subject: |
|
|
Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:
| Code: | ./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3 |
|
|
| Back to top |
|
|
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
| Posted: Thu Feb 09, 2006 6:40 pm Post subject: |
|
|
| gmichels wrote: | Nice script, I just found a little bug when a filename contains weird chars such as a double quote mark (") or a parentheses:
| Code: | ./fragck.pl /home
statfs: No such file or directory
sh: -).jpg: command not found
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Fotos/Pessoal/Celular/Cacio\ ;-\).jpg
.
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at ./fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at ./fragck.pl line 40.
: not understand for /home/gmichels/Documentos/Pessoal/Músicas/U2/War/10\ -\ "40".mp3 |
|
You might want to send _droop_ a pm to let him know. He can fix this easily; I had a similar problem with special characters in file names.
_________________
This space available! |
|
| Back to top |
|
|
dundas
Guru
Joined: 16 Dec 2004
Posts: 317
Location: China, Earth
|
| Posted: Fri Feb 10, 2006 2:04 am Post subject: |
|
|
testing, book marked for now, thx partners.
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer |
|
| Back to top |
|
|
sirtalon42
Tux's lil' helper
Joined: 09 Aug 2005
Posts: 79
|
| Posted: Tue Feb 14, 2006 5:57 am Post subject: |
|
|
I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files):
| Code: | # ./fragchk.pl /usr/portage
0.121585826566503% non contiguous files, 1.00151982283208 average fragments. |
/usr/portage isn't on its own partition, and thats the only thing I can think of that would be throwing off these results.
In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation).
EDIT: Oops for some reason I forgot that an ideal % non-contiguous files is 0% (and not 1%), though I am rather shocked that portage is that little fragmented (though I guess most likely portage itself's files are small enough to fit inbetween other files, but causing other files to fragment...).
Last edited by sirtalon42 on Tue Feb 14, 2006 3:00 pm; edited 1 time in total |
|
| Back to top |
|
|
TGL
Bodhisattva
Joined: 02 Jun 2002
Posts: 1978
Location: Rennes, France
|
| Posted: Tue Feb 14, 2006 10:39 am Post subject: |
|
|
| sirtalon42 wrote: | | I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files) |
Lot of very small (1 block) files, so yes, it's quite possible.
| Quote: | | In the morning I'm gonna try running it on '/', but I'm guessing stuff like /proc and /dev will mess up the results some (lowering the average fragmentation). |
Won't be a problem: it will only check your root partition ("find -xdev"). |
|
| Back to top |
|
|
as
n00b
Joined: 14 Feb 2006
Posts: 1
|
| Posted: Tue Feb 14, 2006 6:46 pm Post subject: |
|
|
With find -print0, putting \0 to record separator and running filefrag without shell interfering you can lose all filename mangling. A lot safer too.
| Code: |
#!/usr/bin/perl -w
#this script search for frag on a fs
use strict;
#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;
#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f -print0 |");
$/ = "\0";
while (defined (my $file = <FILES>)) {
open (FRAG, "-|", "filefrag", $file);
my $res = <FRAG>;
if ($res =~ m/.*:\s+(\d+) extents? found/) {
my $fragment = $1;
$fragments += $fragment;
if ($fragment > 1) {
$fragfiles++;
}
$files++;
} else {
print ("$res : not understand for $file.\n");
}
close (FRAG);
}
close (FILES);
print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");
|
|
|
| Back to top |
|
|
revertex
l33t
Joined: 23 Apr 2003
Posts: 806
|
| Posted: Mon Mar 27, 2006 8:25 am Post subject: |
|
|
@ as,
Thank's for the fix, works fine here.
WTF,all partition was created 3 months ago, reiserfs3, noatime, notail flags.
| Code: | fragck.pl /home
2.5240078184754% non contiguous files, 3.15788221296847 average fragments. |
Looks like the guys at nemesis are telling us a big lie about reiserfs fragmentation.
Reiser is only fast in the first week before format, a few weeks later the slowdown becomes irritating.
| Code: | fragck.pl /tmp
34.8745364350046% non contiguous files, 1.89023744160285 average fragments. |
After koliva's defrag script
| Code: | fragck.pl /tmp
34.8793526946973% non contiguous files, 1.89038192939363 average fragments. |
Look's like it does nothing to reiserfs.
My conclusion, if you reformat once a month then reiserfs should be a good choice. |
|
| Back to top |
|
|
d4rkside
n00b
Joined: 11 Jan 2006
Posts: 5
|
| Posted: Sat Apr 08, 2006 9:29 pm Post subject: |
|
|
Sweet Script for reiserfs users! Thanks
@as what does the script that you posted do differently?
Thanks. |
|
| Back to top |
|
|
revertex
l33t
Joined: 23 Apr 2003
Posts: 806
|
| Posted: Sat Apr 08, 2006 11:27 pm Post subject: |
|
|
| d4rkside wrote: | Sweet Script for reiserfs users! Thanks
@as what does the script that you posted do differently?
Thanks. |
d4rkside,
it handle long filenames with spaces. |
|
| Back to top |
|
|
dundas
Guru
Joined: 16 Dec 2004
Posts: 317
Location: China, Earth
|
| Posted: Wed Apr 12, 2006 11:22 am Post subject: |
|
|
weird, I guess mine looks fine? but, my gentoo is like more than 1 yrs old
| Code: | # ./frag.sh /
3.24979630988634% non contiguous files, 1.13087017572245 average fragments. |
but thx for the new script
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer |
|
| Back to top |
|
|
G2k
l33t
Joined: 06 Mar 2004
Posts: 672
Location: Rome, Italy
|
| Posted: Thu Apr 13, 2006 8:44 am Post subject: |
|
|
I didn't know that ReiserFS fragmented  that makes me sad. Is there a file system that doesn't fragment available for Linux now? EXT3? JFS? Or is this something that Reiser4 promises to bring?
_________________
Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus |
|
| Back to top |
|
|
PabOu
Veteran
Joined: 11 Feb 2004
Posts: 1088
Location: Hélécine - Belgium
|
| Posted: Thu Apr 13, 2006 10:06 am Post subject: |
|
|
all FS got fragmentation with time...
I have read somewhere that Reiser4 got an auto-defrag if you leave the computer ON without disk activity...
_________________
Mangez du poulet ! |
|
| Back to top |
|
|
G2k
l33t
Joined: 06 Mar 2004
Posts: 672
Location: Rome, Italy
|
| Posted: Fri Apr 14, 2006 6:52 am Post subject: |
|
|
I don't know how much of a difference there is, but using' as's code instead of _droop_'s, these were the results:
| Code: | ./fragchk.pl /
2.89009086202895% non contiguous files, 1.21397383030757 average fragments. |
meh...I guess it holds out better than NTFS
_________________
Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
