| View previous topic :: View next topic |
| Author |
Message |
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
 Posted: Wed Feb 01, 2006 1:30 pm Post subject: How can I measure fragmentation on ReiserFS partition? |
 |
|
For the life of me I can't find a way to do this.
Note that I want to measure the extent of fragmentation on a ReiserFS partition, not defrag the partition.
I believe filefrag will do this on a file-by-file basis, but not on a partition.
Thanks.
_________________
This space available! |
|
| Back to top |
|
 |
Dan Forever
Tux's lil' helper
Joined: 13 Jun 2005
Posts: 79
Location: London, England
|
| Posted: Wed Feb 01, 2006 2:39 pm Post subject: |
|
|
| I've been told that ReiserFS doesn't fragment and that it manages itself on the fly. As to how correct that is, I don't know. |
|
| Back to top |
|
|
BlackEdder
Advocate
Joined: 26 Apr 2004
Posts: 2588
Location: Dutch enclave in Egham, UK
|
| Posted: Wed Feb 01, 2006 2:44 pm Post subject: |
|
|
| AFAIK reiserfs is the filesystem most susceptable to fragmentation. In typical Reiser fashion they fixed this by not showing the fragmentation when doing a check. |
|
| Back to top |
|
|
HydroSan
l33t
Joined: 04 Mar 2004
Posts: 764
Location: The Kremlin (aka Canada)
|
| Posted: Wed Feb 01, 2006 3:33 pm Post subject: |
|
|
| BlackEdder wrote: | | AFAIK reiserfs is the filesystem most susceptable to fragmentation. |
Err, no, it is the LEAST susceptible to fragmentation. Unless of course you put "notail" in /etc/fstab.
_________________
I was a Gangster for Capitalism, by Major General Smedley Butler.
Server status: Currently down, being replaced with fresh install - 20% completed. |
|
| Back to top |
|
|
mystified
l33t
Joined: 14 Feb 2004
Posts: 748
|
| Posted: Wed Feb 01, 2006 3:48 pm Post subject: |
|
|
| If you don't use notail what option should you use? |
|
| Back to top |
|
|
HydroSan
l33t
Joined: 04 Mar 2004
Posts: 764
Location: The Kremlin (aka Canada)
|
| Posted: Wed Feb 01, 2006 3:54 pm Post subject: |
|
|
| mystified wrote: | | If you don't use notail what option should you use? |
It defrags by itself on the fly. If you want some added speed without fragmentation, add "noatime" to /etc/fstab to disable access times, which should speed it up some for large sets of files.
_________________
I was a Gangster for Capitalism, by Major General Smedley Butler.
Server status: Currently down, being replaced with fresh install - 20% completed. |
|
| Back to top |
|
|
BlackEdder
Advocate
Joined: 26 Apr 2004
Posts: 2588
Location: Dutch enclave in Egham, UK
|
| Posted: Wed Feb 01, 2006 4:01 pm Post subject: |
|
|
| HydroSan wrote: | | BlackEdder wrote: | | AFAIK reiserfs is the filesystem most susceptable to fragmentation. |
Err, no, it is the LEAST susceptible to fragmentation. Unless of course you put "notail" in /etc/fstab. |
https://forums.gentoo.org/viewtopic-t-422984-highlight-reiserfs+fragmentation.html
This thread seems to disagree with you and I can't test it because reiserfstools won't tell me the current fragmentation of the harddisk. |
|
| Back to top |
|
|
Enlight
Advocate
Joined: 28 Oct 2004
Posts: 3519
Location: Alsace (France)
|
| Posted: Wed Feb 01, 2006 4:05 pm Post subject: |
|
|
| HydroSan wrote: | | mystified wrote: | | If you don't use notail what option should you use? |
It defrags by itself on the fly. If you want some added speed without fragmentation, add "noatime" to /etc/fstab to disable access times, which should speed it up some for large sets of files. |
With tail, reiser-3.6 has the lowest internal fragmentation, because it is able to put multiple small file in a chunk. But the fact that using notails leads to high fragmentation just proves that external fragmentation is badly handled, leading to file fragmentation! So... BlackHedder++ |
|
| Back to top |
|
|
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
| Posted: Wed Feb 01, 2006 6:43 pm Post subject: |
|
|
So am I to assume that there really is no way to measure file fragmentation on a reiserfs 3.6 partition? I tried to figure out mongo, but I didn't have any success.
Strangely, it seems to me that a couple of times when I had to force a shutdown and subsequently rebooted "dirty", the routine filesystem check (reiserfsck?) produced an error message regarding the percentage of "non-contiguous" data. Was I imagining this?
_________________
This space available! |
|
| Back to top |
|
|
Enlight
Advocate
Joined: 28 Oct 2004
Posts: 3519
Location: Alsace (France)
|
| Posted: Wed Feb 01, 2006 9:43 pm Post subject: |
|
|
| dhave wrote: | So am I to assume that there really is no way to measure file fragmentation on a reiserfs 3.6 partition? I tried to figure out mongo, but I didn't have any success.
Strangely, it seems to me that a couple of times when I had to force a shutdown and subsequently rebooted "dirty", the routine filesystem check (reiserfsck?) produced an error message regarding the percentage of "non-contiguous" data. Was I imagining this? |
Non contiguous should be file fragmentation. |
|
| Back to top |
|
|
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
| Posted: Wed Feb 01, 2006 10:33 pm Post subject: |
|
|
| Enlight wrote: | | dhave wrote: | So am I to assume that there really is no way to measure file fragmentation on a reiserfs 3.6 partition? I tried to figure out mongo, but I didn't have any success.
Strangely, it seems to me that a couple of times when I had to force a shutdown and subsequently rebooted "dirty", the routine filesystem check (reiserfsck?) produced an error message regarding the percentage of "non-contiguous" data. Was I imagining this? |
Non contiguous should be file fragmentation. |
It turns out that that message referred to a small ext2 partition that I have, not my rather large reiserfs partition. So I'm still in the dark about whether or not performance on my reiserfs partition is degrading due to file fragmentation.
_________________
This space available! |
|
| Back to top |
|
|
HydroSan
l33t
Joined: 04 Mar 2004
Posts: 764
Location: The Kremlin (aka Canada)
|
| Posted: Wed Feb 01, 2006 10:43 pm Post subject: |
|
|
| Enlight wrote: | | HydroSan wrote: | | mystified wrote: | | If you don't use notail what option should you use? |
It defrags by itself on the fly. If you want some added speed without fragmentation, add "noatime" to /etc/fstab to disable access times, which should speed it up some for large sets of files. |
With tail, reiser-3.6 has the lowest internal fragmentation, because it is able to put multiple small file in a chunk. But the fact that using notails leads to high fragmentation just proves that external fragmentation is badly handled, leading to file fragmentation! So... BlackHedder++ |
So turning off a the defragmentation routines of a file system causes fragmentation, and therefore that FS is shit.
Is that what you're saying? Because it sounds pretty asinine to me.
_________________
I was a Gangster for Capitalism, by Major General Smedley Butler.
Server status: Currently down, being replaced with fresh install - 20% completed. |
|
| Back to top |
|
|
Enlight
Advocate
Joined: 28 Oct 2004
Posts: 3519
Location: Alsace (France)
|
| Posted: Thu Feb 02, 2006 8:14 am Post subject: |
|
|
| HydroSan wrote: | | Enlight wrote: | | HydroSan wrote: | | mystified wrote: | | If you don't use notail what option should you use? |
It defrags by itself on the fly. If you want some added speed without fragmentation, add "noatime" to /etc/fstab to disable access times, which should speed it up some for large sets of files. |
With tail, reiser-3.6 has the lowest internal fragmentation, because it is able to put multiple small file in a chunk. But the fact that using notails leads to high fragmentation just proves that external fragmentation is badly handled, leading to file fragmentation! So... BlackHedder++ |
So turning off a the defragmentation routines of a file system causes fragmentation, and therefore that FS is shit.
Is that what you're saying? Because it sounds pretty asinine to me. |
 tail is an aggregation technique, not an onthe fly defragmentation routine, so please, give up with donkeys stuffs! |
|
| Back to top |
|
|
_droop_
l33t
Joined: 30 May 2004
Posts: 957
|
| Posted: Thu Feb 02, 2006 10:55 am Post subject: |
|
|
Hi,
I ask myself the same question, and I did not fount any good answer. So I wrote a little perl script based on filefrag (included in e2fsprogs, see man for details).
This script will give you the number of files and the number of fragments for a given file system.
| Code: | #!/usr/bin/perl -w
#this script search for frag on a fs
use strict;
#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;
#search fs for all file
open (FILES, "find " . $ARGV[0] . " -xdev -type f |");
while (defined (my $file = <FILES>)) {
#quote some chars in filename
$file =~ s/!/\\!/g;
$file =~ s/#/\\#/g;
$file =~ s/&/\\&/g;
$file =~ s/>/\\>/g;
$file =~ s/</\\</g;
$file =~ s/\$/\\\$/g;
$file =~ s/\(/\\\(/g;
$file =~ s/\)/\\\)/g;
$file =~ s/\|/\\\|/g;
$file =~ s/'/\\'/g;
$file =~ s/ /\\ /g;
#nb of fragment for the file
open (FRAG, "filefrag $file |");
my $res = <FRAG>;
if ($res =~ m/.*:\s+(\d+) extents? found/) {
my $fragment = $1;
$fragments+=$fragment;
if ($fragment > 1) {
$fragfiles++;
}
$files++;
} else {
print ("$res : not understand for $file.\n");
}
close (FRAG);
}
close (FILES);
print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n"); |
I don't know if the result are corrects and exploitable. But, given that filefrag should report properly on reiserfs, this should be ok...
At work, for my home partition (300mbytes used on 3 gbytes) :
| Code: | ./testfrag.pl /home
5.44639779386419% non contiguous files, 1.27680110306791 average fragmentation. |
Not very good, since this partition is only 10% used.
PS : sorry if my english is too bad...
edit : changed reported information...
Last edited by _droop_ on Fri Feb 03, 2006 2:41 pm; edited 6 times in total |
|
| Back to top |
|
|
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
| Posted: Thu Feb 02, 2006 2:43 pm Post subject: |
|
|
| Dan Forever wrote: | | I've been told that ReiserFS doesn't fragment and that it manages itself on the fly. As to how correct that is, I don't know. |
I've been told that there's a Santa Claus. 
_________________
This space available! |
|
| Back to top |
|
|
feld
Guru
Joined: 29 Aug 2004
Posts: 593
Location: WI, USA
|
| Posted: Thu Feb 02, 2006 2:48 pm Post subject: |
|
|
| Code: |
The ReiserFS internal fragmentation solution
When we discussed the internal fragmentation and file system performance, we said administrators often have to choose between performance and space waste. If we now look at the first table, we would see that new fs are able to manage blocks up to 64KB in size. That size of block and even smaller would produce a significant waste of space due to internal fragmentation. In order to make the use of big block sizes feasible, ReiserFS implements a technique that solves the problem.
As we said earlier, ReiserFS uses a B*Tree to organise the file system objects. These objects are the structures used to maintain file information access time, file permissions, etc. In other words, the information contained within an i-node-, directories and the file's data. ReiserFS calls those objects, stat data items, directory items and direct/indirect items, respectively. The indirect items consist of pointers to unformatted nodes. Unformatted nodes are logical blocks with no given format, used to store file data, and the direct items consist of file data itself. Also, those items are of variable size and stored within the leaf nodes of the tree, sometimes with others in case there is enough space within the node. This is why we said before that file information is stored close to file data, since the file system always tries to put stat data items and the direct/indirect items of the same file together. Realise that opposed to direct items, the file data pointed by indirect items is not stored within the tree. This special management of direct items is due to small file support.
The direct items are intended to keep small file data and even the tails of the files. Therefore, several tails could be kept within the same leaf node, producing an important decrease of wasted space. The problem is that using this technique of keeping the file's tails together would increase external fragmentation, since the file data is now further from the file tail. Moreover, the task of packing tails is time-consuming and leads to performance decrease. This is a consequence of the memory shifts needed when someone appends data to a file. Anyway, the tails packing technique can be disabled if the administrator wants to do so. Consequently, it's once again an administrator choice.
|
http://www.linuxgazette.com/issue55/florido.html
So Reiserfs DOES take care of itself as long as you aren't packing tails?
I'd like a real answer... would be nice to know which filesystems have terrible fragmentation.
-Feld
_________________
< bmg505> I think the first line in reiserfsck is
if (random(65535)< 65500) { hose(partition); for (i=0;i<100000000;i++) print_crap(); } |
|
| Back to top |
|
|
Enlight
Advocate
Joined: 28 Oct 2004
Posts: 3519
Location: Alsace (France)
|
| Posted: Thu Feb 02, 2006 3:52 pm Post subject: |
|
|
| Hans Reiser from NameSys, for ReiserFS wrote: |
3. What is the fragmentation policy of ReiserFS and how is it dealing with it?
Hans Reiser: This is an area we are still experimenting with. We currently do what ext2 does, and preallocate blocks. What XFS does is much better, they allocate blocknrs to blocks at the time they are flushed to disk, and this allows a much more efficient and optimal allocation to occur. We knew we couldn't do it the XFS way and make code freeze for 2.4, but reiser4 is being built around delayed allocation, and I'd like to thank the XFS developers for taking the time to personally explain to me why delayed allocation is the way to go. |
Fact is reiser 3.6 isn't good at managing external fragmentation. In fact, there are cases in wich tails will help and other in wich they will make things just worst.
Without tails, reiser has to perform more allocations, so in this case, notail semms to be the worst, as it's just like using the fs for a longer time.
But with tails, if files in the tails become larger, you'll need to allocate a new bloc, but as you're in a tail, next bloc is probably occuped (because when tails are created, it means that several files need only one bloc, so why marking the next one as reserved), therefore growing files end-up with lot's of chances of being fragmented. |
|
| Back to top |
|
|
vipernicus
Veteran
Joined: 17 Jan 2005
Posts: 1462
Location: Your College IT Dept.
|
| Posted: Thu Feb 02, 2006 3:59 pm Post subject: |
|
|
| Enlight wrote: | | Hans Reiser from NameSys, for ReiserFS wrote: |
3. What is the fragmentation policy of ReiserFS and how is it dealing with it?
Hans Reiser: This is an area we are still experimenting with. We currently do what ext2 does, and preallocate blocks. What XFS does is much better, they allocate blocknrs to blocks at the time they are flushed to disk, and this allows a much more efficient and optimal allocation to occur. We knew we couldn't do it the XFS way and make code freeze for 2.4, but reiser4 is being built around delayed allocation, and I'd like to thank the XFS developers for taking the time to personally explain to me why delayed allocation is the way to go. |
Fact is reiser 3.6 isn't good at managing external fragmentation. In fact, there are cases in wich tails will help and other in wich they will make things just worst.
Without tails, reiser has to perform more allocations, so in this case, notail semms to be the worst, as it's just like using the fs for a longer time.
But with tails, if files in the tails become larger, you'll need to allocate a new bloc, but as you're in a tail, next bloc is probably occuped (because when tails are created, it means that several files need only one bloc, so why marking the next one as reserved), therefore growing files end-up with lot's of chances of being fragmented. |
Just to make sure I understand, let's simplify things.
Which answer is correct?
1. Using NOTAIL helps keep your Reiser3.6 filesystem from becoming fragmented.
2. Using DEFAULTS helps keep your Reiser3.6 filesystem from becoming fragmented.
3. Using Reiser3.6 for a period of time with 1 or 2, and filesystem will become fragmented.
_________________
Viper-Sources Maintainer || nesl247 Projects || vipernicus.org blog |
|
| Back to top |
|
|
Enlight
Advocate
Joined: 28 Oct 2004
Posts: 3519
Location: Alsace (France)
|
| Posted: Thu Feb 02, 2006 4:18 pm Post subject: |
|
|
3! but keeping tails may slow down the process if your small files aren't supposed to grow (but will speed it up if they do! ).
But don't take me wrong, every file system may end-up fragmenting files, it's just that some have just extremly low rate of fragmentation and may need years of intense usage to end-up as screwed as some may become 1 a month of normal usage.
The real problem actually is that reiserfs does not seems to have a defraging tool (Other than backup format restore) and then looses it's great speed quickly. The problem unhappily don't seem to have been fixed in r4 even if they uses delayed allocations (but at least delayed allocations are a good thing for our ears)... |
|
| Back to top |
|
|
dhave
Apprentice
Joined: 28 Oct 2005
Posts: 298
Location: Still outside the Matrix ...
|
| Posted: Thu Feb 02, 2006 5:37 pm Post subject: |
|
|
| Enlight wrote: | | The real problem actually is that reiserfs does not seems to have a defraging tool (Other than backup format restore) and then looses it's great speed quickly. The problem unhappily don't seem to have been fixed in r4 even if they uses delayed allocations (but at least delayed allocations are a good thing for our ears)... |
Isn't there a "repacker" that's supposed to be included in Reiser4.1?
_________________
This space available! |
|
| Back to top |
|
|
Enlight
Advocate
Joined: 28 Oct 2004
Posts: 3519
Location: Alsace (France)
|
| Posted: Thu Feb 02, 2006 5:49 pm Post subject: |
|
|
| dhave wrote: | | Enlight wrote: | | The real problem actually is that reiserfs does not seems to have a defraging tool (Other than backup format restore) and then looses it's great speed quickly. The problem unhappily don't seem to have been fixed in r4 even if they uses delayed allocations (but at least delayed allocations are a good thing for our ears)... |
Isn't there a "repacker" that's supposed to be included in Reiser4.1? |
Woops by "the problem don't seem to have been fixed", I meant the problem of handling external fragmentattion. I've heard of a repacker so wait and see, but I guess namesys has got lots of stuffs to do with r4, so the remaining question is when. |
|
| Back to top |
|
|
matobsk
n00b
Joined: 16 Oct 2005
Posts: 64
|
| Posted: Thu Feb 02, 2006 7:58 pm Post subject: |
|
|
I dumped 2 reiser4 partitions and 2 reiser3.6 partitions for this very reason.
I knew the fs was fragmented to hell and back, my hdd was thrashing all over the place, even worse than NTFS! 
When I couldn't find a single utility within the fstools to tell me the non-contiguous data percentage, I cried foul and found a filesystem that really does have an online defragmenter/allocation cache.
XFS is quite fast, and has yet to fragment a single file on me, ever.
Well, it may have initially, but it was never fragmented when I got around to checking it 
I've heard that there may be issues with hard reboots/power failures due to the allocation cache on it, but I run with UPS so the risk is minimal in my opinion. |
|
| Back to top |
|
|
Dan Forever
Tux's lil' helper
Joined: 13 Jun 2005
Posts: 79
Location: London, England
|
| Posted: Thu Feb 02, 2006 11:20 pm Post subject: |
|
|
| dhave wrote: | | Dan Forever wrote: | | I've been told that ReiserFS doesn't fragment and that it manages itself on the fly. As to how correct that is, I don't know. |
I've been told that there's a Santa Claus. |
He brings me presents 
What I'd like to know is, what tools are there to defragment filesystems, such as Reiser? |
|
| Back to top |
|
|
djpharoah
Apprentice
Joined: 24 Nov 2004
Posts: 186
Location: Irvine, California USA
|
| Posted: Fri Feb 03, 2006 6:00 am Post subject: |
|
|
wow this thread sure has me confused.
im running reiserfs 3.6 on my / partition
what options should i have in my /etc/fstab to get the fs to automatically check itself for fragments?
-dj |
|
| Back to top |
|
|
feld
Guru
Joined: 29 Aug 2004
Posts: 593
Location: WI, USA
|
| Posted: Fri Feb 03, 2006 7:16 am Post subject: |
|
|
I too have switched primarily to XFS not too long ago. Performs great!
Right now I have an LVM that has portage, tmp, and /var/tmp on Reiserfs. Figured that would give me pretty decent speed where it may occasionally be needed... and all 3 of those can always be fully wiped without any consquences which can bring the speed back up to its original level.
-Feld
_________________
< bmg505> I think the first line in reiserfsck is
if (random(65535)< 65500) { hose(partition); for (i=0;i<100000000;i++) print_crap(); } |
|
| Back to top |
|
|
|
Off the Wall
