Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Script:Measuring fragmentation on Reiserfs (and other fs)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
johntash
n00b
n00b


Joined: 30 Apr 2005
Posts: 20
Location: KS

PostPosted: Mon Apr 17, 2006 6:45 pm    Post subject: Reply with quote

Awesome. I was just thinking this morning that I should look for a script like this to see how my music partition is doing. I'm trying it out now, thanks! =]
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1342
Location: Sweden

PostPosted: Wed Apr 26, 2006 11:18 am    Post subject: Reply with quote

Does anyone have any idea why the defrag script increases the fragmentation slightly on my /home/ everytime I run it? Reiserfs drive, seems very strange....
Back to top
View user's profile Send private message
Kelvie
Tux's lil' helper
Tux's lil' helper


Joined: 13 Aug 2004
Posts: 88
Location: Richmond, BC

PostPosted: Thu Apr 27, 2006 5:43 am    Post subject: Reply with quote

Just as an FYI:
When large (>100MB) files get overly fragmented, and you attempt to unlink (rm) them, reiserfs craps out and overflows its buffer, causing a kernel panic. I had a few videos stored on a fairly full drive, and let me tell you, this caused way too many problems (including *gasp* reboots). The solution was to upgrade to a 2.6.17 or newer kernel, or use this patch:

Code:

kelvie@valour linux $ cat reiserfs-fix-transaction-overflowing.patch

From: Alexander Zarochentzev <zam@namesys.com>

This patch fixes a bug in reiserfs truncate.  A transaction might overflow
when truncating long highly fragmented file.  The fix is to split
truncation into several transactions to avoid overflowing.

Signed-off-by: Vladimir V. Saveliev <vs@namesys.com>
Cc; Charles McColgan <cm@chuck.net>
Cc: Alexander Zarochentsev <zam@namesys.com>
Cc: Hans Reiser <reiser@namesys.com>
Cc: Chris Mason <mason@suse.com>
Cc: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 fs/reiserfs/stree.c         |  208 +++++++++++-----------------------
 include/linux/reiserfs_fs.h |    5
 2 files changed, 76 insertions(+), 137 deletions(-)

diff -puN fs/reiserfs/stree.c~reiserfs-fix-transaction-overflowing fs/reiserfs/stree.c
--- devel/fs/reiserfs/stree.c~reiserfs-fix-transaction-overflowing   2006-02-21 12:50:26.000000000 -0800
+++ devel-akpm/fs/reiserfs/stree.c   2006-02-21 12:50:26.000000000 -0800
@@ -981,6 +981,8 @@ static inline int prepare_for_direntry_i
    return M_CUT;
 }
 
+#define JOURNAL_FOR_FREE_BLOCK_AND_UPDATE_SD (2 * JOURNAL_PER_BALANCE_CNT + 1)
+
 /*  If the path points to a directory or direct item, calculate mode and the size cut, for balance.
     If the path points to an indirect item, remove some number of its unformatted nodes.
     In case of file truncate calculate whether this item must be deleted/truncated or last
@@ -1020,148 +1022,79 @@ static char prepare_for_delete_or_cut(st
 
    /* Case of an indirect item. */
    {
-      int n_unfm_number,   /* Number of the item unformatted nodes. */
-       n_counter, n_blk_size;
-      __le32 *p_n_unfm_pointer;   /* Pointer to the unformatted node number. */
-      __u32 tmp;
-      struct item_head s_ih;   /* Item header. */
-      char c_mode;   /* Returned mode of the balance. */
-      int need_research;
-
-      n_blk_size = p_s_sb->s_blocksize;
-
-      /* Search for the needed object indirect item until there are no unformatted nodes to be removed. */
-      do {
-         need_research = 0;
-         p_s_bh = PATH_PLAST_BUFFER(p_s_path);
-         /* Copy indirect item header to a temp variable. */
-         copy_item_head(&s_ih, PATH_PITEM_HEAD(p_s_path));
-         /* Calculate number of unformatted nodes in this item. */
-         n_unfm_number = I_UNFM_NUM(&s_ih);
-
-         RFALSE(!is_indirect_le_ih(&s_ih) || !n_unfm_number ||
-                pos_in_item(p_s_path) + 1 != n_unfm_number,
-                "PAP-5240: invalid item %h "
-                "n_unfm_number = %d *p_n_pos_in_item = %d",
-                &s_ih, n_unfm_number, pos_in_item(p_s_path));
-
-         /* Calculate balance mode and position in the item to remove unformatted nodes. */
-         if (n_new_file_length == max_reiserfs_offset(inode)) {   /* Case of delete. */
-            pos_in_item(p_s_path) = 0;
-            *p_n_cut_size = -(IH_SIZE + ih_item_len(&s_ih));
-            c_mode = M_DELETE;
-         } else {   /* Case of truncate. */
-            if (n_new_file_length < le_ih_k_offset(&s_ih)) {
-               pos_in_item(p_s_path) = 0;
-               *p_n_cut_size =
-                   -(IH_SIZE + ih_item_len(&s_ih));
-               c_mode = M_DELETE;   /* Delete this item. */
-            } else {
-               /* indirect item must be truncated starting from *p_n_pos_in_item-th position */
-               pos_in_item(p_s_path) =
-                   (n_new_file_length + n_blk_size -
-                    le_ih_k_offset(&s_ih)) >> p_s_sb->
-                   s_blocksize_bits;
-
-               RFALSE(pos_in_item(p_s_path) >
-                      n_unfm_number,
-                      "PAP-5250: invalid position in the item");
-
-               /* Either convert last unformatted node of indirect item to direct item or increase
-                  its free space.  */
-               if (pos_in_item(p_s_path) ==
-                   n_unfm_number) {
-                  *p_n_cut_size = 0;   /* Nothing to cut. */
-                  return M_CONVERT;   /* Maybe convert last unformatted node to the direct item. */
-               }
-               /* Calculate size to cut. */
-               *p_n_cut_size =
-                   -(ih_item_len(&s_ih) -
-                     pos_in_item(p_s_path) *
-                     UNFM_P_SIZE);
+       int blk_size = p_s_sb->s_blocksize;
+       struct item_head s_ih;
+       int need_re_search;
+       int delete = 0;
+       int result = M_CUT;
+       int pos = 0;
+
+       if ( n_new_file_length == max_reiserfs_offset (inode) ) {
+      /* prepare_for_delete_or_cut() is called by
+       * reiserfs_delete_item() */
+      n_new_file_length = 0;
+      delete = 1;
+       }
+
+       do {
+      need_re_search = 0;
+      *p_n_cut_size = 0;
+      p_s_bh = PATH_PLAST_BUFFER(p_s_path);
+      copy_item_head(&s_ih, PATH_PITEM_HEAD(p_s_path));
+      pos = I_UNFM_NUM(&s_ih);
 
-               c_mode = M_CUT;   /* Cut from this indirect item. */
-            }
-         }
+      while (le_ih_k_offset (&s_ih) + (pos - 1) * blk_size > n_new_file_length) {
+          __u32 *unfm, block;
 
-         RFALSE(n_unfm_number <= pos_in_item(p_s_path),
-                "PAP-5260: invalid position in the indirect item");
+          /* Each unformatted block deletion may involve one additional
+           * bitmap block into the transaction, thereby the initial
+           * journal space reservation might not be enough. */
+          if (!delete && (*p_n_cut_size) != 0 &&
+         reiserfs_transaction_free_space(th) < JOURNAL_FOR_FREE_BLOCK_AND_UPDATE_SD) {
+         break;
+          }
 
-         /* pointers to be cut */
-         n_unfm_number -= pos_in_item(p_s_path);
-         /* Set pointer to the last unformatted node pointer that is to be cut. */
-         p_n_unfm_pointer =
-             (__le32 *) B_I_PITEM(p_s_bh,
-                   &s_ih) + I_UNFM_NUM(&s_ih) -
-             1 - *p_n_removed;
-
-         /* We go through the unformatted nodes pointers of the indirect
-            item and look for the unformatted nodes in the cache. If we
-            found some of them we free it, zero corresponding indirect item
-            entry and log buffer containing that indirect item. For this we
-            need to prepare last path element for logging. If some
-            unformatted node has b_count > 1 we must not free this
-            unformatted node since it is in use. */
-         reiserfs_prepare_for_journal(p_s_sb, p_s_bh, 1);
-         // note: path could be changed, first line in for loop takes care
-         // of it
+          unfm = (__u32 *)B_I_PITEM(p_s_bh, &s_ih) + pos - 1;
+          block = get_block_num(unfm, 0);
 
-         for (n_counter = *p_n_removed;
-              n_counter < n_unfm_number;
-              n_counter++, p_n_unfm_pointer--) {
-
-            cond_resched();
-            if (item_moved(&s_ih, p_s_path)) {
-               need_research = 1;
-               break;
-            }
-            RFALSE(p_n_unfm_pointer <
-                   (__le32 *) B_I_PITEM(p_s_bh, &s_ih)
-                   || p_n_unfm_pointer >
-                   (__le32 *) B_I_PITEM(p_s_bh,
-                         &s_ih) +
-                   I_UNFM_NUM(&s_ih) - 1,
-                   "vs-5265: pointer out of range");
-
-            /* Hole, nothing to remove. */
-            if (!get_block_num(p_n_unfm_pointer, 0)) {
-               (*p_n_removed)++;
-               continue;
-            }
+          if (block != 0) {
+         reiserfs_prepare_for_journal(p_s_sb, p_s_bh, 1);
+         put_block_num(unfm, 0, 0);
+         journal_mark_dirty (th, p_s_sb, p_s_bh);
+         reiserfs_free_block(th, inode, block, 1);
+          }
 
-            (*p_n_removed)++;
+          cond_resched();
 
-            tmp = get_block_num(p_n_unfm_pointer, 0);
-            put_block_num(p_n_unfm_pointer, 0, 0);
-            journal_mark_dirty(th, p_s_sb, p_s_bh);
-            reiserfs_free_block(th, inode, tmp, 1);
-            if (item_moved(&s_ih, p_s_path)) {
-               need_research = 1;
-               break;
-            }
-         }
+          if (item_moved (&s_ih, p_s_path))  {
+         need_re_search = 1;
+         break;
+          }
 
-         /* a trick.  If the buffer has been logged, this
-          ** will do nothing.  If we've broken the loop without
-          ** logging it, it will restore the buffer
-          **
-          */
-         reiserfs_restore_prepared_buffer(p_s_sb, p_s_bh);
-
-         /* This loop can be optimized. */
-      } while ((*p_n_removed < n_unfm_number || need_research) &&
-          search_for_position_by_key(p_s_sb, p_s_item_key,
-                      p_s_path) ==
-          POSITION_FOUND);
-
-      RFALSE(*p_n_removed < n_unfm_number,
-             "PAP-5310: indirect item is not found");
-      RFALSE(item_moved(&s_ih, p_s_path),
-             "after while, comp failed, retry");
-
-      if (c_mode == M_CUT)
-         pos_in_item(p_s_path) *= UNFM_P_SIZE;
-      return c_mode;
+          pos --;
+          (*p_n_removed) ++;
+          (*p_n_cut_size) -= UNFM_P_SIZE;
+
+          if (pos == 0) {
+         (*p_n_cut_size) -= IH_SIZE;
+         result = M_DELETE;
+         break;
+          }
+      }
+      /* a trick.  If the buffer has been logged, this will do nothing.  If
+      ** we've broken the loop without logging it, it will restore the
+      ** buffer */
+      reiserfs_restore_prepared_buffer(p_s_sb, p_s_bh);
+       } while (need_re_search &&
+           search_for_position_by_key(p_s_sb, p_s_item_key, p_s_path) == POSITION_FOUND);
+       pos_in_item(p_s_path) = pos * UNFM_P_SIZE;
+
+       if (*p_n_cut_size == 0) {
+      /* Nothing were cut. maybe convert last unformatted node to the
+       * direct item? */
+      result = M_CONVERT;
+       }
+       return result;
    }
 }
 
@@ -1948,7 +1881,8 @@ int reiserfs_do_truncate(struct reiserfs
        ** sure the file is consistent before ending the current trans
        ** and starting a new one
        */
-      if (journal_transaction_should_end(th, th->t_blocks_allocated)) {
+      if (journal_transaction_should_end(th, 0) ||
+          reiserfs_transaction_free_space(th) <= JOURNAL_FOR_FREE_BLOCK_AND_UPDATE_SD) {
          int orig_len_alloc = th->t_blocks_allocated;
          decrement_counters_in_path(&s_search_path);
 
@@ -1962,7 +1896,7 @@ int reiserfs_do_truncate(struct reiserfs
          if (err)
             goto out;
          err = journal_begin(th, p_s_inode->i_sb,
-                   JOURNAL_PER_BALANCE_CNT * 6);
+                   JOURNAL_FOR_FREE_BLOCK_AND_UPDATE_SD + JOURNAL_PER_BALANCE_CNT * 4) ;
          if (err)
             goto out;
          reiserfs_update_inode_transaction(p_s_inode);
diff -puN include/linux/reiserfs_fs.h~reiserfs-fix-transaction-overflowing include/linux/reiserfs_fs.h
--- devel/include/linux/reiserfs_fs.h~reiserfs-fix-transaction-overflowing   2006-02-21 12:50:26.000000000 -0800
+++ devel-akpm/include/linux/reiserfs_fs.h   2006-02-21 12:50:26.000000000 -0800
@@ -1704,6 +1704,11 @@ static inline int reiserfs_transaction_r
    return 0;
 }
 
+static inline int reiserfs_transaction_free_space(struct reiserfs_transaction_handle *th)
+{
+   return th->t_blocks_allocated - th->t_blocks_logged;
+}
+
 int reiserfs_async_progress_wait(struct super_block *s);
 
 struct reiserfs_transaction_handle *reiserfs_persistent_transaction(struct
_


I've applied this patch to all three of my boxes (2.6.16-reiser4 opteron, 2.6.16-gentoo-sources athlon, 2.6.15-suspend2 pentium-m), and it fixed the issues.

Sadly, reiserfs is still the best filesystem out there, despite stupid bugs like these. Just have to wait until reiser4 comes with a grow utility, I guess.

Kelvie
Back to top
View user's profile Send private message
G2k
l33t
l33t


Joined: 06 Mar 2004
Posts: 672
Location: Rome, Italy

PostPosted: Thu Apr 27, 2006 6:19 am    Post subject: Reply with quote

Kelvie wrote:
Sadly, reiserfs is still the best filesystem out there
I don't second that although I use reiserfs myself. Ext3 and XFS are good competitors depending on what you have to do with your partition.
_________________
Animula vagula blandula,
Hospes comesque corporis,
Quae nunc abibis in loca
Pallidula rigida nudula,
Nec ut soles dabis iocos...
- Imp. Caesar Hadrianus
Back to top
View user's profile Send private message
c07
Veteran
Veteran


Joined: 25 Oct 2002
Posts: 1091

PostPosted: Fri Apr 28, 2006 10:12 pm    Post subject: Fragmentation viewer Reply with quote

sirtalon42 wrote:
I ran your program on my /usr/portage directory and got results that don't seem possible (<1% non contiguous files):
Code:
# ./fragchk.pl /usr/portage
0.121585826566503% non contiguous files, 1.00151982283208 average fragments.

/usr/portage isn't on its own partition, and thats the only thing I can think of that would be throwing off these results.

Use a more verbose script like the following one (based on as' script):

Code:
#! /usr/bin/perl -w
use strict;

@ARGV >= 1 && @ARGV <= 2 or die "usage: $0 <dir> [<block size in KB>]";

$/= "\0";
my ($files, $blocks, $fragments, $frag, $fragblocks, $multi, $empty)= (0) x 7;
my $dir= shift;
my $blocksize= (shift || 4) + 0;

print qq|scanning "$dir", using block size $blocksize KB ...\n|;

open my $find, "-|", "find", $dir, qw"-xdev -type f -print0";

while ( my $file= <$find> ) {
  { open my $fh, "-|", "filefrag", $file; $_= <$fh> }
  /:\s+(\d+) extents? found/ or (print qq|"$_"?\n|), next;
  my $n= $1 + 0;
  { open my $fh, "-|", "ls", "-sk", $file; $_= <$fh> }
  /^(\d+)\s/ or (print qq|"$_" (ls)?\n|), next;
  my $s= $1 / $blocksize;
  ++$files;
  $s or ++$empty, next;
  $blocks += $s;
  $fragments += $n;
  ++$frag, $fragblocks += $s if $n > 1;
  ++$multi if $s > 1;
}

my $single= $files - $multi - $empty;
my $nonfrag= $files - $frag - $empty;

if ( ! $files ) { print "no files\n" }
else {
  printf "$files files, $frag (%.3f %%) fragmented\n", 100 * $frag / $files;
  if ( ! $multi ) { print "no multi-block files\n" }
  else {
    printf "$multi multi-block files, %.3f %% fragmented\n",
      100 * $frag / $multi;
  }
  print "$blocks blocks, $fragments fragments, $empty empty files\n";
  if ( $fragments ) {
    printf "average %.3f fragments per file, %.3f blocks per fragment,\n",
      $fragments / $files, $blocks / $fragments;
    if ( $multi ) {
      printf "%.3f fragments per multi-block file, %.3f blocks each,\n",
        ($fragments - $single) / $multi,
        ($blocks - $single) / ($fragments - $single);
      if ( $frag ) {
        printf "%.3f fragments per fragmented file, %.3f blocks each\n",
        ($fragments - $nonfrag) / $frag,
        $fragblocks / ($fragments - $nonfrag);
} } } }

Assumes 4 KB block size, but you can override it by supplying a second argument.

Performance for the portage tree doesn't depend much on internal file fragmentation, but on directory fragmentation (files scattered all around the filesystem).

To make fragmentation visible, you can use a script like this:

Code:
#! /usr/bin/perl -w
use strict;
require "linux/fs.ph";

@ARGV == 1 or die "usage: $0 <dir>";

$/= "\0";
my (%blocks, $last);
my $dir= shift;

print qq|scanning "$dir" ...\n|;

open my $find, "-|", "find", $dir, qw"-xdev -type f -print0";

while ( my $file= <$find> ) {
  open my $fh, "<", $file or die $!;
  my $x; { no warnings; ioctl $fh, &FIGETBSZ, $x }
  $x= unpack "L!", $x;
  $x or next;
  my ($blocks, $frag, $last)= ((-s $fh) / $x, 1);
  for ( my $i= 0; $i < $blocks; ++$i ) {
    $x= pack "L!", $i;
    ioctl $fh, &FIBMAP, $x;
    $x= unpack "L!", $x;
    ++$frag if defined $last && $x != $last + 1;
    $last= $x;
    $blocks{$x}= exists $blocks{$x} ? "+" :
      $frag == 1 ? "*" : $frag > 9 ? "#" : $frag;
  }
}

for my $k ( sort { $a <=> $b } keys %blocks ) {
  if ( ! defined $last || $k - $last > 128 ) {
    printf "\n0x%.8x: ", $k;
    print " " x ($k & 0x3f);
    $last= $k;
  }
  else {
    while ( 1 ) {
      ++$last & 0x3f or print "\n", " " x 12;
      last if $last == $k;
      print ".";
    }
  }
  print $blocks{$k};
}

print "\n";

Takes a directoty (or single file) as argument and prints a block map of all contained files. "*" marks first fragments, "2" to "9" 2nd to 9th fragments, "#" further fragments, "+" multiple used blocks (this should only apply to block #0, which is returned for any tails that do not have a real block associated) and "." blocks not used by these files (but note that they may be used by ext2's indirect blocks (which are supposed to fragment larger files (filefrag accounts for this)), directory lists, inodes, ...). Skips to the next used block if 128 or more consecutive blocks aren't used, and prints the number of the next block.

Probably it should do some error checking. Use it at your own risk.
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Fri May 05, 2006 1:14 am    Post subject: Reply with quote

I keep getting this:

Code:
root@smoker / # /root/fragck.pl /
/root/fragck.pl: line 5: use: command not found
/root/fragck.pl: line 8: my: command not found
/root/fragck.pl: line 10: my: command not found
/root/fragck.pl: line 12: my: command not found
/root/fragck.pl: line 15: syntax error near unexpected token `FILES,'
/root/fragck.pl: line 15: `open (FILES, "find " . $ARGV[0] . " -xdev -type f |");'
root@smoker / #


What package has these commands so I can install them?

I read that XFS is really fast and a really good file system. Anybody here use it a lot??

Thanks

:D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2506
Location: Silver City, NM

PostPosted: Fri May 05, 2006 5:29 am    Post subject: Reply with quote

Dalek, it looks like you may have copied the script incorrectly.

It looks like Bash is trying to interpret the script instead of Perl. Also, the "use" error is reported to be on line 5 while in the actual script that line is line 4. The very first line must be:
Code:
#!/usr/bin/perl -w


A quick and dirty fix would be to run the Perl program explicitly:
Code:
# perl -w /root/fragck.pl /

If this works then that would tend to confirm there is a problem with the first line in the script.
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Fri May 05, 2006 6:11 am    Post subject: Reply with quote

That was it. There was a blank line for the first line. Then there was a space in what was suposed to be the first line as well. I had that thing badly confused.

Thanks

:D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Fri May 05, 2006 6:39 am    Post subject: Reply with quote

Well, I have been running mine for a good while and this was the one that had the highest fragments:

Code:
root@smoker / # /root/fragck.pl /
6.23781676413255% non contiguous files, 1.16560503036536 average fragments.
root@smoker / #


I have been using it for a couple years so that is not to bad really. What do you folks think??

Oh, here is how mine is set up:

Code:
root@smoker / # df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda6             14647740   2951284  11696456  21% /
udev                    517288       196    517092   1% /dev
/dev/hda1               146612     39832    106780  28% /boot
/dev/hda7              9765136   3496720   6268416  36% /home
/dev/hda8             14647740   6256632   8391108  43% /usr
/dev/hda9              4882532   2134980   2747552  44% /usr/portage
/dev/hda10            35462336  14888940  20573396  42% /mnt/data
none                    517288         0    517288   0% /dev/shm
root@smoker / #


Thoughts??

:D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2506
Location: Silver City, NM

PostPosted: Fri May 05, 2006 6:52 am    Post subject: Reply with quote

One quick check would be to run:
Code:
# time du -sh /usr/portage


I used the procedure outlined in this post and the time dropped from over 3 minutes to 20 seconds, almost a factor of 10! My system was brand new two months ago. It has also made the rsync portion of "emerge --sync" much faster but I don't have actually numbers to report.

If you want to try this make sure you move distfiles out of /usr/portage and make /usr/portage/distfiles a symlink to the real location.
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Tue Aug 22, 2006 6:09 am    Post subject: Reply with quote

I'm not sure if this is a bug or what. I get a lot of these on one of my partitions:

Code:
statfs: No such file or directory
sh: [1].htm: command not found
Use of uninitialized value in pattern match (m//) at /root/fragck.pl line 32.
Use of uninitialized value in concatenation (.) or string at /root/fragck.pl line 41.
 : not understand for /mnt/data/Teresa/Documents\ and\ Settings2/Tee/Local\ Settings/Temporary\ Internet\ Files/Content.IE5/C18XM70P/keywords;kw=computer+keyboards+with+lights;cat=58058;tcat=51148;items=1743;sz=440x198;tile=5;ord=1146016301420;[1].htm
.


I notice that it has a lot of \ in the paths. Could that have something to do with it? I back up my wife's laptop on this partition with samba. Resierfack says it is fine though. Any ideas?

:D :D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Sat Aug 26, 2006 1:05 pm    Post subject: Reply with quote

Quote:
Sadly, reiserfs is still the best filesystem out there,

No , sadly reiserfs sucks. The good news is its not the best out there. :wink:

R4 is streets ahead of reiserfs and if you really cant stand the idea of a fs that is not included in main line kernel yet use ext3 (with a few -b tweeks etcetera if you feel like.)

I dont see why having growfs should the determining factor of the choice of fs.

Learn to manage your disk space better with several smaller partitions and you'll find backing up and life in general a lot easier.

To the guy who has all on a single 250G :roll: try using tar to archive portage, clean out the files and untar it back. This will defrag that bit at least.

8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Sat Aug 26, 2006 2:42 pm    Post subject: Reply with quote

Any idea what version of gentoo-sources will have reiserfs4 in it? I have read it is a good thing.

Just curious.

:D :D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Sat Aug 26, 2006 6:59 pm    Post subject: Reply with quote

no version of std gentoo , vanilla or other sources support this yet, although it is getting active attention.

you will need to either patch your kernel sources (very simple , go get the reiser4 patch for your kernel version, apply patch , rebuild kernel. Really a doodle to do.) or use one of the many patched kernel sources in unsupported forum that do. eg. no-sources , beyond , viper ...... many of these are much more responcive as well and well worth checking out.

8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Sat Aug 26, 2006 7:07 pm    Post subject: Reply with quote

I tried to patch a gentoo kernel once and I kept getting errors because it was already patched with the Gentoo stuff and things were not where they are suposed to be or something. Would that still be a issue or should I just wait till it comes out and is stable?

I looked everywhere to see if I could find a date it would be released and I found nothing. I looked here:

http://www.namesys.com/

Any place else that may have a clue??

:D :D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Sat Aug 26, 2006 7:42 pm    Post subject: Reply with quote

you should find the appropriate patch file here , ftp://ftp.namesys.com/pub/reiser4-for-2.6

if you are a bit unsure just go for something like no-sources and enable support through make menuconfig.

8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5350
Location: France, Old Europe

PostPosted: Sun Aug 27, 2006 5:13 am    Post subject: Reply with quote

getting a bit OT here so back On T:

just ran droop's script on two of my R4 partitions.
Code:
bash-3.1#fragck.pl /tmpd
28.0939116593713% non contiguous files, 1.60206923995225 average fragments.

bash-3.1#fragck.pl /usr/portage
6.75392192630427% non contiguous files, 1.28991345413272 average fragments.



ran beautifully, surprisingly fast.

Nice tool. Thanks a lot. 8)


One reason I like to keep my partitions small. Every week or so I do a clone of my root fs as a backup. If I need to defrag I clone with cp -ax and reboot to the clone. Sys backup and defrag takes about 15m , 12 of which is disk activity.
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86
Back to top
View user's profile Send private message
dalek
Veteran
Veteran


Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA

PostPosted: Sun Aug 27, 2006 8:50 pm    Post subject: Reply with quote

Well, how is this?

Code:
root@smoker / # /root/fragck.pl /
16.4263677754833% non contiguous files, 1.46472644490334 average fragments.
root@smoker / # /root/fragck.pl /boot/
4% non contiguous files, 1.12 average fragments.
root@smoker / # /root/fragck.pl /home/
8.08759345705719% non contiguous files, 1.70317659114198 average fragments.
root@smoker / # /root/fragck.pl /usr/
1.93224184188215% non contiguous files, 1.06032884115818 average fragments.
root@smoker / # /root/fragck.pl /usr/portage/
0.199848115432271% non contiguous files, 1.00777808865262 average fragments.
root@smoker / #   


Looks like the / partition is the worst don't it? Or does that sort of include all the others too? May be time for a backup and redoing my partitions again. That will fix it. 8O

:D :D :D :D :D
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case
Back to top
View user's profile Send private message
nico_calais
l33t
l33t


Joined: 09 Jun 2005
Posts: 628
Location: Saint Julien en Genevois

PostPosted: Sun Aug 27, 2006 10:12 pm    Post subject: Reply with quote

Here's my /home partition after two weeks of use :
Code:
7.87931034482759% non contiguous files, 24.4593103448276 average fragments.

And after the defrag tool
Code:
3.271913208197% non contiguous files, 2.54537627001894 average fragments.


Thanks for your tool. And the defrag tool is very nice too :wink:
_________________
"Unix IS user friendly... It's just selective about who its friends are." — Tollef Fog Heen tollef@add.no
Back to top
View user's profile Send private message
gerardo
Apprentice
Apprentice


Joined: 05 Feb 2004
Posts: 228
Location: Belgium

PostPosted: Sun Nov 12, 2006 2:04 pm    Post subject: Reply with quote

Defrag increases fragmentation for me.
Code:
/mnt/data # ~/defrag.sh /mnt/data/
Creating list of files...
156 files will be reordered

 0 files left                                                               
Succeeded
/mnt/data # ~/fragcheck.pl /mnt/data/
scanning "/mnt/data/", using block size 4 KB ...
155 files, 60 (38.710 %) fragmented
139 multi-block files, 43.165 % fragmented
3740532.75 blocks, 7376 fragments, 0 empty files
average 47.587 fragments per file, 507.122 blocks per fragment,
52.950 fragments per multi-block file, 508.222 blocks each,
121.350 fragments per fragmented file, 509.856 blocks each
/mnt/data # ~/defrag.sh /mnt/data/
Creating list of files...
156 files will be reordered

 0 files left                                                               
Succeeded
/mnt/data # ~/fragcheck.pl /mnt/data/
scanning "/mnt/data/", using block size 4 KB ...
155 files, 63 (40.645 %) fragmented
139 multi-block files, 45.324 % fragmented
3740532.75 blocks, 6444 fragments, 0 empty files
average 41.574 fragments per file, 580.468 blocks per fragment,
46.245 fragments per multi-block file, 581.910 blocks each,
100.825 fragments per fragmented file, 584.524 blocks each


This happens on a drive formatted with reiserfs 3.6 where there are a lot of files > 100 Mbytes.
Disk usage is only 48%:
Code:
/dev/md/2             31261248  14995300  16265948  48% /mnt/data


Next, I did some moving of the largest files (700 Mb and more) to other partitions.
Now usage is 39%:
Code:
/dev/md/2             31261248  12126788  19134460  39% /mnt/data


After another defrag, it is even worse:
Code:
/mnt/data # ~/defrag.sh /mnt/data/
Creating list of files...
153 files will be reordered

 0 files left                                                             
Succeeded
/mnt/data # ~/fragcheck.pl /mnt/data/
scanning "/mnt/data/", using block size 4 KB ...
152 files, 69 (45.395 %) fragmented
146 multi-block files, 47.260 % fragmented
3023422.5 blocks, 3035 fragments, 0 empty files
average 19.967 fragments per file, 996.185 blocks per fragment,
20.747 fragments per multi-block file, 998.157 blocks each,
42.783 fragments per fragmented file, 1015.926 blocks each


Any clues why and how to solve it?
_________________
Windoze : Plug and Pay...
Back to top
View user's profile Send private message
buddabrod
Apprentice
Apprentice


Joined: 15 Oct 2006
Posts: 241
Location: Germany

PostPosted: Fri Jan 12, 2007 9:23 pm    Post subject: Reply with quote

Little late, i know, but where is your problem? It's much better, look at the number of fragments per file an at the size of the fragments.
Back to top
View user's profile Send private message
user11
Tux's lil' helper
Tux's lil' helper


Joined: 25 Nov 2005
Posts: 149

PostPosted: Fri Jun 13, 2008 2:13 pm    Post subject: Reply with quote

We may use xargs to improve performance.

The script below:
1. is based on as's script (uses -print0),
2. has much better performance (due to xargs),
3. reports a little more details if -v option is specified,
4. rounds values to reasonable precision.

Note: xargs is really safe and powerful tool :)

Code:

#!/usr/bin/perl -w

#this script search for frag on a fs
use strict;

#number of files
my $files = 0;
#number of fragment
my $fragments = 0;
#number of fragmented files
my $fragfiles = 0;

my $verbose;

if ($ARGV[0] eq '-v') { shift @ARGV; $verbose++; }

open (REPORT, "find " . $ARGV[0] . " -xdev -type f -print0 | xargs -0 filefrag |");

while (defined (my $res = <REPORT>)) {
        if ($res =~ m/.*:\s+(\d+) extents? found$/) {
                my $fragment = $1;
                $fragments += $fragment;
                if ($fragment > 1) {
                        $fragfiles++;
                }
                $files++;
        } else {
                print ("Failed to parse: $res\n");
        }
}
close (REPORT);

if ($verbose) {
   print "Total files:      $files\n";
   print "Fragmented files: $fragfiles\n";
   print "Fragments:        $fragments\n";
}

sub round($$) {
   my $v = shift; # value
   my $p = shift; # rouding divisor (1 for '123', 10 for '123.4', 100 for '123.45')
   return int($v * $p) / $p;
}
print ( round($fragfiles / $files * 100, 10) . "% non contiguous files, " . round($fragments / $files, 10) . " average fragments.\n");
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6050
Location: Removed by Neddy

PostPosted: Fri Jun 13, 2008 3:18 pm    Post subject: Reply with quote

Quote:

16:18:16 < greybot> xargs is a broken tool if you do not use the -0 option. Use ''find -exec'' or ''for file in *''
instead if at all possible. Two xargs 'bugs' in one: xargs rm <<< "Don't cry.mp3"



DON'T use xargs...
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
user11
Tux's lil' helper
Tux's lil' helper


Joined: 25 Nov 2005
Posts: 149

PostPosted: Fri Jun 13, 2008 4:38 pm    Post subject: Reply with quote

Quote:
xargs is a broken tool if you do not use the -0 option

Hmmm... i DO use -0 option in script above. Afaik (and as far as I tested) it is really safe.

Quote:
DON'T use xargs...

Any other objection?
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 6050
Location: Removed by Neddy

PostPosted: Fri Jun 13, 2008 6:09 pm    Post subject: Reply with quote

there are better ways todo it

that is what the -exec option to find is for
_________________
Quote:
Removed by Chiitoo
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum