Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Shake your files for their own good
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Merlin-TC
l33t
l33t


Joined: 16 May 2003
Posts: 603
Location: Germany

PostPosted: Tue Jun 06, 2006 4:12 pm    Post subject: Reply with quote

Well, I do have version [ebuild R ] sys-apps/shake-0.19 0 kB [1] emerged so that's not it?

Any clues?
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Tue Jun 06, 2006 5:28 pm    Post subject: Reply with quote

Merlin-TC wrote:
Well, I do have version [ebuild R ] sys-apps/shake-0.19 0 kB [1] emerged so that's not it?

Any clues?
Yes: the bugged tarball had the same name as working one, as the buggy wasn't mean to be released (it were just for testing the webpage) :?
if you get the new ebuild and then
Code:
rm /usr/portage/distfiles/shake-0.19.tar.bz2
emerge shake
all should works.
Back to top
View user's profile Send private message
Merlin-TC
l33t
l33t


Joined: 16 May 2003
Posts: 603
Location: Germany

PostPosted: Tue Jun 06, 2006 5:52 pm    Post subject: Reply with quote

Ah ok, seems to work now. Using it on a 200gig drive now ;)
Back to top
View user's profile Send private message
beatryder
Veteran
Veteran


Joined: 08 Apr 2005
Posts: 1138

PostPosted: Wed Jun 07, 2006 5:31 am    Post subject: Reply with quote

98111 wrote:
Merlin-TC wrote:
I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.

Then I added "user_xattr" as option for the partitions I want to use it on.
Am I missing anything?
Sorry, the version you tested wasn't mean to be distributed, I didn't think that someone would try to find the webpage :oops:
But it wasn't harmfull.

From now, version distributed on the page http://vleu.net/shake/ will be stable one.
Direct link to the ebuild : http://vleu.net/shake/shake-0.19.ebuild

THE SEVERE BUG IS FIXED :D (Changelog)

Also, the output should now be readable :wink:.

If it works well, I'll try to submit the ebuild for inclusion in Gentoo.


I just installed shake now, and it gave me the very same problem.


I even tried remounting the partition with the user_xattr flag like you suggested on your website, and it gives me the following:
Quote:
Septu Windows 98 # shake -p ./
Septu Windows 98 # shake -p ./*
Septu Windows 98 # shake -p --old 0 ./
Septu Windows 98 # shake -p-v --old 0 ./
shake: invalid option -- -
shake: invalid args, aborting
Septu Windows 98 # shake -p -v --old 0 ./
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
0 692098 692120 1 0 32 1 .//vmware.log
0 691682 692084 2 1 32 1 .//floppy
626140 591883 591885 1 0 0 1 .//Windows 98.nvram
657421 691687 691725 2 0 32 1 .//vmware-0.log
526352 591887 591887 1 0 0 1 .//Windows 98.vmx
657423 591889 591889 1 0 0 1 .//Windows 98.vmdk
Septu Windows 98 # shake -v --old 0 ./
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
0 692098 692120 1 0 32 1 .//vmware.log
shake: .//floppy: failed to set position time, check user_xattr: Operation not supported
shake: .//floppy: temporary copy failed
Septu Windows 98 # shake -v --old 0 .
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: ./vmware.log: failed to set position time, check user_xattr: Operation not supported
0 591891 591896 1 0 0 1 ./vmware.log
shake: ./floppy: failed to set position time, check user_xattr: Operation not supported
shake: ./floppy: temporary copy failed

Septu Windows 98 # ls -l
total 1529
-rw------- 1 ryan ryan 8664 Feb 19 09:24 Windows 98.nvram
-rw------- 1 ryan ryan 408 Feb 19 09:19 Windows 98.vmdk
-rw------- 1 ryan ryan 0 Feb 19 09:19 Windows 98.vmsd
-rwxr-xr-x 1 ryan ryan 926 Feb 19 09:24 Windows 98.vmx
-rw------- 1 ryan ryan 1474560 Feb 19 09:24 floppy
-rw-r--r-- 1 ryan ryan 44362 Feb 19 09:21 vmware-0.log
-rw-r--r-- 1 ryan ryan 23492 Feb 19 09:24 vmware.log


Also this is on a reiserfs partition. does that matter?
_________________
Dont make it idiot proof, make it work.
Neucode.org
<suppressed key>
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Wed Jun 07, 2006 10:52 am    Post subject: Reply with quote

beatryder wrote:
Quote:
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
0 692098 692120 1 0 32 1 .//vmware.log
shake: .//floppy: failed to set position time, check user_xattr: Operation not supported
shake: .//floppy: temporary copy failed
Septu Windows 98 # shake -v --old 0 .
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: ./vmware.log: failed to set position time, check user_xattr: Operation not supported
0 591891 591896 1 0 0 1 ./vmware.log
shake: ./floppy: failed to set position time, check user_xattr: Operation not supported
Also this is on a reiserfs partition. does that matter?
There is 2 possibility : either your kernel doesn't support user_xattr, or you use the old version (as Merlin-TC was doing).
To know about this, please remove "/usr/portage/distfiles/shake*", make sure you have the latest ebuild, and emerge shake again.
If it still don't works, I would be really happy to see the output of "mount", and the last line of dmesg :).

The next version (which will probably be released this night, or tommorrow :wink:) will have a switch to disable user_xattr, but I would like to have them tested before, because they are really usefull.
Back to top
View user's profile Send private message
Merlin-TC
l33t
l33t


Joined: 16 May 2003
Posts: 603
Location: Germany

PostPosted: Wed Jun 07, 2006 2:16 pm    Post subject: Reply with quote

I have a little bit off topic question about xattr. Does it costs speed or does it improve the speed for programs that support it?
Does it help "gamin" as well?
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Wed Jun 07, 2006 7:09 pm    Post subject: Reply with quote

Merlin-TC wrote:
I have a little bit off topic question about xattr. Does it costs speed or does it improve the speed for programs that support it?
Does it help "gamin" as well?
eXtented ATTRibutes consists in pair of name/values attached to a file.
By exemple, Shake use them for recording the time of space allocation, using the pair "shake_ptime"/time (with time coded in little endian on four bytes).
A file manager could use them to associate preview or mimetype with files, but I don't see how it could help gamin.

And it doesn't cost speed, just a bit of HD space, 5megs for all ptimes of files on my system.
Back to top
View user's profile Send private message
Merlin-TC
l33t
l33t


Joined: 16 May 2003
Posts: 603
Location: Germany

PostPosted: Fri Jun 09, 2006 3:38 pm    Post subject: Reply with quote

Is it possible that xattr takes up a huge amount of space when the partition consists of many small files?
I have one partition where the portage tree is and I just formated it and the tree takes up 122MiB but df -h shows /sda5 4.7G 559M 4.2G 12% /usr/portage.

So it must have a huge overhead.
Is there any way to remove the xattr stuff?
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Sat Jun 10, 2006 12:54 am    Post subject: Reply with quote

Merlin-TC wrote:
Is it possible that xattr takes up a huge amount of space when the partition consists of many small files?
I have one partition where the portage tree is and I just formated it and the tree takes up 122MiB but df -h shows /sda5 4.7G 559M 4.2G 12% /usr/portage.

So it must have a huge overhead.
Is there any way to remove the xattr stuff?
I don't think that the difference is due to xattr. Have you tried to do a "df -h" when there's no files ?
I can write something to remove xattr, but before I would like to ask you something... could you delete this portage tree, mount this partition without user_xattr, re-get the portage three and tell if "df -h" return something different ?
I think that wasted space is not wasted due to xattr, but due to internal structures of reiserfs.
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5213

PostPosted: Sat Jun 10, 2006 2:15 am    Post subject: Reply with quote

i tried running it on /usr/portage and i got a whack of "too many open files". is that directory too big to use shake on?

edit: i have a feature request. can you put in a --one-file-system option? say i wanted to run it on just /usr (not /usr/portage, or /usr/portage/distfiles which is running over nfs).

nice script, i'll play around with it on my systems to see if it makes anything faster.
_________________
goodbye fgo. it was nice knowing you.
Back to top
View user's profile Send private message
Merlin-TC
l33t
l33t


Joined: 16 May 2003
Posts: 603
Location: Germany

PostPosted: Sat Jun 10, 2006 5:45 am    Post subject: Reply with quote

You were right, it had nothing to do with xattr.
I was mounting the partition with the "notail" option (reiserfs) and this caused the huge overhead. There is also another option calles atime, is this somehow connected with xattr?
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Sat Jun 10, 2006 10:03 am    Post subject: Reply with quote

Merlin-TC wrote:
You were right, it had nothing to do with xattr.
I was mounting the partition with the "notail" option (reiserfs) and this caused the huge overhead. There is also another option calles atime, is this somehow connected with xattr?
tail mode spare space, but it is however recomended to disable it on reiserfs<4, which cannot manage it properly.
Atimes helps shake to see the correlation between files.


Last edited by 98111 on Sat Jun 10, 2006 10:09 am; edited 2 times in total
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Sat Jun 10, 2006 10:08 am    Post subject: Reply with quote

bunder wrote:
i tried running it on /usr/portage and i got a whack of "too many open files". is that directory too big to use shake on?
8O ! No, shake open only three files at once, whatever is the size of the directory. I have already used it on /usr/portage...

bunder wrote:
edit: i have a feature request. can you put in a --one-file-system option? say i wanted to run it on just /usr (not /usr/portage, or /usr/portage/distfiles which is running over nfs).
Good idea, it will be done arround tomorrow.

bunder wrote:
nice script, i'll play around with it on my systems to see if it makes anything faster.
OK ^^. If you have some time to experiments with default values... I know they could be improved because I just set them arbitrary :?
Back to top
View user's profile Send private message
Cintra
Advocate
Advocate


Joined: 03 Apr 2004
Posts: 2111
Location: Norway

PostPosted: Sat Jun 10, 2006 12:56 pm    Post subject: Reply with quote

Very nice, well done 98111 :-D

I just ran it on the /root folder of my 2nd Gentoo copy and it works like a charm! It cleaned up a lot of 'guilty' files, and everything still seems to be working fine on that copy ;-)

Btw, one thing that could make shake even nicer is a summary after 'shake -pvv <whatever>' (like XP's analyze)..

Mvh

Amost forgot.. does it work only on Linux partitions?
_________________
"I am not bound to please thee with my answers" W.S.
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5213

PostPosted: Sat Jun 10, 2006 6:10 pm    Post subject: Reply with quote

i found out why i was getting the "too many open files" errors. i didn't happen to look at the path of the files before, but those were the ones on the nfs share. :oops:
_________________
goodbye fgo. it was nice knowing you.
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Sat Jun 10, 2006 10:19 pm    Post subject: Reply with quote

bunder wrote:
i found out why i was getting the "too many open files" errors. i didn't happen to look at the path of the files before, but those were the ones on the nfs share. :oops:
Effectively, I was handling badly the failure of get_testimony... it should be fixed !
I also added --one-file-system, fixed some little bugs, and trapped signals during the file copy (Changelog ebuild).


Last edited by 98111 on Sat Jun 10, 2006 10:36 pm; edited 1 time in total
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Sat Jun 10, 2006 10:32 pm    Post subject: Reply with quote

Cintra wrote:
Very nice, well done 98111 :-D
Thanks :D

Cintra wrote:
Btw, one thing that could make shake even nicer is a summary after 'shake -pvv <whatever>' (like XP's analyze)..
In long term (newt weeks), I will try to contact authors of davtools; or to make a python module that will wrap arround shake and allow to add better UIs while keeping the core source code light.
During this time, adding some statistics in the main source code to remove it just after seems a bad idea...

Cintra wrote:
Amost forgot.. does it work only on Linux partitions?
I would love to have it running on all posix-compliant systems, but that will require a little works.
And it's efficacity depends of the quality of the filesystem implementation, so using it on vfat or ntfs from Linux is definitively a bad idea.
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5213

PostPosted: Sun Jun 11, 2006 5:03 am    Post subject: Reply with quote

my box crashed when running the new version. i'm not sure if it was because i ran out of /dev/shm or if it was because the filesystem was mounted. just something to look for :wink:

edit: maybe it didn't crash. it still responds to pings, so i bet the kernel killed sshd.
_________________
goodbye fgo. it was nice knowing you.


Last edited by bunder on Sun Jun 11, 2006 8:16 am; edited 1 time in total
Back to top
View user's profile Send private message
beatryder
Veteran
Veteran


Joined: 08 Apr 2005
Posts: 1138

PostPosted: Sun Jun 11, 2006 6:15 am    Post subject: Reply with quote

98111 wrote:
beatryder wrote:
Quote:
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
0 692098 692120 1 0 32 1 .//vmware.log
shake: .//floppy: failed to set position time, check user_xattr: Operation not supported
shake: .//floppy: temporary copy failed
Septu Windows 98 # shake -v --old 0 .
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: ./vmware.log: failed to set position time, check user_xattr: Operation not supported
0 591891 591896 1 0 0 1 ./vmware.log
shake: ./floppy: failed to set position time, check user_xattr: Operation not supported
Also this is on a reiserfs partition. does that matter?
There is 2 possibility : either your kernel doesn't support user_xattr, or you use the old version (as Merlin-TC was doing).
To know about this, please remove "/usr/portage/distfiles/shake*", make sure you have the latest ebuild, and emerge shake again.
If it still don't works, I would be really happy to see the output of "mount", and the last line of dmesg :).

The next version (which will probably be released this night, or tommorrow :wink:) will have a switch to disable user_xattr, but I would like to have them tested before, because they are really usefull.



I have not used shaker since AFTER you said the major bug was repaired. I have the xattr in my kernel, and have mounted with the user_xattr option
_________________
Dont make it idiot proof, make it work.
Neucode.org
<suppressed key>
Back to top
View user's profile Send private message
Uncle_Psychosis
Guru
Guru


Joined: 31 Jan 2004
Posts: 387

PostPosted: Mon Jun 12, 2006 5:13 pm    Post subject: Reply with quote

Code:
 # ebuild shake-0.20.ebuild compile
>>> md5 files   ;-) shake-0.17.ebuild
>>> md5 files   ;-) shake-0.20.ebuild
>>> md5 files   ;-) files/digest-shake-0.20
>>> md5 src_uri ;-) shake-0.20.tar.bz2
>>> Checking shake-0.20.tar.bz2's mtime...
>>> WORKDIR is up-to-date, keeping...
gcc -O2 -march=athlon-xp -pipe -std=gnu99 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Werror -pedantic-errors -Wconversion -Wpointer-arith -Wbad-function-cast -Wcast-align -D_XOPEN_SOURCE=600 -O2 -D_POSIX_C_SOURCE=200112L -O0 -ggdb3  -DNDEBUG -c linux.c -o linux.o
linux.c:23:43: attr/attributes.h: No such file or directory
linux.c: In function `set_ptime':
linux.c:44: warning: implicit declaration of function `attr_setf'
linux.c:44: error: `ATTR_DONTFOLLOW' undeclared (first use in this function)
linux.c:44: error: (Each undeclared identifier is reported only once
linux.c:44: error: for each function it appears in.)
linux.c: In function `get_ptime':
linux.c:56: warning: implicit declaration of function `attr_getf'
linux.c:56: error: `ATTR_DONTFOLLOW' undeclared (first use in this function)
make: *** [linux.o] Error 1

!!! ERROR: sys-apps/shake-0.20 failed.
!!! Function src_compile, Line 566, Exitcode 2
!!! emake failed
!!! If you need support, post the topmost build error, NOT this status message.


Any ideas? I'd like to give this a shot!

Sam [/code]
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Tue Jun 13, 2006 3:23 pm    Post subject: Reply with quote

beatryder wrote:
I have not used shaker since AFTER you said the major bug was repaired. I have the xattr in my kernel, and have mounted with the user_xattr option

Could you try
Code:
touch testfile
attr -s testattr -V testval testfile
?
Anyway, the new release of xattr have --no-xattr, but this is not optimal.
http://vleu.net/shake/shake-0.21.ebuild

Uncle_Psychosis wrote:
Code:
 # ebuild shake-0.20.ebuild compile

!!! ERROR: sys-apps/shake-0.20 failed.
!!! Function src_compile, Line 566, Exitcode 2
!!! emake failed
!!! If you need support, post the topmost build error, NOT this status message.


Any ideas? I'd like to give this a shot!

Sam
You probably didn't installed sys-apps/attr. By the way, are you sure that manually calling "ebuild" is a good idea ?
Back to top
View user's profile Send private message
Uncle_Psychosis
Guru
Guru


Joined: 31 Jan 2004
Posts: 387

PostPosted: Tue Jun 13, 2006 3:28 pm    Post subject: Reply with quote

Ok, when I get home I'll try and install atrr.

As for ebuild, I'm new to trying to install using ebuilds that aren't in the portage repository. I was following the guide here:

http://gentoo-wiki.com/HOWTO_Installing_3rd_Party_Ebuilds

Cheers

Sam
Back to top
View user's profile Send private message
Lokheed
Veteran
Veteran


Joined: 12 Jul 2004
Posts: 1295
Location: /usr/src/linux

PostPosted: Tue Jun 13, 2006 4:26 pm    Post subject: Reply with quote

So how do I know this works? What I mean is, how do I know it did anything? I notice no speed improvements. I also don't understand what "guilty" means. Windows defrag programs show clearly that there is less fragmentation after running them. With shake, I have no clue it did anything.

Maybe you should make things a little more nonambiguious. Not using words like "guilty" because most people won't know what you are talking about. The layout is pretty nonfunctional, seeing tens of files fly past you with no overall rundown on the directory. Have you tried looking into ncurses? It could add a lot of functionality to shake.

Cheers.
_________________
You're not afraid of the dark are you?
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Tue Jun 13, 2006 5:49 pm    Post subject: Reply with quote

Lokheed wrote:
So how do I know this works? What I mean is, how do I know it did anything?

You can make a diff between two
Code:
shake -pvv
, before and after a
Code:
shake
.

Lokheed wrote:
I notice no speed improvements. I also don't understand what "guilty" means. Windows defrag programs show clearly that there is less fragmentation after running them. With shake, I have no clue it did anything.
guilty mean "fragmented". I should add something about this in the documentation.

Lokheed wrote:
Maybe you should make things a little more nonambiguious. Not using words like "guilty" because most people won't know what you are talking about. The layout is pretty nonfunctional, seeing tens of files fly past you with no overall rundown on the directory. Have you tried looking into ncurses? It could add a lot of functionality to shake.
The problem is that showing some nice graphics suppose to scan the whole partition to find the block layout (else, there will be "gaps" everywere).
That said, if you wants to see the disk before and after, try using a tool like http://davtools.sourceforge.net/ .
I *will* add statistics, but IMHO those things should not go in the core of shake. I will provide them as seperate tools.

And it's complectely normal that you don't see a performance improvement : most FS don't fragment significantly, excepted after a long time or in pathologic cases (many bittorent clients).
Back to top
View user's profile Send private message
Lokheed
Veteran
Veteran


Joined: 12 Jul 2004
Posts: 1295
Location: /usr/src/linux

PostPosted: Wed Jun 14, 2006 8:54 pm    Post subject: Reply with quote

If guilty means fragmented, why not simply use fragmented? You are putting more cognitive strain on people using "fancy" words when you could simply state what you mean. There is also no semantic connection between guilty and fragmented.

I guess shake is limited to a tiny select few if it is "normal" to not see a performance improvement. I guess this is why no one using Linux really cares about fragmentation...

Shake seems to be more of a specialty tool than a must have system utility. Not that I miss defragging ;)

Cheers
_________________
You're not afraid of the dark are you?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6  Next
Page 2 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum