| View previous topic :: View next topic |
| Author |
Message |
Merlin-TC
l33t
Joined: 16 May 2003
Posts: 603
Location: Germany
|
 Posted: Tue Jun 06, 2006 4:12 pm Post subject: |
 |
|
Well, I do have version [ebuild R ] sys-apps/shake-0.19 0 kB [1] emerged so that's not it?
Any clues? |
|
| Back to top |
|
 |
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Tue Jun 06, 2006 5:28 pm Post subject: |
|
|
| Merlin-TC wrote: | Well, I do have version [ebuild R ] sys-apps/shake-0.19 0 kB [1] emerged so that's not it?
Any clues? |
Yes: the bugged tarball had the same name as working one, as the buggy wasn't mean to be released (it were just for testing the webpage) 
if you get the new ebuild and then | Code: | rm /usr/portage/distfiles/shake-0.19.tar.bz2
emerge shake |
all should works. |
|
| Back to top |
|
|
Merlin-TC
l33t
Joined: 16 May 2003
Posts: 603
Location: Germany
|
| Posted: Tue Jun 06, 2006 5:52 pm Post subject: |
|
|
Ah ok, seems to work now. Using it on a 200gig drive now  |
|
| Back to top |
|
|
beatryder
Veteran
Joined: 08 Apr 2005
Posts: 1138
|
| Posted: Wed Jun 07, 2006 5:31 am Post subject: |
|
|
| 98111 wrote: | | Merlin-TC wrote: | I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.
Then I added "user_xattr" as option for the partitions I want to use it on.
Am I missing anything? |
Sorry, the version you tested wasn't mean to be distributed, I didn't think that someone would try to find the webpage 
But it wasn't harmfull.
From now, version distributed on the page http://vleu.net/shake/ will be stable one.
Direct link to the ebuild : http://vleu.net/shake/shake-0.19.ebuild
THE SEVERE BUG IS FIXED  (Changelog)
Also, the output should now be readable .
If it works well, I'll try to submit the ebuild for inclusion in Gentoo. |
I just installed shake now, and it gave me the very same problem.
I even tried remounting the partition with the user_xattr flag like you suggested on your website, and it gives me the following:
| Quote: | Septu Windows 98 # shake -p ./
Septu Windows 98 # shake -p ./*
Septu Windows 98 # shake -p --old 0 ./
Septu Windows 98 # shake -p-v --old 0 ./
shake: invalid option -- -
shake: invalid args, aborting
Septu Windows 98 # shake -p -v --old 0 ./
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
0 692098 692120 1 0 32 1 .//vmware.log
0 691682 692084 2 1 32 1 .//floppy
626140 591883 591885 1 0 0 1 .//Windows 98.nvram
657421 691687 691725 2 0 32 1 .//vmware-0.log
526352 591887 591887 1 0 0 1 .//Windows 98.vmx
657423 591889 591889 1 0 0 1 .//Windows 98.vmdk
Septu Windows 98 # shake -v --old 0 ./
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
0 692098 692120 1 0 32 1 .//vmware.log
shake: .//floppy: failed to set position time, check user_xattr: Operation not supported
shake: .//floppy: temporary copy failed
Septu Windows 98 # shake -v --old 0 .
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: ./vmware.log: failed to set position time, check user_xattr: Operation not supported
0 591891 591896 1 0 0 1 ./vmware.log
shake: ./floppy: failed to set position time, check user_xattr: Operation not supported
shake: ./floppy: temporary copy failed
Septu Windows 98 # ls -l
total 1529
-rw------- 1 ryan ryan 8664 Feb 19 09:24 Windows 98.nvram
-rw------- 1 ryan ryan 408 Feb 19 09:19 Windows 98.vmdk
-rw------- 1 ryan ryan 0 Feb 19 09:19 Windows 98.vmsd
-rwxr-xr-x 1 ryan ryan 926 Feb 19 09:24 Windows 98.vmx
-rw------- 1 ryan ryan 1474560 Feb 19 09:24 floppy
-rw-r--r-- 1 ryan ryan 44362 Feb 19 09:21 vmware-0.log
-rw-r--r-- 1 ryan ryan 23492 Feb 19 09:24 vmware.log
|
Also this is on a reiserfs partition. does that matter?
_________________
Dont make it idiot proof, make it work.
Neucode.org
<suppressed key> |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Wed Jun 07, 2006 10:52 am Post subject: |
|
|
| beatryder wrote: | | Quote: | IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
0 692098 692120 1 0 32 1 .//vmware.log
shake: .//floppy: failed to set position time, check user_xattr: Operation not supported
shake: .//floppy: temporary copy failed
Septu Windows 98 # shake -v --old 0 .
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: ./vmware.log: failed to set position time, check user_xattr: Operation not supported
0 591891 591896 1 0 0 1 ./vmware.log
shake: ./floppy: failed to set position time, check user_xattr: Operation not supported
Also this is on a reiserfs partition. does that matter? |
|
There is 2 possibility : either your kernel doesn't support user_xattr, or you use the old version (as Merlin-TC was doing).
To know about this, please remove "/usr/portage/distfiles/shake*", make sure you have the latest ebuild, and emerge shake again.
If it still don't works, I would be really happy to see the output of "mount", and the last line of dmesg  .
The next version (which will probably be released this night, or tommorrow ) will have a switch to disable user_xattr, but I would like to have them tested before, because they are really usefull. |
|
| Back to top |
|
|
Merlin-TC
l33t
Joined: 16 May 2003
Posts: 603
Location: Germany
|
| Posted: Wed Jun 07, 2006 2:16 pm Post subject: |
|
|
I have a little bit off topic question about xattr. Does it costs speed or does it improve the speed for programs that support it?
Does it help "gamin" as well? |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Wed Jun 07, 2006 7:09 pm Post subject: |
|
|
| Merlin-TC wrote: | I have a little bit off topic question about xattr. Does it costs speed or does it improve the speed for programs that support it?
Does it help "gamin" as well? |
eXtented ATTRibutes consists in pair of name/values attached to a file.
By exemple, Shake use them for recording the time of space allocation, using the pair "shake_ptime"/time (with time coded in little endian on four bytes).
A file manager could use them to associate preview or mimetype with files, but I don't see how it could help gamin.
And it doesn't cost speed, just a bit of HD space, 5megs for all ptimes of files on my system. |
|
| Back to top |
|
|
Merlin-TC
l33t
Joined: 16 May 2003
Posts: 603
Location: Germany
|
| Posted: Fri Jun 09, 2006 3:38 pm Post subject: |
|
|
Is it possible that xattr takes up a huge amount of space when the partition consists of many small files?
I have one partition where the portage tree is and I just formated it and the tree takes up 122MiB but df -h shows /sda5 4.7G 559M 4.2G 12% /usr/portage.
So it must have a huge overhead.
Is there any way to remove the xattr stuff? |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Sat Jun 10, 2006 12:54 am Post subject: |
|
|
| Merlin-TC wrote: | Is it possible that xattr takes up a huge amount of space when the partition consists of many small files?
I have one partition where the portage tree is and I just formated it and the tree takes up 122MiB but df -h shows /sda5 4.7G 559M 4.2G 12% /usr/portage.
So it must have a huge overhead.
Is there any way to remove the xattr stuff? |
I don't think that the difference is due to xattr. Have you tried to do a "df -h" when there's no files ?
I can write something to remove xattr, but before I would like to ask you something... could you delete this portage tree, mount this partition without user_xattr, re-get the portage three and tell if "df -h" return something different ?
I think that wasted space is not wasted due to xattr, but due to internal structures of reiserfs. |
|
| Back to top |
|
|
bunder
Bodhisattva
Joined: 10 Apr 2004
Posts: 5934
|
| Posted: Sat Jun 10, 2006 2:15 am Post subject: |
|
|
i tried running it on /usr/portage and i got a whack of "too many open files". is that directory too big to use shake on?
edit: i have a feature request. can you put in a --one-file-system option? say i wanted to run it on just /usr (not /usr/portage, or /usr/portage/distfiles which is running over nfs).
nice script, i'll play around with it on my systems to see if it makes anything faster.
_________________
| Neddyseagoon wrote: | | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017 |
|
| Back to top |
|
|
Merlin-TC
l33t
Joined: 16 May 2003
Posts: 603
Location: Germany
|
| Posted: Sat Jun 10, 2006 5:45 am Post subject: |
|
|
You were right, it had nothing to do with xattr.
I was mounting the partition with the "notail" option (reiserfs) and this caused the huge overhead. There is also another option calles atime, is this somehow connected with xattr? |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Sat Jun 10, 2006 10:03 am Post subject: |
|
|
| Merlin-TC wrote: | You were right, it had nothing to do with xattr.
I was mounting the partition with the "notail" option (reiserfs) and this caused the huge overhead. There is also another option calles atime, is this somehow connected with xattr? |
tail mode spare space, but it is however recomended to disable it on reiserfs<4, which cannot manage it properly.
Atimes helps shake to see the correlation between files.
Last edited by 98111 on Sat Jun 10, 2006 10:09 am; edited 2 times in total |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Sat Jun 10, 2006 10:08 am Post subject: |
|
|
| bunder wrote: | | i tried running it on /usr/portage and i got a whack of "too many open files". is that directory too big to use shake on? |
 ! No, shake open only three files at once, whatever is the size of the directory. I have already used it on /usr/portage...
| bunder wrote: | | edit: i have a feature request. can you put in a --one-file-system option? say i wanted to run it on just /usr (not /usr/portage, or /usr/portage/distfiles which is running over nfs). |
Good idea, it will be done arround tomorrow.
| bunder wrote: | | nice script, i'll play around with it on my systems to see if it makes anything faster. |
OK ^^. If you have some time to experiments with default values... I know they could be improved because I just set them arbitrary |
|
| Back to top |
|
|
Cintra
Advocate
Joined: 03 Apr 2004
Posts: 2111
Location: Norway
|
| Posted: Sat Jun 10, 2006 12:56 pm Post subject: |
|
|
Very nice, well done 98111
I just ran it on the /root folder of my 2nd Gentoo copy and it works like a charm! It cleaned up a lot of 'guilty' files, and everything still seems to be working fine on that copy
Btw, one thing that could make shake even nicer is a summary after 'shake -pvv <whatever>' (like XP's analyze)..
Mvh
Amost forgot.. does it work only on Linux partitions?
_________________
"I am not bound to please thee with my answers" W.S. |
|
| Back to top |
|
|
bunder
Bodhisattva
Joined: 10 Apr 2004
Posts: 5934
|
| Posted: Sat Jun 10, 2006 6:10 pm Post subject: |
|
|
i found out why i was getting the "too many open files" errors. i didn't happen to look at the path of the files before, but those were the ones on the nfs share.
_________________
| Neddyseagoon wrote: | | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017 |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Sat Jun 10, 2006 10:19 pm Post subject: |
|
|
| bunder wrote: | | i found out why i was getting the "too many open files" errors. i didn't happen to look at the path of the files before, but those were the ones on the nfs share. |
Effectively, I was handling badly the failure of get_testimony... it should be fixed !
I also added --one-file-system, fixed some little bugs, and trapped signals during the file copy (Changelog ebuild).
Last edited by 98111 on Sat Jun 10, 2006 10:36 pm; edited 1 time in total |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Sat Jun 10, 2006 10:32 pm Post subject: |
|
|
| Cintra wrote: | | Very nice, well done 98111 |
Thanks
| Cintra wrote: | | Btw, one thing that could make shake even nicer is a summary after 'shake -pvv <whatever>' (like XP's analyze).. |
In long term (newt weeks), I will try to contact authors of davtools; or to make a python module that will wrap arround shake and allow to add better UIs while keeping the core source code light.
During this time, adding some statistics in the main source code to remove it just after seems a bad idea...
| Cintra wrote: | | Amost forgot.. does it work only on Linux partitions? |
I would love to have it running on all posix-compliant systems, but that will require a little works.
And it's efficacity depends of the quality of the filesystem implementation, so using it on vfat or ntfs from Linux is definitively a bad idea. |
|
| Back to top |
|
|
bunder
Bodhisattva
Joined: 10 Apr 2004
Posts: 5934
|
| Posted: Sun Jun 11, 2006 5:03 am Post subject: |
|
|
my box crashed when running the new version. i'm not sure if it was because i ran out of /dev/shm or if it was because the filesystem was mounted. just something to look for
edit: maybe it didn't crash. it still responds to pings, so i bet the kernel killed sshd.
_________________
| Neddyseagoon wrote: | | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017
Last edited by bunder on Sun Jun 11, 2006 8:16 am; edited 1 time in total |
|
| Back to top |
|
|
beatryder
Veteran
Joined: 08 Apr 2005
Posts: 1138
|
| Posted: Sun Jun 11, 2006 6:15 am Post subject: |
|
|
| 98111 wrote: | | beatryder wrote: | | Quote: | IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: .//vmware.log: failed to set position time, check user_xattr: Operation not supported
0 692098 692120 1 0 32 1 .//vmware.log
shake: .//floppy: failed to set position time, check user_xattr: Operation not supported
shake: .//floppy: temporary copy failed
Septu Windows 98 # shake -v --old 0 .
IDEAL START END FRAGC CRUMBC AGE GUILTY NAME
shake: ./vmware.log: failed to set position time, check user_xattr: Operation not supported
0 591891 591896 1 0 0 1 ./vmware.log
shake: ./floppy: failed to set position time, check user_xattr: Operation not supported
Also this is on a reiserfs partition. does that matter? |
|
There is 2 possibility : either your kernel doesn't support user_xattr, or you use the old version (as Merlin-TC was doing).
To know about this, please remove "/usr/portage/distfiles/shake*", make sure you have the latest ebuild, and emerge shake again.
If it still don't works, I would be really happy to see the output of "mount", and the last line of dmesg .
The next version (which will probably be released this night, or tommorrow ) will have a switch to disable user_xattr, but I would like to have them tested before, because they are really usefull. |
I have not used shaker since AFTER you said the major bug was repaired. I have the xattr in my kernel, and have mounted with the user_xattr option
_________________
Dont make it idiot proof, make it work.
Neucode.org
<suppressed key> |
|
| Back to top |
|
|
Uncle_Psychosis
Guru
Joined: 31 Jan 2004
Posts: 387
|
| Posted: Mon Jun 12, 2006 5:13 pm Post subject: |
|
|
| Code: | # ebuild shake-0.20.ebuild compile
>>> md5 files ;-) shake-0.17.ebuild
>>> md5 files ;-) shake-0.20.ebuild
>>> md5 files ;-) files/digest-shake-0.20
>>> md5 src_uri ;-) shake-0.20.tar.bz2
>>> Checking shake-0.20.tar.bz2's mtime...
>>> WORKDIR is up-to-date, keeping...
gcc -O2 -march=athlon-xp -pipe -std=gnu99 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Werror -pedantic-errors -Wconversion -Wpointer-arith -Wbad-function-cast -Wcast-align -D_XOPEN_SOURCE=600 -O2 -D_POSIX_C_SOURCE=200112L -O0 -ggdb3 -DNDEBUG -c linux.c -o linux.o
linux.c:23:43: attr/attributes.h: No such file or directory
linux.c: In function `set_ptime':
linux.c:44: warning: implicit declaration of function `attr_setf'
linux.c:44: error: `ATTR_DONTFOLLOW' undeclared (first use in this function)
linux.c:44: error: (Each undeclared identifier is reported only once
linux.c:44: error: for each function it appears in.)
linux.c: In function `get_ptime':
linux.c:56: warning: implicit declaration of function `attr_getf'
linux.c:56: error: `ATTR_DONTFOLLOW' undeclared (first use in this function)
make: *** [linux.o] Error 1
!!! ERROR: sys-apps/shake-0.20 failed.
!!! Function src_compile, Line 566, Exitcode 2
!!! emake failed
!!! If you need support, post the topmost build error, NOT this status message. |
Any ideas? I'd like to give this a shot!
Sam [/code] |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Tue Jun 13, 2006 3:23 pm Post subject: |
|
|
| beatryder wrote: | | I have not used shaker since AFTER you said the major bug was repaired. I have the xattr in my kernel, and have mounted with the user_xattr option |
Could you try
| Code: | touch testfile
attr -s testattr -V testval testfile |
?
Anyway, the new release of xattr have --no-xattr, but this is not optimal.
http://vleu.net/shake/shake-0.21.ebuild
| Uncle_Psychosis wrote: | | Code: | # ebuild shake-0.20.ebuild compile
!!! ERROR: sys-apps/shake-0.20 failed.
!!! Function src_compile, Line 566, Exitcode 2
!!! emake failed
!!! If you need support, post the topmost build error, NOT this status message. |
Any ideas? I'd like to give this a shot!
Sam |
You probably didn't installed sys-apps/attr. By the way, are you sure that manually calling "ebuild" is a good idea ? |
|
| Back to top |
|
|
Uncle_Psychosis
Guru
Joined: 31 Jan 2004
Posts: 387
|
| Posted: Tue Jun 13, 2006 3:28 pm Post subject: |
|
|
Ok, when I get home I'll try and install atrr.
As for ebuild, I'm new to trying to install using ebuilds that aren't in the portage repository. I was following the guide here:
http://gentoo-wiki.com/HOWTO_Installing_3rd_Party_Ebuilds
Cheers
Sam |
|
| Back to top |
|
|
Lokheed
Veteran
Joined: 12 Jul 2004
Posts: 1295
Location: /usr/src/linux
|
| Posted: Tue Jun 13, 2006 4:26 pm Post subject: |
|
|
So how do I know this works? What I mean is, how do I know it did anything? I notice no speed improvements. I also don't understand what "guilty" means. Windows defrag programs show clearly that there is less fragmentation after running them. With shake, I have no clue it did anything.
Maybe you should make things a little more nonambiguious. Not using words like "guilty" because most people won't know what you are talking about. The layout is pretty nonfunctional, seeing tens of files fly past you with no overall rundown on the directory. Have you tried looking into ncurses? It could add a lot of functionality to shake.
Cheers.
_________________
You're not afraid of the dark are you? |
|
| Back to top |
|
|
98111
n00b
Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )
|
| Posted: Tue Jun 13, 2006 5:49 pm Post subject: |
|
|
| Lokheed wrote: | | So how do I know this works? What I mean is, how do I know it did anything? |
You can make a diff between two , before and after a .
| Lokheed wrote: | | I notice no speed improvements. I also don't understand what "guilty" means. Windows defrag programs show clearly that there is less fragmentation after running them. With shake, I have no clue it did anything. |
guilty mean "fragmented". I should add something about this in the documentation.
| Lokheed wrote: | | Maybe you should make things a little more nonambiguious. Not using words like "guilty" because most people won't know what you are talking about. The layout is pretty nonfunctional, seeing tens of files fly past you with no overall rundown on the directory. Have you tried looking into ncurses? It could add a lot of functionality to shake. |
The problem is that showing some nice graphics suppose to scan the whole partition to find the block layout (else, there will be "gaps" everywere).
That said, if you wants to see the disk before and after, try using a tool like http://davtools.sourceforge.net/ .
I *will* add statistics, but IMHO those things should not go in the core of shake. I will provide them as seperate tools.
And it's complectely normal that you don't see a performance improvement : most FS don't fragment significantly, excepted after a long time or in pathologic cases (many bittorent clients). |
|
| Back to top |
|
|
Lokheed
Veteran
Joined: 12 Jul 2004
Posts: 1295
Location: /usr/src/linux
|
| Posted: Wed Jun 14, 2006 8:54 pm Post subject: |
|
|
If guilty means fragmented, why not simply use fragmented? You are putting more cognitive strain on people using "fancy" words when you could simply state what you mean. There is also no semantic connection between guilty and fragmented.
I guess shake is limited to a tiny select few if it is "normal" to not see a performance improvement. I guess this is why no one using Linux really cares about fragmentation...
Shake seems to be more of a specialty tool than a must have system utility. Not that I miss defragging
Cheers
_________________
You're not afraid of the dark are you? |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
