View previous topic :: View next topic |
Author |
Message |
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Wed May 17, 2006 2:28 pm Post subject: Shake your files for their own good |
|
|
EDIT : The latest shake (0.29: tarball, Changelog, ebuild) is available at http://vleu.net/shake/.
(Please forgive me my bad English, if I say something arrogant or insulting, it's an probably an error)
I recently hacked something which could help to reduce fragmentation, without using dangerous and low level, and filesystem specific black magic.
It's main idea is that the lack of defragment software on GNU/Linux is mainly due to the lack of interest, due to the fact that filesystems do their job right.
But, even making their best efforts, FS can't perfectly anticipate years of usage (in fact, tests seems to show that just filling sparses files is a problem for ReiserFS 3).
That's when Shake intervene : using a few criteria, such as fragments number, or distance between two files used at the same time, it tries to detect which files are fragmented or misplaced. Then it writes them again to make the FS re-allocate the space.
It's stupid, but seems to works.
In fact, it's even not so bad in comparison to other systems : if the filesystem is journalized, then shake will be. And one can shake only a folder, even while the rest of partition is used. It also avoid useless moves that would occur if filesystem and defragment software were using conflicting algorithms, and risks of corrupting the FS.
If you want to test^Wuse it, you only need to get it's tarball, call "make", and then invocate.. Code: | ./shake -pvv a_file | as root (a --pretend and two --verbose).
There is a manpage, but I can already say that "-p or --pretend" put Shake in read only mode, and that "-vv"; show statistics like this Quote: | FILENAME: start=FILE_START, ideal=IDEAL_START, end=FILE_END, fragc=FRAGMENTS_NUMBER, crumbc=CRUMB_NUMBER, age=FILE_AGE, CULPABILITY |
Where FILE_START, FILE_END refers to file position's on the disk, IDEAL_START is where the file should be (according to the position of files used at the same time), CRUMB are small fragments (by default less than 0.95% of the file, 2 seconds of a 210 seconds OGG Vorbis).
Adding "-o0" allow you to force shake to force file "shaking" (rewrite).
Adding an extra "-v" allow you to see the position of each fragment, to see by yourself how ReiserFS to do his job (very well).
A side effect of shake is that it makes files sparse, so you could gain some disk space.
If you don't want to risk your files, can test it on a special disk image created to show it to my teachers (it was a project for my university at Lyon). If you speak French, you can an also read le dossier.
I don't see it as something you should use every month... maybe a time each year, unless you use intensively sparses files.
It use locks, make backups and some extra checks to detect by himself eventual corruptions, but you should not use it on important datas without reading by yourself the source code.
It is also less efficient on partitions mounted with notail, due to a limitation of FIBMAP, but you should not use notail anyway.
Don't hesitate if you have questions ! I am thinking to a new way of presenting statistics, and to the way I could port it on non-GNU systems.
Last edited by 98111 on Thu Feb 01, 2007 9:40 pm; edited 3 times in total |
|
Back to top |
|
|
steveb Advocate
Joined: 18 Sep 2002 Posts: 4564
|
Posted: Wed May 17, 2006 2:52 pm Post subject: |
|
|
Two things:- What about a ebuild?
- Is that usefull for me (I use SGI XFS and xfs_fsr already does a defragmentation for me)?
cheers
SteveB |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Wed May 17, 2006 5:49 pm Post subject: |
|
|
steveb wrote: | What about a ebuild? |
Good idea.
http://vleu.net/shake/shake-0.17.ebuild
steveb wrote: | Is that usefull for me (I use SGI XFS and xfs_fsr already does a defragmentation for me)? | If I understood xfs_fsr, it makes a job similar to the one of shake, excepted that it use a file system specific call to swap extents, so applications which don't honor locks won't have problems.
So, if I have understood, Shake is probably more efficient, because instead of just copying the 10% most fragmented files during 2 hours, it takes in account many parameters wich should represent more accuratly the effect of fragmentation on the performance. It also takes in account the age of the file, and so should reorganize free space.
To resume : Shake should be smarter and do less/better job. But it's a 0.17 release, and at least "magic values" probably could be improved...
One could also try to write an XFS-specific version of shake_reg() (in executive.c) wich would use XFS-specific calls on this filesystem... |
|
Back to top |
|
|
steveb Advocate
Joined: 18 Sep 2002 Posts: 4564
|
Posted: Wed May 17, 2006 6:07 pm Post subject: |
|
|
Now be nice and post that ebuild to bugs.gentoo.org
cheers
SteveB |
|
Back to top |
|
|
Braempje l33t
Joined: 31 Jan 2003 Posts: 748
|
|
Back to top |
|
|
p0uLp3 n00b
Joined: 26 May 2005 Posts: 14 Location: /dev/proc
|
Posted: Thu May 18, 2006 1:20 pm Post subject: |
|
|
Has anyone done a benchmark for this soft, before and after ? It seems interesting but I haven't my PC avalaible today :/
++ |
|
Back to top |
|
|
jim_crow n00b
Joined: 01 Aug 2004 Posts: 13 Location: Poland
|
Posted: Mon May 22, 2006 4:49 am Post subject: |
|
|
MAN! YOU'RE A GENIUS!!!
i've benchmarked it with one file (700MB).
the file was on reiserfs partition, very fragmented due to constant lack of space.
first i had problems making this little prodigy to work...
finaly i came to this: Code: | # shake -o0 -n0 -vv -S1024000 lola_rennt.avi |
Code: | output: lola_rennt.avi: start=14011849, ideal=0, end=14011840, fragc=9713, crumbc=9712, age=7, guilty |
so! this is it! some hd activity during the process.
then a test: Code: | # shake -pvv lola_rennt.avi |
Code: | lola_rennt.avi: start=193514, ideal=0, end=7390636, fragc=633, crumbc=617, age=0, innocent |
and a benchmark:
copying the file with mc to /dev/null
35MB/s (before shake it was ca 9,5MB/s)
this isn't a slight improvement, it's a rocket!!!
write a man, Man :D like i said, i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...) _________________ gentoo rulez :) |
|
Back to top |
|
|
jim_crow n00b
Joined: 01 Aug 2004 Posts: 13 Location: Poland
|
Posted: Mon May 22, 2006 5:07 am Post subject: |
|
|
second thoughts:
(oh, there is a man, though, i couldn't make it to work right now)
-S0 forces it to work with each file.
-c1 - even less fragmentation? but further position, if i read it right...
my question is: could this program be used to defragment huge amout of files on a partiton, like for example /usr ?
how does it affect partition when used on greater amount of files? i mean, how does it change free space and positions of files?
would it still be performance increase when used on large number of small files?
i can hardly wait to test it :) _________________ gentoo rulez :) |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Mon May 22, 2006 7:39 am Post subject: |
|
|
jim_crow wrote: |
-S0 forces it to work with each file.
-c1 - even less fragmentation? but further position, if i read it right... |
No, in fact -S0 make it thinks that a "big file" is a file of at least 0 bytes.
So all files are considered big, and by default, big files are not shaked (to change that, set -T to something else).
If you want to shake all files, you should use "-S0 -n0 -c1", to tell him that no files are too big (-S0), no files are too young (-n0) and that 0 fragment is already too much (-c0). But I don't think it would be usefull, you should better let (-n) at it's original value, or at 1, because rewriting files that have been wrote this morning is probably useless.
Also, if you don't change -S and you "cd /dev/shm" before shaking, shake will use the "dynamic ramdisk" that is there instead of you harddisk for temporary copy, so it will be faster. But you won't be hable to shake files bigger than your available ram space (in fact, you can change -S to make it equal to your amount of free ram space in kB).
I know it's really not intuitive and that this project should have a decent documentation, interface, and web page.
I'll do all that after my exams (3 June). For now i just changed main.c in a such way that -c0 is considerd as valid (please delete /usr/portage/distfiles/shake-0.17.tar.bz2 and re-emerge it).
jim_crow wrote: | my question is: could this program be used to defragment huge amout of files on a partiton, like for example /usr ? |
Yes ! Shaking the whole partition is the best thing to do IMHO, because it will reorganise free space.
jim_crow wrote: |
how does it affect partition when used on greater amount of files? i mean, how does it change free space and positions of files?
|
In order to reorganize free space, it shakes files of the same directory that are far from each other, and files that are "old".
To change position of files, it just write them again and let the filesystem do his job... it's stupid but it's seems that xfs_fsr do the same (should check again, but don't have time until 3 June - i would also be interested if someone points me to others defragment software source code). I will change it if I find a better way, even if this way is not available on all filesystems (by doing filesystem specifics version of shake_reg() ? (executive.c))... but only after my exams.
jim_crow wrote: |
would it still be performance increase when used on large number of small files?
|
I think so: at least it should reorganise free space and put together files that are used at the same time.
But if you're thinking to /usr/portage, the better way IMHO is to shake the whole partition to reorganize free space, then do a Code: | rm -Rf /usr/portage/* # supress ebuilds
emerge-webrsync # get them again | so you're sure that they will be new files.
jim_crow wrote: | i can hardly wait to test it |
Thanks ! But be carefull, it hasn't yet been tested as it should.
By the way if someone has idea to improve shake (such as secret ioctl's, a better syntax, a better idea for the text output...), he is welcome
Last edited by 98111 on Mon May 22, 2006 7:53 am; edited 1 time in total |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Mon May 22, 2006 7:42 am Post subject: |
|
|
Braempje wrote: | Do you have a project page? This seems interesting... | Thanks !
I'll make one soon after my exams (3 June) ! |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Mon May 22, 2006 7:50 am Post subject: |
|
|
jim_crow wrote: | first i had problems making this little prodigy to work... | My user interface sux... I will try to improve that after my exams.
jim_crow wrote: |
finaly i came to this: Code: | # shake -o0 -n0 -vv -S1024000 lola_rennt.avi |
Code: | output: lola_rennt.avi: start=14011849, ideal=0, end=14011840, fragc=9713, crumbc=9712, age=7, guilty |
so! this is it! some hd activity during the process.
then a test: Code: | # shake -pvv lola_rennt.avi |
Code: | lola_rennt.avi: start=193514, ideal=0, end=7390636, fragc=633, crumbc=617, age=0, innocent |
and a benchmark:
copying the file with mc to /dev/null
35MB/s (before shake it was ca 9,5MB/s)
this isn't a slight improvement, it's a rocket!!!
| Glad it worked for you !
jim_crow wrote: | i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...) | You should find better explanations in my previous post. And Shake should work recursively. If something like "shake -pvv /etc/" don't go through subdirectories, it's a bug, so please test again and tell me ! |
|
Back to top |
|
|
gentoobunny n00b
Joined: 29 Apr 2006 Posts: 18
|
Posted: Mon May 22, 2006 1:52 pm Post subject: |
|
|
jim_crow wrote: | and a benchmark:
copying the file with mc to /dev/null
35MB/s (before shake it was ca 9,5MB/s)
this isn't a slight improvement, it's a rocket!!!
write a man, Man like i said, i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...) |
Now try it again with another file and copy the file with mc twice before using "shake". You will find that it's caching that made the file go 35MB/s the second time. |
|
Back to top |
|
|
jim_crow n00b
Joined: 01 Aug 2004 Posts: 13 Location: Poland
|
Posted: Mon May 22, 2006 6:18 pm Post subject: |
|
|
i get your point. but notice that the file size is over 700MB and it's definately too much for cache on my 512MB machine. so i tried it three times before (time after time), each time getting transfer ca 9,5MB/s. so - it really works, try for yourself... but, the file you try must be really fragmented, or you won't see the real difference.
oh, and i know it reads hd not the cache, because hd led is on almost all the time.
well, i wonder about one more thing:
i have some folders on my reiserfs partition with huge amount of files, and i've noticed that access to those folder with mc and such is very slow and quite loud. i wonder why... _________________ gentoo rulez :) |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Mon May 22, 2006 9:29 pm Post subject: |
|
|
jim_crow wrote: | i have some folders on my reiserfs partition with huge amount of files, and i've noticed that access to those folder with mc and such is very slow and quite loud. i wonder why... | What does Code: | shake -n0 -pv your_folder | grep guilty | wc -l # count fragmented files | , Code: | find your_folder -type f | wc-l # count files | and Code: | mount | grep reiser | grep -v notail # show reiserfs with packed files | return ?
Note it's important to use "shake FOLDER" and not "shake FOLDER/*", because if the argument is a folder name, shake will be hable to take distance between two files in account. |
|
Back to top |
|
|
jim_crow n00b
Joined: 01 Aug 2004 Posts: 13 Location: Poland
|
Posted: Mon May 22, 2006 9:44 pm Post subject: |
|
|
Code: | shake -n0 -pv /usr/lib | grep guilty | wc -l # count fragmented files |
returns lots of Quote: | No such file or directory |
and then 13325.
Code: | find /usr/lib -type f | wc -l # count files |
returns 26389.
Code: | mount | grep reiser | grep -v notail # show reiserfs with packed files |
returns
Quote: | /dev/hda5 on /data type reiserfs (rw) |
well, it seems my /dev/hda1 uses reiserfs without notail option. _________________ gentoo rulez :) |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Mon May 22, 2006 10:15 pm Post subject: |
|
|
jim_crow wrote: | Code: | shake -n0 -pv /usr/lib | grep guilty | wc -l # count fragmented files |
returns lots of Quote: | No such file or directory |
and then 13325. Code: | find /usr/lib -type f | wc -l # count files |
returns 26389. |
I should filter "No such file or directory" (they correspond to invalid symlinks, try to look at app-misc/symlinks)... Anyway, the fact is that you have 13325 misplaced file in this dir (~50% of them). It should be better after a Code: | cd /dev/shm/
shake -n0 /usr/lib | . If it is not, this either mean that shake is too severe with files, or that your empty space is really fragmented (plausible as you don't use notail).
jim_crow wrote: |
Code: | mount | grep reiser | grep -v notail # show reiserfs with packed files |
returns
Quote: | /dev/hda5 on /data type reiserfs (rw) |
well, it seems my /dev/hda1 uses reiserfs without notail option. | You should add it to your /etc/fstab (and maybe shake your whole partition one or two times with -n0) before doing anything else. tail mode kill reiserfs performance.
shaking /usr/lib or a whole partition may be dangerous |
|
Back to top |
|
|
jim_crow n00b
Joined: 01 Aug 2004 Posts: 13 Location: Poland
|
Posted: Tue May 23, 2006 3:52 am Post subject: |
|
|
i checked i do use notail option in fstab for /dev/hda1 and not for /dev/hda5. maybe thats why /dev/hda5 is so slow in folders with lots of files.
i've tried
Code: | cd /dev/shm/
shake -n0 /usr/lib |
and...
it has to be quite dangerous, well it crashed my machine in very weird way (static on display, freeze).
maybe it's because /dev/shm/ ran out of space?
anyway i hope no file was harm. i'm glad to still have my gentoo alive :) (after small fsck at boot)
i've tried to shake -n0 /data/mp3 (over 30GB, /dev/hda5) ...
well, 776 fragmented files before, 702 after... shouldn't it be 0? what's wrong?
oh, btw, it took over 10 minutes, so it would be nice, if you could add some % thing or something...
is it safe to ctrl+c working shake?
--
ok, i did it.
Code: | # shake -n0 -vv /usr |
and took a nap, than a bath. results:
booting time: non-x - shortened from 1:10 to 1:00, total: 1:45 to 1:27.
so, definately - it works.
still i wonder why does it take so damn long for kdm to load :/ fonts? _________________ gentoo rulez :) |
|
Back to top |
|
|
beatryder Veteran
Joined: 08 Apr 2005 Posts: 1138
|
Posted: Tue May 23, 2006 11:34 pm Post subject: |
|
|
So, you wrote this from scratch? I am really impressed. I am watching this topic, I am interested in trying shake, however my computer working for me is very important so I think I will wait for a few versions. _________________ Dont make it idiot proof, make it work.
Neucode.org
<suppressed key> |
|
Back to top |
|
|
shebang_line n00b
Joined: 18 Jul 2004 Posts: 24
|
Posted: Wed May 24, 2006 1:22 am Post subject: |
|
|
This is very impressive, I cant wait for future releases keep up the good work! |
|
Back to top |
|
|
Jinidog Guru
Joined: 26 Nov 2003 Posts: 593 Location: Berlin
|
Posted: Mon May 29, 2006 2:04 pm Post subject: |
|
|
I would want to let this run over my filesystems.
Can you suggest a command, that defrags all files of the root-directory and is as secure as it can be? _________________ Just unused Microsoft-Software is good Microsoft-Software |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Mon May 29, 2006 6:33 pm Post subject: |
|
|
Jinidog wrote: | I would want to let this run over my filesystems.
Can you suggest a command, that defrags all files of the root-directory and is as secure as it can be? | should do that. But I found many bugs, so don't use it before the 3 June, when I'll release the new version.
The main problem on the actual version is that it update the ctime to avoid to shake again recent files. That could conflict with emerge !
The consequence is that emerge *may* let shaked files after an unmerge. If that happened to you, you should use a tools like findcruft. If more than one person had the problem, i'l try to code something to revert ctimes. Else I'll try to find a way to fix it by hand.
Ask me by mail if something gone wrong. I'm really sorry.
To conclude DO NOT USE SHAKE BEFORE V 0.18. |
|
Back to top |
|
|
smurfer n00b
Joined: 19 Feb 2005 Posts: 57 Location: Hannover, Germany
|
Posted: Wed May 31, 2006 8:37 am Post subject: |
|
|
Hi all,
sounds very promising, I think I will give it a try. For those of you who want to defrag whole partitions and have enough free space, there's the old way of
- Copy
- Format
- Copy back
I usually keep one free partition with the size of the largest partition used to do this job. Using two drives makes copying faster.
Regards, smurfer... _________________ true refinement seeks simplicity |
|
Back to top |
|
|
Allochtoon Tux's lil' helper
Joined: 22 Jan 2004 Posts: 123 Location: Rotterdam, The Netherlands
|
Posted: Thu Jun 01, 2006 1:16 pm Post subject: |
|
|
Sounds good, will try it with more stable version |
|
Back to top |
|
|
Merlin-TC l33t
Joined: 16 May 2003 Posts: 603 Location: Germany
|
Posted: Mon Jun 05, 2006 5:14 pm Post subject: |
|
|
I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.
Then I added "user_xattr" as option for the partitions I want to use it on.
Am I missing anything? |
|
Back to top |
|
|
98111 n00b
Joined: 23 Feb 2003 Posts: 41 Location: Lyon, France (aka un_brice )
|
Posted: Mon Jun 05, 2006 7:42 pm Post subject: |
|
|
Merlin-TC wrote: | I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.
Then I added "user_xattr" as option for the partitions I want to use it on.
Am I missing anything? | Sorry, the version you tested wasn't mean to be distributed, I didn't think that someone would try to find the webpage
But it wasn't harmfull.
From now, version distributed on the page http://vleu.net/shake/ will be stable one.
Direct link to the ebuild : http://vleu.net/shake/shake-0.19.ebuild
THE SEVERE BUG IS FIXED (Changelog)
Also, the output should now be readable .
If it works well, I'll try to submit the ebuild for inclusion in Gentoo. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|