Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Shake your files for their own good
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3, 4, 5, 6  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Wed May 17, 2006 2:28 pm    Post subject: Shake your files for their own good Reply with quote

EDIT : The latest shake (0.29: tarball, Changelog, ebuild) is available at http://vleu.net/shake/.

(Please forgive me my bad English, if I say something arrogant or insulting, it's an probably an error)
I recently hacked something which could help to reduce fragmentation, without using dangerous and low level, and filesystem specific black magic.

It's main idea is that the lack of defragment software on GNU/Linux is mainly due to the lack of interest, due to the fact that filesystems do their job right.
But, even making their best efforts, FS can't perfectly anticipate years of usage (in fact, tests seems to show that just filling sparses files is a problem for ReiserFS 3).

That's when Shake intervene : using a few criteria, such as fragments number, or distance between two files used at the same time, it tries to detect which files are fragmented or misplaced. Then it writes them again to make the FS re-allocate the space.
It's stupid, but seems to works.
In fact, it's even not so bad in comparison to other systems : if the filesystem is journalized, then shake will be. And one can shake only a folder, even while the rest of partition is used. It also avoid useless moves that would occur if filesystem and defragment software were using conflicting algorithms, and risks of corrupting the FS.

If you want to test^Wuse it, you only need to get it's tarball, call "make", and then invocate..
Code:
./shake -pvv a_file
as root (a --pretend and two --verbose).
There is a manpage, but I can already say that "-p or --pretend" put Shake in read only mode, and that "-vv"; show statistics like this
Quote:
FILENAME: start=FILE_START, ideal=IDEAL_START, end=FILE_END, fragc=FRAGMENTS_NUMBER, crumbc=CRUMB_NUMBER, age=FILE_AGE, CULPABILITY

Where FILE_START, FILE_END refers to file position's on the disk, IDEAL_START is where the file should be (according to the position of files used at the same time), CRUMB are small fragments (by default less than 0.95% of the file, 2 seconds of a 210 seconds OGG Vorbis).
Adding "-o0" allow you to force shake to force file "shaking" (rewrite).
Adding an extra "-v" allow you to see the position of each fragment, to see by yourself how ReiserFS to do his job (very well).

A side effect of shake is that it makes files sparse, so you could gain some disk space.

If you don't want to risk your files, can test it on a special disk image created to show it to my teachers (it was a project for my university at Lyon). If you speak French, you can an also read le dossier.

I don't see it as something you should use every month... maybe a time each year, unless you use intensively sparses files.
It use locks, make backups and some extra checks to detect by himself eventual corruptions, but you should not use it on important datas without reading by yourself the source code.
It is also less efficient on partitions mounted with notail, due to a limitation of FIBMAP, but you should not use notail anyway.

Don't hesitate if you have questions ! I am thinking to a new way of presenting statistics, and to the way I could port it on non-GNU systems.


Last edited by 98111 on Thu Feb 01, 2007 9:40 pm; edited 3 times in total
Back to top
View user's profile Send private message
steveb
Advocate
Advocate


Joined: 18 Sep 2002
Posts: 4564

PostPosted: Wed May 17, 2006 2:52 pm    Post subject: Reply with quote

Two things:
  • What about a ebuild?
  • Is that usefull for me (I use SGI XFS and xfs_fsr already does a defragmentation for me)?



cheers

SteveB
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Wed May 17, 2006 5:49 pm    Post subject: Reply with quote

steveb wrote:
What about a ebuild?

Good idea.
http://vleu.net/shake/shake-0.17.ebuild

steveb wrote:
Is that usefull for me (I use SGI XFS and xfs_fsr already does a defragmentation for me)?
If I understood xfs_fsr, it makes a job similar to the one of shake, excepted that it use a file system specific call to swap extents, so applications which don't honor locks won't have problems.
So, if I have understood, Shake is probably more efficient, because instead of just copying the 10% most fragmented files during 2 hours, it takes in account many parameters wich should represent more accuratly the effect of fragmentation on the performance. It also takes in account the age of the file, and so should reorganize free space.

To resume : Shake should be smarter and do less/better job. But it's a 0.17 release, and at least "magic values" probably could be improved...
One could also try to write an XFS-specific version of shake_reg() (in executive.c) wich would use XFS-specific calls on this filesystem...
Back to top
View user's profile Send private message
steveb
Advocate
Advocate


Joined: 18 Sep 2002
Posts: 4564

PostPosted: Wed May 17, 2006 6:07 pm    Post subject: Reply with quote

98111 wrote:
steveb wrote:
What about a ebuild?

Good idea.
http://vleu.net/shake/shake-0.17.ebuild
Now be nice and post that ebuild to bugs.gentoo.org ;)

cheers

SteveB
Back to top
View user's profile Send private message
Braempje
l33t
l33t


Joined: 31 Jan 2003
Posts: 748

PostPosted: Thu May 18, 2006 6:39 am    Post subject: Reply with quote

Do you have a project page? This seems interesting...
_________________
Dictionary of the Flemish Sign Language - Woordenboek Vlaamse Gebarentaal
Back to top
View user's profile Send private message
p0uLp3
n00b
n00b


Joined: 26 May 2005
Posts: 14
Location: /dev/proc

PostPosted: Thu May 18, 2006 1:20 pm    Post subject: Reply with quote

Has anyone done a benchmark for this soft, before and after ? It seems interesting but I haven't my PC avalaible today :/

++
Back to top
View user's profile Send private message
jim_crow
n00b
n00b


Joined: 01 Aug 2004
Posts: 13
Location: Poland

PostPosted: Mon May 22, 2006 4:49 am    Post subject: Reply with quote

MAN! YOU'RE A GENIUS!!!

i've benchmarked it with one file (700MB).

the file was on reiserfs partition, very fragmented due to constant lack of space.

first i had problems making this little prodigy to work...

finaly i came to this:
Code:
# shake -o0 -n0 -vv -S1024000 lola_rennt.avi

Code:
output: lola_rennt.avi: start=14011849, ideal=0, end=14011840, fragc=9713, crumbc=9712, age=7, guilty


so! this is it! some hd activity during the process.

then a test:
Code:
# shake -pvv lola_rennt.avi

Code:
lola_rennt.avi: start=193514, ideal=0, end=7390636, fragc=633, crumbc=617, age=0, innocent


and a benchmark:
copying the file with mc to /dev/null
35MB/s (before shake it was ca 9,5MB/s)

this isn't a slight improvement, it's a rocket!!!

write a man, Man :D like i said, i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...)
_________________
gentoo rulez :)
Back to top
View user's profile Send private message
jim_crow
n00b
n00b


Joined: 01 Aug 2004
Posts: 13
Location: Poland

PostPosted: Mon May 22, 2006 5:07 am    Post subject: Reply with quote

second thoughts:

(oh, there is a man, though, i couldn't make it to work right now)

-S0 forces it to work with each file.
-c1 - even less fragmentation? but further position, if i read it right...

my question is: could this program be used to defragment huge amout of files on a partiton, like for example /usr ?

how does it affect partition when used on greater amount of files? i mean, how does it change free space and positions of files?

would it still be performance increase when used on large number of small files?

i can hardly wait to test it :)
_________________
gentoo rulez :)
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Mon May 22, 2006 7:39 am    Post subject: Reply with quote

jim_crow wrote:

-S0 forces it to work with each file.
-c1 - even less fragmentation? but further position, if i read it right...

No, in fact -S0 make it thinks that a "big file" is a file of at least 0 bytes.
So all files are considered big, and by default, big files are not shaked (to change that, set -T to something else).
If you want to shake all files, you should use "-S0 -n0 -c1", to tell him that no files are too big (-S0), no files are too young (-n0) and that 0 fragment is already too much (-c0). But I don't think it would be usefull, you should better let (-n) at it's original value, or at 1, because rewriting files that have been wrote this morning is probably useless.
Also, if you don't change -S and you "cd /dev/shm" before shaking, shake will use the "dynamic ramdisk" that is there instead of you harddisk for temporary copy, so it will be faster. But you won't be hable to shake files bigger than your available ram space (in fact, you can change -S to make it equal to your amount of free ram space in kB).
I know it's really not intuitive and that this project should have a decent documentation, interface, and web page.
I'll do all that after my exams (3 June). For now i just changed main.c in a such way that -c0 is considerd as valid (please delete /usr/portage/distfiles/shake-0.17.tar.bz2 and re-emerge it).

jim_crow wrote:
my question is: could this program be used to defragment huge amout of files on a partiton, like for example /usr ?

Yes ! Shaking the whole partition is the best thing to do IMHO, because it will reorganise free space.

jim_crow wrote:

how does it affect partition when used on greater amount of files? i mean, how does it change free space and positions of files?

In order to reorganize free space, it shakes files of the same directory that are far from each other, and files that are "old".
To change position of files, it just write them again and let the filesystem do his job... it's stupid but it's seems that xfs_fsr do the same (should check again, but don't have time until 3 June - i would also be interested if someone points me to others defragment software source code). I will change it if I find a better way, even if this way is not available on all filesystems (by doing filesystem specifics version of shake_reg() ? (executive.c))... but only after my exams.

jim_crow wrote:

would it still be performance increase when used on large number of small files?

I think so: at least it should reorganise free space and put together files that are used at the same time.
But if you're thinking to /usr/portage, the better way IMHO is to shake the whole partition to reorganize free space, then do a
Code:
rm -Rf /usr/portage/* # supress ebuilds
emerge-webrsync # get them again
so you're sure that they will be new files.


jim_crow wrote:
i can hardly wait to test it :)

Thanks :) ! But be carefull, it hasn't yet been tested as it should.

By the way if someone has idea to improve shake (such as secret ioctl's, a better syntax, a better idea for the text output...), he is welcome :wink:


Last edited by 98111 on Mon May 22, 2006 7:53 am; edited 1 time in total
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Mon May 22, 2006 7:42 am    Post subject: Reply with quote

Braempje wrote:
Do you have a project page? This seems interesting...
Thanks ! :D
I'll make one soon after my exams (3 June) !
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Mon May 22, 2006 7:50 am    Post subject: Reply with quote

jim_crow wrote:
first i had problems making this little prodigy to work...
My user interface sux... I will try to improve that after my exams.

jim_crow wrote:

finaly i came to this:
Code:
# shake -o0 -n0 -vv -S1024000 lola_rennt.avi

Code:
output: lola_rennt.avi: start=14011849, ideal=0, end=14011840, fragc=9713, crumbc=9712, age=7, guilty


so! this is it! some hd activity during the process.

then a test:
Code:
# shake -pvv lola_rennt.avi

Code:
lola_rennt.avi: start=193514, ideal=0, end=7390636, fragc=633, crumbc=617, age=0, innocent


and a benchmark:
copying the file with mc to /dev/null
35MB/s (before shake it was ca 9,5MB/s)

this isn't a slight improvement, it's a rocket!!!
Glad it worked for you ! 8)

jim_crow wrote:
i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...)
You should find better explanations in my previous post. And Shake should work recursively. If something like "shake -pvv /etc/" don't go through subdirectories, it's a bug, so please test again and tell me !
Back to top
View user's profile Send private message
gentoobunny
n00b
n00b


Joined: 29 Apr 2006
Posts: 18

PostPosted: Mon May 22, 2006 1:52 pm    Post subject: Reply with quote

jim_crow wrote:
and a benchmark:
copying the file with mc to /dev/null
35MB/s (before shake it was ca 9,5MB/s)

this isn't a slight improvement, it's a rocket!!!

write a man, Man :D like i said, i had problems to make it run with my file, it could be a little easier, and, of course possibility to run on /dir/* recursive (yes, i know i could write a script myself...)

Now try it again with another file and copy the file with mc twice before using "shake". You will find that it's caching that made the file go 35MB/s the second time.
Back to top
View user's profile Send private message
jim_crow
n00b
n00b


Joined: 01 Aug 2004
Posts: 13
Location: Poland

PostPosted: Mon May 22, 2006 6:18 pm    Post subject: Reply with quote

i get your point. but notice that the file size is over 700MB and it's definately too much for cache on my 512MB machine. so i tried it three times before (time after time), each time getting transfer ca 9,5MB/s. so - it really works, try for yourself... but, the file you try must be really fragmented, or you won't see the real difference.

oh, and i know it reads hd not the cache, because hd led is on almost all the time.

well, i wonder about one more thing:

i have some folders on my reiserfs partition with huge amount of files, and i've noticed that access to those folder with mc and such is very slow and quite loud. i wonder why...
_________________
gentoo rulez :)
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Mon May 22, 2006 9:29 pm    Post subject: Reply with quote

jim_crow wrote:
i have some folders on my reiserfs partition with huge amount of files, and i've noticed that access to those folder with mc and such is very slow and quite loud. i wonder why...
What does
Code:
shake -n0 -pv your_folder | grep guilty | wc -l # count fragmented files
,
Code:
find your_folder -type f | wc-l # count files
and
Code:
mount | grep reiser | grep -v notail # show reiserfs with packed files
return ?
Note it's important to use "shake FOLDER" and not "shake FOLDER/*", because if the argument is a folder name, shake will be hable to take distance between two files in account.
Back to top
View user's profile Send private message
jim_crow
n00b
n00b


Joined: 01 Aug 2004
Posts: 13
Location: Poland

PostPosted: Mon May 22, 2006 9:44 pm    Post subject: Reply with quote

Code:
shake -n0 -pv /usr/lib | grep guilty | wc -l # count fragmented files

returns lots of
Quote:
No such file or directory

and then 13325.
Code:
find /usr/lib -type f | wc -l # count files

returns 26389.
Code:
mount | grep reiser | grep -v notail # show reiserfs with packed files

returns
Quote:
/dev/hda5 on /data type reiserfs (rw)


well, it seems my /dev/hda1 uses reiserfs without notail option.
_________________
gentoo rulez :)
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Mon May 22, 2006 10:15 pm    Post subject: Reply with quote

jim_crow wrote:
Code:
shake -n0 -pv /usr/lib | grep guilty | wc -l # count fragmented files

returns lots of
Quote:
No such file or directory

and then 13325.
Code:
find /usr/lib -type f | wc -l # count files

returns 26389.

I should filter "No such file or directory" (they correspond to invalid symlinks, try to look at app-misc/symlinks)... Anyway, the fact is that you have 13325 misplaced file in this dir (~50% of them). It should be better after a
Code:
 cd /dev/shm/
shake -n0 /usr/lib
. If it is not, this either mean that shake is too severe with files, or that your empty space is really fragmented (plausible as you don't use notail).

jim_crow wrote:

Code:
mount | grep reiser | grep -v notail # show reiserfs with packed files

returns
Quote:
/dev/hda5 on /data type reiserfs (rw)

well, it seems my /dev/hda1 uses reiserfs without notail option.
You should add it to your /etc/fstab (and maybe shake your whole partition one or two times with -n0) before doing anything else. tail mode kill reiserfs performance.

:!: shaking /usr/lib or a whole partition may be dangerous :!:
Back to top
View user's profile Send private message
jim_crow
n00b
n00b


Joined: 01 Aug 2004
Posts: 13
Location: Poland

PostPosted: Tue May 23, 2006 3:52 am    Post subject: Reply with quote

i checked i do use notail option in fstab for /dev/hda1 and not for /dev/hda5. maybe thats why /dev/hda5 is so slow in folders with lots of files.

i've tried
Code:
cd /dev/shm/
shake -n0 /usr/lib

and...

it has to be quite dangerous, well it crashed my machine in very weird way (static on display, freeze).

maybe it's because /dev/shm/ ran out of space?

anyway i hope no file was harm. i'm glad to still have my gentoo alive :) (after small fsck at boot)

i've tried to shake -n0 /data/mp3 (over 30GB, /dev/hda5) ...
well, 776 fragmented files before, 702 after... shouldn't it be 0? what's wrong?

oh, btw, it took over 10 minutes, so it would be nice, if you could add some % thing or something...

is it safe to ctrl+c working shake?

--

ok, i did it.
Code:
# shake -n0 -vv /usr


and took a nap, than a bath. results:

booting time: non-x - shortened from 1:10 to 1:00, total: 1:45 to 1:27.

so, definately - it works.

still i wonder why does it take so damn long for kdm to load :/ fonts?
_________________
gentoo rulez :)
Back to top
View user's profile Send private message
beatryder
Veteran
Veteran


Joined: 08 Apr 2005
Posts: 1138

PostPosted: Tue May 23, 2006 11:34 pm    Post subject: Reply with quote

So, you wrote this from scratch? I am really impressed. I am watching this topic, I am interested in trying shake, however my computer working for me is very important so I think I will wait for a few versions.
_________________
Dont make it idiot proof, make it work.
Neucode.org
<suppressed key>
Back to top
View user's profile Send private message
shebang_line
n00b
n00b


Joined: 18 Jul 2004
Posts: 24

PostPosted: Wed May 24, 2006 1:22 am    Post subject: Reply with quote

This is very impressive, I can’t wait for future releases :-) keep up the good work!
Back to top
View user's profile Send private message
Jinidog
Guru
Guru


Joined: 26 Nov 2003
Posts: 593
Location: Berlin

PostPosted: Mon May 29, 2006 2:04 pm    Post subject: Reply with quote

I would want to let this run over my filesystems.
Can you suggest a command, that defrags all files of the root-directory and is as secure as it can be?
_________________
Just unused Microsoft-Software is good Microsoft-Software
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Mon May 29, 2006 6:33 pm    Post subject: Reply with quote

Jinidog wrote:
I would want to let this run over my filesystems.
Can you suggest a command, that defrags all files of the root-directory and is as secure as it can be?
Code:
shake /
should do that. But I found many bugs, so don't use it before the 3 June, when I'll release the new version.
The main problem on the actual version is that it update the ctime to avoid to shake again recent files. That could conflict with emerge !
The consequence is that emerge *may* let shaked files after an unmerge. If that happened to you, you should use a tools like findcruft. If more than one person had the problem, i'l try to code something to revert ctimes. Else I'll try to find a way to fix it by hand.
Ask me by mail if something gone wrong. I'm really sorry.

To conclude DO NOT USE SHAKE BEFORE V 0.18.
Back to top
View user's profile Send private message
smurfer
n00b
n00b


Joined: 19 Feb 2005
Posts: 57
Location: Hannover, Germany

PostPosted: Wed May 31, 2006 8:37 am    Post subject: Reply with quote

Hi all,

sounds very promising, I think I will give it a try. For those of you who want to defrag whole partitions and have enough free space, there's the old way of

- Copy
- Format
- Copy back

I usually keep one free partition with the size of the largest partition used to do this job. Using two drives makes copying faster.

Regards, smurfer...
_________________
true refinement seeks simplicity
Back to top
View user's profile Send private message
Allochtoon
Tux's lil' helper
Tux's lil' helper


Joined: 22 Jan 2004
Posts: 123
Location: Rotterdam, The Netherlands

PostPosted: Thu Jun 01, 2006 1:16 pm    Post subject: Reply with quote

Sounds good, will try it with more stable version :)
Back to top
View user's profile Send private message
Merlin-TC
l33t
l33t


Joined: 16 May 2003
Posts: 603
Location: Germany

PostPosted: Mon Jun 05, 2006 5:14 pm    Post subject: Reply with quote

I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.

Then I added "user_xattr" as option for the partitions I want to use it on.
Am I missing anything?
Back to top
View user's profile Send private message
98111
n00b
n00b


Joined: 23 Feb 2003
Posts: 41
Location: Lyon, France (aka un_brice )

PostPosted: Mon Jun 05, 2006 7:42 pm    Post subject: Reply with quote

Merlin-TC wrote:
I still get the error "failed to set position time, check user_xattr: Operation not supported" even though I think I have all I need.
I added this to my kernel "CONFIG_EXT3_FS_XATTR=y" for ext3 and "CONFIG_REISERFS_FS_XATTR=y" for reiserfs.

Then I added "user_xattr" as option for the partitions I want to use it on.
Am I missing anything?
Sorry, the version you tested wasn't mean to be distributed, I didn't think that someone would try to find the webpage :oops:
But it wasn't harmfull.

From now, version distributed on the page http://vleu.net/shake/ will be stable one.
Direct link to the ebuild : http://vleu.net/shake/shake-0.19.ebuild

THE SEVERE BUG IS FIXED :D (Changelog)

Also, the output should now be readable :wink:.

If it works well, I'll try to submit the ebuild for inclusion in Gentoo.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3, 4, 5, 6  Next
Page 1 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum