| View previous topic :: View next topic |
| Author |
Message |
slick
Bodhisattva
Joined: 20 Apr 2003
Posts: 3495
|
 Posted: Sat Feb 18, 2017 3:07 pm Post subject: filebased offline deduplication script for btrfs |
 |
|
Here is a script that dedup files in btrfs.
I found this script on the web and made some changes.
I love it because I can understand and see what it will do. No hidden magic.
* whole files only, not block based
* the files must not be in access
* it is not possible to dedup on readonly subvolumes
You can dedup directories or the whole (sub-) volume.
I did not test it over multiple subvolumes. Try it and give feedback please.
Be sure there are no other mountpoints in the directory to examinate.
It should not work as expected and I guess it will be make trouble.
Be sure to run it only on btrfs 
As always, be sure to have backups. 
| Code: | #!/bin/sh
# some simple tests
test -e "$1" || exit 1
mkdir "$1/~~~dedup" || exit 1
cd "$1" || exit 1
# dedup
find -type f -not -path "./~~~dedup/*" -print0 | while read -d $'\0' -r F ;
do
echo -n "$F : "
FHASH=$(sha256sum "$F" | cut -d" " -f1) ;
if [[ -f "~~~dedup/$FHASH" ]] && cmp -s "~~~dedup/$FHASH" "$F" ;
then
echo "Dup." ;
cp --reflink "~~~dedup/$FHASH" "$F"
else
echo "New." ;
cp --reflink "$F" "~~~dedup/$FHASH" ;
fi
done
# cleanup
find "$1/~~~dedup/" -type f -exec rm {} \;
rmdir "$1/~~~dedup/"
|
To run, give the directory as $1, i.E.:
| Code: | | dedupscript.sh /mnt/btrfs/folder |
(if you cancel it, please cleanup the "~~~dedup" working dir. It is safe to remove it.) |
|
| Back to top |
|
 |
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
