Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
filebased offline deduplication script for btrfs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
slick
Bodhisattva
Bodhisattva


Joined: 20 Apr 2003
Posts: 3491

PostPosted: Sat Feb 18, 2017 3:07 pm    Post subject: filebased offline deduplication script for btrfs Reply with quote

Here is a script that dedup files in btrfs.

I found this script on the web and made some changes.
I love it because I can understand and see what it will do. No hidden magic.

* whole files only, not block based
* the files must not be in access
* it is not possible to dedup on readonly subvolumes

You can dedup directories or the whole (sub-) volume.
I did not test it over multiple subvolumes. Try it and give feedback please.

Be sure there are no other mountpoints in the directory to examinate.
It should not work as expected and I guess it will be make trouble.

Be sure to run it only on btrfs :!:

As always, be sure to have backups. :wink:

Code:
#!/bin/sh

# some simple tests
test -e "$1" || exit 1
mkdir "$1/~~~dedup" || exit 1
cd "$1" || exit 1

# dedup
find -type f -not -path "./~~~dedup/*" -print0 | while read -d $'\0' -r F ;
do
   echo -n "$F : "
   FHASH=$(sha256sum "$F" | cut -d" " -f1) ;
   if [[ -f "~~~dedup/$FHASH" ]] && cmp -s "~~~dedup/$FHASH" "$F" ;
   then
      echo "Dup." ;
      cp --reflink "~~~dedup/$FHASH" "$F"
   else
      echo "New." ;
      cp --reflink "$F" "~~~dedup/$FHASH" ;
   fi
done

# cleanup
find "$1/~~~dedup/" -type f -exec rm {} \;
rmdir "$1/~~~dedup/"



To run, give the directory as $1, i.E.:

Code:
dedupscript.sh /mnt/btrfs/folder


(if you cancel it, please cleanup the "~~~dedup" working dir. It is safe to remove it.)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum