Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] (BA)SH - Problems with "finding duplicates" script
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
aka.doode
n00b
n00b


Joined: 16 May 2005
Posts: 45
Location: Sweden

PostPosted: Thu Apr 15, 2010 11:25 pm    Post subject: [SOLVED] (BA)SH - Problems with "finding duplicates&quo Reply with quote

Hello fellas!

All is not well in shell script land, and sadly I'm not proficient enough to solve it myself. Perhaps someone else can easily spot the error and provide a solution?
I found this script on some site a year ago or so and used it heavily at first to weed out some duplicate files in my backups:

Code:
#!/bin/sh

#get a unique filename in /tmp
CHECKSUMS="/tmp/checksums_$RANDOM"
while [ -f "$CHECKSUMS" ]; do
  CHECKSUMS="/tmp/checksums_$RANDOM"
done

#put the md5sums of all files in the passed directories into the file
find "$@" -type f -exec md5sum "{}" + >> "$CHECKSUMS"

#loop through each md5sum in the file
NUM_MATCHES=0
MATCHES=""
for i in $(grep -o '^[^ ]*' "$CHECKSUMS"); do
  #grab all entries that have the same md5sum as the current one
  ENTRIES="$(grep "$i" "$CHECKSUMS")"
  #parse out the filenames
  FILES="$(echo "$ENTRIES" | sed 's/^[0-9a-f]\{32\}\s*//')"
  #if there are more than one file with the same md5sum as the current one, and
  #the current one hasn't been used before, then list the files as being identical
  if [ $( echo "$FILES" | grep -c '^') -gt 1 ] && [ "$(echo ${MATCHES[*]} | grep $i)" = "" ]; then
    #and add the md5sum to the list of used ones in $MATCHES
    MATCHES[$NUM_MATCHES]=$i
    NUM_MATCHES=$[$NUM_MATCHES+1]
    echo "These files are identical:"
    echo "$FILES"
    echo
  fi
done

#clean up
rm -f "$CHECKSUMS"



At that time it worked great. Now, when needing to use it again, it gives some weird results.
It appears to get all the md5sums without any problems, but when it starts to compare them and write the results to file it outputs a bunch of "File name too long" to the console. Not sure if it means just the file name, or file name + path. If it's the latter it could be a reason to why it worked better before, although I doubt it.

Last couple of errors:
Code:
...$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[0+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]]=e869180c0b9cde1d7e29e91f23b40b0e: File name too long
duplicates.sh: 30: Bad substitution
duplicates.sh: 30: MATCHES[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[
$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$
[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[$[0+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+
1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]
+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1]+1
]+1]+1]+1]+1]]=e869180c0b9cde1d7e29e91f23b40b0e: File name too long


(Please disregard non symmetrical formatting errors ;))

The non error output (redirected to a file with '>') has a truck load of duplicates among the duplicates (like, several "file recipes_cookies and file cookies_recipes are identical"). From what I can tell it's mostly (I've seen some oddities, but not many) accurate.

Anyong willing to lend a hand?


Last edited by aka.doode on Fri Apr 16, 2010 12:27 am; edited 1 time in total
Back to top
View user's profile Send private message
Sadako
Advocate
Advocate


Joined: 05 Aug 2004
Posts: 3792
Location: sleeping in the bathtub

PostPosted: Thu Apr 15, 2010 11:39 pm    Post subject: Reply with quote

Check to make sure /bin/sh is a symlink to /bin/bash, if not change the head of that file to '#!/bin/bash'.
That script may work in bash, but might not under other shells, under dash the 'NUM_MATCHES=$[$NUM_MATCHES+1]' bit seems to output what you're seeing.

Anyways, you could just use 'fdupes' for this, a wonderful little utility available in portage.
_________________
"You have to invite me in"
Back to top
View user's profile Send private message
Mike Hunt
Watchman
Watchman


Joined: 19 Jul 2009
Posts: 5287

PostPosted: Thu Apr 15, 2010 11:57 pm    Post subject: Reply with quote

The script works fine here with bash-4.0.35(2)-release.

I had no idea I had so many dupes in $HOME. :)
Back to top
View user's profile Send private message
aka.doode
n00b
n00b


Joined: 16 May 2005
Posts: 45
Location: Sweden

PostPosted: Fri Apr 16, 2010 12:27 am    Post subject: Reply with quote

Hopeless wrote:
Check to make sure /bin/sh is a symlink to /bin/bash, if not change the head of that file to '#!/bin/bash'.
That script may work in bash, but might not under other shells, under dash the 'NUM_MATCHES=$[$NUM_MATCHES+1]' bit seems to output what you're seeing.

Anyways, you could just use 'fdupes' for this, a wonderful little utility available in portage.


Thanks a bunch for the fast reply!
I'm in a different environment than Gentoo atm:
Code:
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 2010-01-25 03:32 /bin/sh -> dash

Changed the script to #!/bin/bash and now it works correctly. Thanks mate :)

I'll have to give that fdupes a spin as well. Sounds promising (antyhing to try to improve the speed of finding dupes among tens of gigs of data)


Mike Hunt wrote:
I had no idea I had so many dupes in $HOME. :)


It's amazing what something like that can do for your home cleaning motivation ;)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum