| View previous topic :: View next topic |
| Author |
Message |
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 4045
Location: Rasi, Finland
|
 Posted: Thu Mar 09, 2017 12:29 pm Post subject: rsync and case-insensitive excludes |
 |
|
So.
I plan to ditch rdiff-backup and go with rsyncing and btrfs snapshots.
The idea is to rsync / with long exclude list to /var/backups/current.
Then snapshot that directory to /var/backups/$(date +someformat).
In the rsync exclude file I have several rules to exclude files with certain extensions using this syntax:- **.extension. However rsync does not seem to have case-insensitive pattern matching capabilities.
... but if there's *NIX, there's a way.
So what I came up is a somewhat dirty way around this limitation: | case-insensitive file extension match in rsync: | | rsync -a --switches --exclude-from=<(awk '{ if ($0 ~ /^\- \*\*\.[^/]/) {a=""; for (i = 6; i <= length($0); i++) {b = substr($0, i, 1); if (b ~ /[[:alpha:]]/) a = a "[" toupper(b) tolower(b)"]"; else a = a b} print "- **." a} else print}' /path/to/exclude.list) / /var/backups/current |
Note: <(command) isn't sh compatible, and I'm not sure if that awk code is POSIX awk compatible. I used gawk.
What awk is doing there is matching all lines starting with "- **." and then going trough every character one-by-one and if the character is a letter then replace it with upper AND lower -case match pattern. For example - **.avi turns into - **.[Aa][Vv][Ii].
So. Does anyone know a cleaner way, or should I keep using this "horror code"?
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
 |
cboldt
Veteran
Joined: 24 Aug 2005
Posts: 1046
|
| Posted: Thu Mar 09, 2017 1:13 pm Post subject: |
|
|
rysnc has no case-insensitive pattern matching. ... but
I'd use that "dirty code" one time, or convert a pattern list one time (technically, each time the "exclude list" was modified.) I have something close to that in my .dircolors, but with extensions, each (or some) have two lines, one all lowercase, and one all uppercase. I doubt you will run into .avI, .AvI, .aVi, .aVI, and so on. It'll be either .avi or .AVI.
I recently found rsync's --filter parameter, which I now greatly prefer over --exclude-from/--include-from. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 4045
Location: Rasi, Finland
|
| Posted: Thu Mar 09, 2017 1:45 pm Post subject: |
|
|
| cboldt wrote: | | I doubt you will run into .avI, .AvI, .aVi, .aVI, and so on. |
It's rare, yes. But I've encountered *.Ext files.
Anyway I read more and found out that you can build rsync with '--ignore-case', but that's not stable yet... Something to do with windows filesystems being case insensitive, but still writing case sensitive filenames.
Oh well... I guess I go with that awk. I'll modify it when needed.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Mar 09, 2017 6:45 pm Post subject: |
|
|
Zucca ...rather than look to rsync for the solution, you might look to the shell (in this case zsh):
| Code: | % mkdir -p test/test1
% cd test
% touch {,test1/}foo.{JPG,AVI,WMV}
% touch {,test1/}ba.{jPG,aVI,WMv}
% zmv '(**/)(*).((#i)(jp(|e)g|avi|wmv))' '$1$2.${(L)3}'
mv -- test1/ba.WMv test1/ba.wmv
mv -- test1/ba.aVI test1/ba.avi
mv -- test1/ba.jPG test1/ba.jpg
mv -- test1/foo.AVI test1/foo.avi
mv -- test1/foo.JPG test1/foo.jpg
mv -- test1/foo.WMV test1/foo.wmv
mv -- ba.WMv ba.wmv
mv -- ba.aVI ba.avi
mv -- ba.jPG ba.jpg
mv -- foo.AVI foo.avi
mv -- foo.JPG foo.jpg
mv -- foo.WMV foo.wmv |
While that is only useful if you happen to be using zsh, you could probably work out a similar solution using 'rename' and 'find' (but I'll leave that as an exercise ... and a push for zsh).
best ... khay |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 4045
Location: Rasi, Finland
|
| Posted: Fri Mar 10, 2017 2:24 pm Post subject: |
|
|
| khayyam wrote: |
While that is only useful if you happen to be using zsh, you could probably work out a similar solution using 'rename' and 'find' (but I'll leave that as an exercise ... and a push for zsh).
best ... khay |
Thanks.  I used zsh before, but went back to bash and dash. However I could possibly get that working with extglob enabled in bash.
However, so far I've gotten rsync to work. If I later found the features of rsync too limited then I'll propably go with some shell scripting and some diff capable copy tool... |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Fri Mar 10, 2017 4:56 pm Post subject: |
|
|
| Zucca wrote: | | khayyam wrote: | | While that is only useful if you happen to be using zsh, you could probably work out a similar solution using 'rename' and 'find' (but I'll leave that as an exercise ... and a push for zsh). |
Thanks. :) I used zsh before, but went back to bash and dash. However I could possibly get that working with extglob enabled in bash. |
Zucca ... you're welcome. I don't think bash could provide the above without using external tools (ie, rename ... which is written in perl), extglob doesn't have qualifiers, the last I looked it only had character classes and some switches for negation, etc.
| Zucca wrote: | | However, so far I've gotten rsync to work. If I later found the features of rsync too limited then I'll propably go with some shell scripting and some diff capable copy tool... |
Of course, however you want to handle the problem. In my case I do the above to avoid issues, or fix badly named files, ie, windows tuncating files to *.JPE, or files with spaces:
| Code: | | % zmv -Q '(**/)* *(.D)' '$f:gs/ /_' |
... or padding zeros (and so have the files listed in a reasonable order):
| Code: | | % i=1; for f (*.jpg) zmv $f '${(l:3::0:)$((++i))}'$f |
Doing so just means that down the line I don't have to worry about rsync, or what-have-you, needing to be hand held so that --exclude matches the desired files.
best ... khay |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
