| View previous topic :: View next topic |
| Author |
Message |
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 918
|
 Posted: Wed Nov 09, 2016 10:05 pm Post subject: Problem with flexbackup |
 |
|
In case anyone here is using it.
A few days ago, I suddenly found the daily incremental backups ballooning. Say, for a given directory, from 1MB to 50+MB the next day, nearly double the actual size of the directory. I eventually figured out that the find command that returns files newer than the marker under /var/lib/flexbackup includes "." in its output, so flexbackup backs up the complete directory plus the files that changed, leading to files that are really no longer incrementals.
The interesting part is that find itself was updated back in June, so it is not responsible for this new behaviour. Flexbackup was last updated more than two years ago. The only related tool that was updated on the day was GNU tar, from 1.28-r1 to 1.29-r1.
I wasn't able to figure out if tar really is the culprit, but decided to add a workaround to flexbackup directly (the line numbers will be off since I have some more code added).
| Code: |
@@ -1408,6 +1416,7 @@
$cmd .= "$::path{tar} --create ";
$cmd .= "--null ";
+ $cmd .= "--exclude=\"\.\" ";
$cmd .= "--files-from=- ";d probably
$cmd .= "--ignore-failed-read ";
$cmd .= "--same-permissions ";
@@ -2718,7 +2727,7 @@
|
A similar change is probably needed in the star section, but I don't have it installed (-V pattern="." or something along those lines). |
|
| Back to top |
|
 |
John R. Graham
Administrator
Joined: 08 Mar 2005
Posts: 10618
Location: Somewhere over Atlanta, Georgia
|
| Posted: Wed Nov 09, 2016 10:19 pm Post subject: |
|
|
I'm not seeing this behavior, but I use $type = 'afio';. Could you post your flexbackup.conf file, please?
Interestingly enough, tar was updated recently.
- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters. |
|
| Back to top |
|
|
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 918
|
| Posted: Thu Nov 10, 2016 11:57 pm Post subject: |
|
|
Well, scratch that. The change I made does not work, nor any other changes in flexbackup.
It is definitely the new version of GNU tar breaking it. I compiled vanilla 1.29 from source and used that through $path{'tar'}. The result is a 99MB .tar.xz file for a 38MB directory, and looking at the archive, all files have been backed up 5-6 times each. E.g.
| Code: |
# xz -dc usr-local.0.201611102343.tar.xz |tar tvf - |sort -k3n
[...]
-r--r--r-- root/root 1059609 2016-06-16 21:57 ./pg_overlay/.git/objects/90/3a67f65697f8727f0a296ec83edc709f033700
-r--r--r-- root/root 5074125 2016-11-08 22:02 ./pg_overlay/.git/objects/pack/pack-023e296ffb7e16e75224f368e06e76507c8dc537.pack
-r--r--r-- root/root 5074125 2016-11-08 22:02 ./pg_overlay/.git/objects/pack/pack-023e296ffb7e16e75224f368e06e76507c8dc537.pack
-r--r--r-- root/root 5074125 2016-11-08 22:02 ./pg_overlay/.git/objects/pack/pack-023e296ffb7e16e75224f368e06e76507c8dc537.pack
-r--r--r-- root/root 5074125 2016-11-08 22:02 ./pg_overlay/.git/objects/pack/pack-023e296ffb7e16e75224f368e06e76507c8dc537.pack
-r--r--r-- root/root 5074125 2016-11-08 22:02 ./pg_overlay/.git/objects/pack/pack-023e296ffb7e16e75224f368e06e76507c8dc537.pack
-r--r--r-- root/root 5074125 2016-11-08 22:02 ./pg_overlay/.git/objects/pack/pack-023e296ffb7e16e75224f368e06e76507c8dc537.pack
|
I'm masking 1.29 for the time being. |
|
| Back to top |
|
|
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 918
|
| Posted: Fri Nov 11, 2016 12:29 am Post subject: |
|
|
Got lucky in the Bug-tar list archive.
Several options of tar are (only now?) position-sensitive. Both "--null" and "--no-recursion" must come before "--files-from=-". I.e. this code
| Code: |
$cmd .= "--null ";
$cmd .= "--files-from=- ";
$cmd .= "--ignore-failed-read ";
$cmd .= "--same-permissions ";
$cmd .= "--no-recursion ";
$cmd .= "--totals ";
|
needs to change to
| Code: |
$cmd .= "--null ";
$cmd .= "--no-recursion ";
$cmd .= "--files-from=- ";
$cmd .= "--ignore-failed-read ";
$cmd .= "--same-permissions ";
$cmd .= "--totals ";
|
Wrong option ordering leads to progressive recursion into subdirectories where the number of times a file gets backed up is proportional to the number of directories of its own path. The deeper you go, the more copies you get.
Ref.: https://lists.gnu.org/archive/html/bug-tar/2016-05/msg00008.html |
|
| Back to top |
|
|
jahun
n00b
Joined: 11 Nov 2016
Posts: 3
|
| Posted: Fri Nov 11, 2016 11:52 am Post subject: |
|
|
| i can confirm bug in flexbackup + new tar. pls post bug for flexbackup to https://bugs.gentoo.org with your patch. tnx |
|
| Back to top |
|
|
jahun
n00b
Joined: 11 Nov 2016
Posts: 3
|
| Posted: Fri Nov 11, 2016 1:32 pm Post subject: |
|
|
patch is working ok:
| Code: | --- old/flexbackup 2016-12-01 10:52:59.088767278 -0500
+++ new/flexbackup 2016-12-01 10:54:18.006882216 -0500
@@ -1408,10 +1408,10 @@
$cmd .= "$::path{tar} --create ";
$cmd .= "--null ";
+ $cmd .= "--no-recursion ";
$cmd .= "--files-from=- ";
$cmd .= "--ignore-failed-read ";
$cmd .= "--same-permissions ";
- $cmd .= "--no-recursion ";
$cmd .= "--totals ";
if ($cfg::label ne 'false') {
if (length($title) > $::tar_max_label) {
|
[Moderator edit: added [code] tags to preserve output layout. -Hu]
Further moderator edit: cleaned up the patch. — JRG |
|
| Back to top |
|
|
John R. Graham
Administrator
Joined: 08 Mar 2005
Posts: 10618
Location: Somewhere over Atlanta, Georgia
|
| Posted: Thu Dec 01, 2016 5:02 pm Post subject: |
|
|
In the spirit of the aphorism, "No job is complete until the paperwork is done," I filed the bug report. (I hope there's no toilet paper stuck to my shoe.)
- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters. |
|
| Back to top |
|
|
|
Portage & Programming
