| View previous topic :: View next topic |
| Author |
Message |
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
 Posted: Mon Jul 07, 2003 4:02 am Post subject: a great backup system for gentoo workstations |
 |
|
ok... i started looking for simple backup solutions for just the bare essentials (mostly config and irreplaceable files) and i came to this script.
people have extended this script by adding functionality, and i've linked the msgs with their mods at the bottom of this first post....
key points:
meant for personal workstations, not servers
allows for scalability by using user definable lists
extremely small and imho, has a very simple configuration
basically a fancy and automatic tar based backup solution
i call it backup.cron and put it in /etc/cron.weekly...  :
| Code: |
#!/bin/sh
# for FULL backups
# this backs up the important stuffs listed in ${lists} to ${BKPDIR}
# the lists *should* be in ${BKPDIR} and named <whatever>.lst
# the resulting backups will be <whatever>.<timestamp>.tgz
#
# notes:
# variables in CAPS are ok for you to set
# change the other variables if you know what you're doing
# you can get fancy in the lists... (think xargs *wink*)
# assumes /boot is usually unmounted, and has an /etc/fstab entry
# follow your security policy when setting perms on ${BKPDIR}
#
# written by razamatan
#
# DISCLAIMER: razamatan didn't write this if something goes wrong
BKPDIR=/home/.backup # where the backups go
BOOT=sys # list that has /boot
NUMBKPS=3 # how many backups to keep
if [ ! -d ${BKPDIR} ] ; then
echo ${BKPDIR} is not a valid directory or is non-existant
fi
lists=${BKPDIR}/*.lst
ext=tgz
for list in `ls ${lists}`; do
type=`basename ${list} .lst`
if [ ${type} = ${BOOT} ] ; then mount /boot ; fi
cat ${list} | xargs tar zlcf \
${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1
if [ ${type} = ${BOOT} ] ; then umount /boot ; fi
num=${NUMBKPS}
for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
if [ ${num} -le 0 ] ; then rm -f ${evict}
else num=$((${num}-1)) ; fi
done
done
|
you can download the latest version of the script from here: http://www-personal.umich.edu/~jyi/backup.cron
i actually only have one list, the /home/.backup/sys.lst:
| Code: |
/boot
/etc
/root
/usr/local/portage
--exclude=/usr/local/portage/distfiles
/usr/src/linux/.config
/var/lib
--exclude=/var/lib/texmf
|
EXTENSIONS & COMMENTS WELCOME!!!
here's some contributed extensions
iso images
iso images, tar splitting, pre & post-backup command processing, command line option to backup specific lists
specification of lists in a file (so you can have lists at various places
**EDIT** fixed numbkps issue
**EDIT** links to messages of extended versions
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Last edited by razamatan on Fri Feb 06, 2004 12:42 am; edited 3 times in total |
|
| Back to top |
|
 |
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Mon Jul 07, 2003 4:22 am Post subject: |
|
|
oh... and if you should ever need to recover from said backups, make sure you use the --same-owner and --preserve flags w/ the tar command...
heh...
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
gatiba
Guru
Joined: 01 Sep 2002
Posts: 434
|
| Posted: Mon Jul 07, 2003 10:23 am Post subject: |
|
|
Great script !
Would be nice if it could split the backup file in more parts if it's > 700mb for example (for burning ) |
|
| Back to top |
|
|
vulcan_
n00b
Joined: 06 May 2003
Posts: 61
Location: Gent, Belgium
|
| Posted: Mon Jul 07, 2003 11:12 am Post subject: splitting a tar file |
|
|
note that GNU tar has this option
| Quote: |
-L --tape-length N
change tapes after writing N*1024 bytes
|
but of course, this works to any file system that tar can output to
_________________
vulcan was a Roman myth - god of smiths |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Mon Jul 07, 2003 12:57 pm Post subject: Re: splitting a tar file |
|
|
yeah... you can do that on a per list basis. so say your <username>.lst would be huge cus of all those multimedia files, then you just need something like this to split stuff.
| Code: |
/home/raza
--exclude=/home/raza/*cache
--exclude=/home/raza/*Cache
-L 7000 # ~700 MB chunks
|
| vulcan_ wrote: | note that GNU tar has this option
| Quote: |
-L --tape-length N
change tapes after writing N*1024 bytes
|
but of course, this works to any file system that tar can output to |
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
gatiba
Guru
Joined: 01 Sep 2002
Posts: 434
|
| Posted: Tue Jul 08, 2003 7:35 am Post subject: |
|
|
Mmmhhh i have problems.
This is my sys.lst:
| Code: | /boot
/etc
/home/httpd
/windows/D/____progetti
/windows/D/____progetti/progetti_Gabriele/Alberone/vari
--exclude=/windows/D/____progetti/progetti_Gabriele/Alberone/vari/foto
/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce
--exclude=/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce/foto
-L 6500 |
When i start the script i got this error:
| Code: | | ls: /windows/D/backups/sys.*.tgz: No such file or directory |
If i delete the "-L 6500" line it works ... |
|
| Back to top |
|
|
sessionID
Apprentice
Joined: 11 Nov 2002
Posts: 266
Location: hungary
|
| Posted: Tue Jul 08, 2003 2:38 pm Post subject: |
|
|
There should be an "umount /boot" line somewhere...
_________________
((( WeFunk ))) |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Wed Jul 09, 2003 12:27 am Post subject: |
|
|
there is... look right below the crazy tar line.. 
| sessionID wrote: | | There should be an "umount /boot" line somewhere... |
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Wed Jul 09, 2003 12:41 am Post subject: |
|
|
hmmm...
this script doesn't handle archive spanning.. :/ ... spanning requires interaction. i really don't do archive spanning so i didn't test for that functionality. i though the -L would do it, but you'd have to modify the script a bit.
to span archives, you can't compress it. this would require the script to handle compression on a list by list basis, and that would add complexity. i'll see what i can whip up .
| gatiba wrote: | Mmmhhh i have problems.
This is my sys.lst:
| Code: | /boot
/etc
/home/httpd
/windows/D/____progetti
/windows/D/____progetti/progetti_Gabriele/Alberone/vari
--exclude=/windows/D/____progetti/progetti_Gabriele/Alberone/vari/foto
/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce
--exclude=/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce/foto
-L 6500 |
When i start the script i got this error:
| Code: | | ls: /windows/D/backups/sys.*.tgz: No such file or directory |
If i delete the "-L 6500" line it works ... |
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Wed Jul 09, 2003 12:58 am Post subject: |
|
|
gah... tar's -L option is weird...
this'll be a bit more research than i planned... in the meantime... treat this backup system for compressed archives... and DOESN'T SUPPORT SPANNING!!
sigh..
sorry spanning folks... i'll program my way outta this.
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
gatiba
Guru
Joined: 01 Sep 2002
Posts: 434
|
| Posted: Wed Jul 09, 2003 6:47 am Post subject: |
|
|
|
|
| Back to top |
|
|
sessionID
Apprentice
Joined: 11 Nov 2002
Posts: 266
Location: hungary
|
| Posted: Wed Jul 09, 2003 2:54 pm Post subject: |
|
|
| razamatan wrote: | there is... look right below the crazy tar line..
| sessionID wrote: | | There should be an "umount /boot" line somewhere... |
|
oops  sorry
_________________
((( WeFunk ))) |
|
| Back to top |
|
|
S_aIN_t
Guru
Joined: 11 May 2002
Posts: 488
Location: Ottawa
|
| Posted: Thu Jul 10, 2003 9:38 am Post subject: |
|
|
well done. i have been considering something similar.. althought i need a script that will backup linux and (gasp) windows.. but it would run from a linux server. i think there is something that samba has that would allow me to tar over an smb share.
more research is required for that. i'll be sure to post it here once i have something working.
_________________
"That which is overdesigned, too highly
specific, anticipates outcome; the anicipation of
outcome guatantees, if not failure, the
absence of grace."
- William Gibson, "All Tomorrow's Parties"
----
http://petro.tanreisoftware.com |
|
| Back to top |
|
|
danhan79
n00b
Joined: 17 Jul 2003
Posts: 7
|
| Posted: Thu Jul 17, 2003 4:22 am Post subject: |
|
|
| odd, i get a bus error when i run your script. queer no? |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Thu Jul 17, 2003 7:23 am Post subject: |
|
|
lol... so you've managed to switch from rh to gen2? hehehe...
the moderators on these forums are pretty tuff dan... just im me later.
| danhan79 wrote: | | odd, i get a bus error when i run your script. queer no? |
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
Elm0
Apprentice
Joined: 24 Nov 2002
Posts: 281
Location: UK
|
| Posted: Sat Jul 19, 2003 12:25 pm Post subject: |
|
|
| Great little script, just what I've been looking for Thanks! |
|
| Back to top |
|
|
Elm0
Apprentice
Joined: 24 Nov 2002
Posts: 281
Location: UK
|
| Posted: Sat Jul 19, 2003 12:47 pm Post subject: |
|
|
I don't know if the script is designed to this but...
The expected behavior would be to keep NUMBKPS of EACH list's backup in the backup directory. However, the current behavior means that if you have 5 lists and NUMBKPS = 3, the script will only backup 3 of the 5 and delete two existing files.
I would fix it myself but I'm not really a bash monkey, so maybe if you could take a look at this... |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Sat Jul 19, 2003 5:10 pm Post subject: |
|
|
ahh!!
stupid global/local variable issue.. thanks for spelunking! the code is fixed and reposted.
| Elm0 wrote: | I don't know if the script is designed to this but...
The expected behavior would be to keep NUMBKPS of EACH list's backup in the backup directory. However, the current behavior means that if you have 5 lists and NUMBKPS = 3, the script will only backup 3 of the 5 and delete two existing files.
I would fix it myself but I'm not really a bash monkey, so maybe if you could take a look at this... |
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
Zeitgeist
Apprentice
Joined: 13 Mar 2003
Posts: 165
Location: Ouagadougou, Burkina Faso
|
| Posted: Sun Oct 05, 2003 5:32 am Post subject: |
|
|
Awsome script.
I have it running in /etc/cron.monthly, /etc/cron.daily, /etc/cron.weekly, modified so they look for different *.suffixs.
I have my schoolwork from /home/zeitgeist/school backedup hourly, my movies and music dirs are backed up weekly, and my root and etc dirs are backed up daily.
It's great Thanks |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Sun Oct 05, 2003 4:42 pm Post subject: |
|
|
| Zeitgeist wrote: | Awsome script.
I have it running in /etc/cron.monthly, /etc/cron.daily, /etc/cron.weekly, modified so they look for different *.suffixs.
I have my schoolwork from /home/zeitgeist/school backedup hourly, my movies and music dirs are backed up weekly, and my root and etc dirs are backed up daily.
It's great Thanks |
i feel all squishy and warm... thanks man. 
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
Jeedo
Apprentice
Joined: 02 May 2003
Posts: 202
Location: Akureyri, Iceland
|
| Posted: Sun Oct 05, 2003 8:25 pm Post subject: |
|
|
Your posted script did not work for me, howver this one did: http://www-personal.umich.edu/~jyi/backup.cron
I've been looking for something like this, i even was going to write one myself but didnt know how (suck at bash)..
Anyway i intend to to what Zeitgeist did and back up my files on a regularly..
However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.
Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..
Another idea is to add this to the comments:
| Code: |
# DISCLAIMER: razamatan didn't write this if something goes wrong
#
# The latest version can always be found at : http://www-personal.umich.edu/~jyi/backup.cron
|
Anyway thanks alot man. |
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Mon Oct 06, 2003 2:45 am Post subject: |
|
|
| Jeedo wrote: |
However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.
Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..
|
you can remove the "x" option from tar to remove default compression... you can then put -x or -j in your lst files to do a case by case compression.... but then, you'd have to deal with naming the files with the proper extension for every respective file... so that's why i went with default gzipping... since tar and gzip is usually around, i don't mind compressing even my most important files.
| Jeedo wrote: |
Another idea is to add this to the comments:
| Code: |
# DISCLAIMER: razamatan didn't write this if something goes wrong
#
# The latest version can always be found at : http://www-personal.umich.edu/~jyi/backup.cron
|
Anyway thanks alot man. |
no problem... i should put the url for this forum thread instead. more support.. heh.
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
nat
Apprentice
Joined: 04 Sep 2002
Posts: 205
|
| Posted: Tue Oct 07, 2003 1:41 pm Post subject: |
|
|
Here is a small diff that mounts /boot if it finds '/boot' in the .lst file. If /boot already is mounted it doesnt touch it. In other words, you can put your /boot in any .lst file.
| Code: | --- backup.cron.orig 2003-10-07 15:25:16.282737029 +0200
+++ backup.cron 2003-10-07 15:41:29.382176277 +0200
@@ -16,7 +16,6 @@
# DISCLAIMER: razamatan didn't write this if something goes wrong
BKPDIR=/home/.backup # where the backups go
-BOOT=sys # list that has /boot
NUMBKPS=3 # how many backups to keep
if [ ! -d ${BKPDIR} ] ; then
@@ -29,10 +28,20 @@
for list in `ls ${lists}`; do
type=`basename ${list} .lst`
- if [ ${type} = ${BOOT} ] ; then mount /boot ; fi
+# if the .lst file contains '/boot' lets mount /boot before backing up
+# unless its already mounted.
+
+ if grep '/boot' ${list} > /dev/null && \
+ ! mount | grep /boot > /dev/null ; then
+ BOOT=1
+ else
+ BOOT=""
+ fi
+
+ [ ${BOOT} ] && mount /boot
cat ${list} | xargs tar zlcf \
${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1
- if [ ${type} = ${BOOT} ] ; then umount /boot ; fi
+ [ ${BOOT} ] && umount /boot
num=${NUMBKPS}
for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
|
|
|
| Back to top |
|
|
razamatan
Apprentice
Joined: 28 Feb 2003
Posts: 160
|
| Posted: Tue Oct 07, 2003 7:55 pm Post subject: |
|
|
| nat wrote: | Here is a small diff that mounts /boot if it finds '/boot' in the .lst file. If /boot already is mounted it doesnt touch it. In other words, you can put your /boot in any .lst file.
|
i thougth about this... but then i realized.... there's no need to have boot in more than 1 archive. so let's just specify it. even in my incarnation, you can stick /boot in any list.. just name that list in the env. variable.
your code actually has a slight problem. say that i have a situation where i have boot as a subdir of someplace that i wanted (/this/boot).. or even that i don't wanna include boot (--exclude=/boot)... it greps for any instance of /boot... thereby, it does unnecessary (u)mounts. just a slight problem.
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity" |
|
| Back to top |
|
|
Jeedo
Apprentice
Joined: 02 May 2003
Posts: 202
Location: Akureyri, Iceland
|
| Posted: Wed Oct 08, 2003 11:04 pm Post subject: |
|
|
| razamatan wrote: | | Jeedo wrote: |
However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.
Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..
|
you can remove the "x" option from tar to remove default compression... you can then put -x or -j in your lst files to do a case by case compression.... but then, you'd have to deal with naming the files with the proper extension for every respective file... so that's why i went with default gzipping... since tar and gzip is usually around, i don't mind compressing even my most important files.
|
I know, and that is what i did, but what i meant is that it would be nice to add it as a variable, like
compression=tbz2
or tgz og just tar and the script would then add xj, xz or just nothing respectively |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
