View previous topic :: View next topic |
Author |
Message |
BlinkEye Veteran
Joined: 21 Oct 2003 Posts: 1046 Location: Gentoo Forums
|
Posted: Tue Jan 17, 2006 7:30 pm Post subject: |
|
|
mattjgalloway wrote: | That seems to be working now, but I thought of another thing...
Perhaps there could be a progress bar incorporated into the script... It could just be along the lines of files left to do / total files to get the percentage done. Obviously a better way would be the size of files, but it'd be a good start at least.
Is that do-able do you think? |
yes, i think so. if you use the script with the verbose option ( -v/ --verbose ) you see the command it does. removing the tar command and combining it with
gives you the amount of files the command will process to tar. for one box of mine this would be about 166000 files (counted within a few seconds) - so, still no loss. i think the number of files suffice for the progress bar. now, using the vebose option will give you the files processed by tar - and from there one could create this progress bar. i need a command that can say, print a '#' for every 100 lines being output on stdout. any ideas? _________________ Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick |
|
Back to top |
|
|
BlinkEye Veteran
Joined: 21 Oct 2003 Posts: 1046 Location: Gentoo Forums
|
Posted: Tue Jan 17, 2006 7:32 pm Post subject: |
|
|
carpman wrote: | Seems to have started working but still have one issue, /var/log/portage i created this and given it lots chmod options, it is currently 777 but still emerges complain it is not writable? i set it to owner/group 'portage' as is on other systems but no joy?
PS great script otherwise
Cheers |
could it be you you created a file instead of a directory?
Code: | drwxrws--- 2 portage portage 114136 Jan 16 00:28 portage |
_________________ Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick |
|
Back to top |
|
|
gnychis Veteran
Joined: 23 Mar 2005 Posts: 1004 Location: Pittsburgh, PA
|
Posted: Mon Jan 23, 2006 3:02 am Post subject: |
|
|
for some reason my custom exclude folders do not seem to be working
I try to exclude /160gb and /200gb, however whenever I run the script in verbose mode it is recursively backing up all of /160gb and /200gb
when i start the script i use option 4
here is my file:
Code: |
#!/bin/bash
# Backup script for Gentoo Linux
# Copyright Reto Glauser aka Blinkeye
# Distributed under the terms of the GNU General Public License v2
# Mailto: stage4 at blinkeye dot ch
# Forum post: http://forums.gentoo.org/viewtopic-t-312817.html
# Date: 2005-06-30
version=v3.5
basename=`basename $0`
find=/usr/bin/find
tar=/bin/tar
# these are the commands we actually need for the backup
command_list=(cut date echo $find grep hostname mount sh split $tar umount uname which)
# verify that each command we use exists. if one can't be found use $PATH and make a suggestion if possible.
for command in ${command_list[@]}; do
if [ ! -x "`which $command 2>&1`" ]; then
echo -e "\nERROR: $command not found! "
base=`basename $command`
if [ "`which $base 2>&1 | grep "no \`basename $command\` in"`" != "" ]; then
echo -e "ERROR: $base is not in your \$PATH."
fi
exit -1
fi
done
help="\nUsage:\n\nsh `basename $0` [[-v]|[--verbose]] [[-s]|[--split]] \n\nTo run the script NOT in verbose mode comes in handy if you want to see only the errors that occur during the backup.\n"
# Defaults to creating one tarball
tar_output="--file"
# split command
split_options="--suffix-length=1 --bytes=685m"
# options for the tar command
tarOptions=" --preserve-permissions --create --absolute-names --totals --ignore-failed-read"
# where to put the stage4
stage4Location=/mnt/backups/stage4
# name prefix
stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d`
# patterns which should not be backed up (like iso files).
# example: default_exclude_pattern="*.iso *.divx"
# These pattern count only for files NOT listed in the $custom_include_list.
default_exclude_pattern=""
# these files/directories are always excluded. don't add trailing slashes.
# don't touch it unless you know what you are doing!
# /var/db and /var/cache/edb are intentionally added here. they are listed
# in $default_include_folders
default_exclude_list="
/dev
/lost+found
/mnt
/proc
/sys
/tmp
/usr/portage
/usr/src
/var/log
/var/tmp
/var/db
/var/cache/edb
$stage4Location
`echo $CCACHE_DIR`"
# files/devices/folders, which need to be backed up (preserve folder structure).
# don't touch it unless you know what you are doing! no recursive backup of folders.
# use $default_include_folders instead.
default_include_files="
/dev/null
/dev/console
/home
/mnt
/proc
/sys
/tmp
/usr/portage
/usr/src
/var/log/emerge.log"
# folders, which need to be backed up recursively on every backup.
# don't touch it unless you know what you are doing! the reason for this
# variable is that some users add /var to the $default_exclude_list. here
# we ensure that portage's memory is backed up in any case.
default_include_folders="
/var/db"
# IMPORTANT: A minimal backup will EXCLUDE files/folders listed here. A custom backup will
# include/exclude these files/folders depening on your answer.
custom_include_list="
/home/*
/usr/src/linux-`uname -r`"
# add files/folders here which are subfolders of a folder listed in $custom_include_list which should NOT
# be backed up. eg.
#custom_exclude_list="/home/foo/mp3 /home/foo/downloads /home/foo/.*"
custom_exclude_list="/home/hedpe/server /home/hedpe/ut2004 /home/hedpe/downloads /home/hedpe/.rstw16 /200gb /160gb"
# Only files/folders within the $custom_include_list are checked against these patterns
# custom_exclude_pattern="*.mp3 *.iso"
custom_exclude_pattern="*.img *.VOB *.BUP *.mp3 *.iso *.gcm"
# the find_command
find_command="$find /*"
# don't backup anything which matches pattern listed in $default_exclude_pattern
for pattern in $default_exclude_pattern; do
find_command="$find_command -not -name $pattern"
done
# assemble the find_command
function find_files()
{
for folder in $default_exclude_list; do
find_command="$find_command -path $folder -prune -o"
done
find_command="$find_command -print"
for i in $default_include_files; do
find_command="echo $i; $find_command"
done
for i in $default_include_folders; do
if [ -d $i ]; then
find_command="$find $i; $find_command"
else
find_command="echo $i; $find_command"
fi
done
}
# check the exclude/include variables for non-existing entries
function verify()
{
for i in $1; do
if [ ! -e "`echo "$i" | cut -d'=' -f2 | cut -d'*' -f1`" -a "$i" != "/lost+found" -a "$i" != "$stage4Location" ]; then
echo "ERROR: `echo "$i" | cut -d'=' -f2` not found! Check your "$2
exit 0
fi
done
}
# check input parameters
while [ $1 ]; do
case $1 in
"-h" | "--help")
echo -e $help
exit 0;;
"-v" | "--verbose")
verbose=$1;;
"-s" | "--split")
tar_output="--split";;
"");;
*)
echo -e $help
exit 0;;
esac
shift
done
echo ""
# check folder/files listed in $default_exclude_list exist
verify "$default_exclude_list" "\$default_exclude_list"
# check files listed in $default_include_files exist
verify "$default_include_files" "\$default_include_files"
# check folder listed in $default_include_folders exist
verify "$default_include_folders" "\$default_include_folders"
#check folder listed in $custom_include_list exist
verify "$custom_include_list" "\$custom_include_list"
#check folder listed in $custom_exclude_list exist
verify "$custom_exclude_list" "\$custom_exclude_list"
# print out the version
echo -e "\nBackup script $version"
echo -e "=================="
# how do you want to backup?
echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n
Fast (tar.gz):
(1) Minimal backup
(2) Interactive backup
Best (tar.bz2):
(3) Minimal backup
(4) Interactive backup\n"
while [ "$option" != '1' -a "$option" != '2' -a "$option" != '3' -a "$option" != '4' ]; do
echo -en "Please enter your option: "
read option
done
case $option in
[1,3])
stage4Name=$stage4Location/$stage4prefix-minimal.tar;;
[2,4])
stage4Name=$stage4Location/$stage4prefix-custom.tar
for folder in $custom_include_list; do
echo -en "\nDo you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) "
read answer
while [ "$answer" != 'y' -a "$answer" != 'n' ]; do
echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) "
read answer
done
if [ "$answer" == 'n' ]; then
find_command="$find_command -path $folder -prune -o"
else
custom_find="$find $folder"
for i in $custom_exclude_pattern; do
custom_find="$custom_find -name $i -o"
done
for i in $custom_exclude_list; do
custom_find="$custom_find -path $i -prune -o"
done
find_command="$custom_find -print; $find_command"
fi
done ;;
esac
# add $custom_include_list to the $default_exclude_list as we assembled
# $custom_find with $custom_include_list already.
default_exclude_list="$default_exclude_list $custom_include_list"
case $option in
[1,2])
stage4postfix="gz"
zip="--gzip";;
[3,4])
stage4postfix="bz2"
zip="--bzip2";;
esac
# mount boot
echo -e "\n* mounting boot"
mount /boot >/dev/null 2>&1
# find the files/folder to backup
find_files
find_command="($find_command)"
# create the final command
if [ "$tar_output" == "--file" ]; then
tar_command="$find_command | $tar $zip $tarOptions $verbose --file $stage4Name.$stage4postfix --no-recursion -T -"
else
tar_command="$find_command | $tar $zip $tarOptions $verbose --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"
fi
if [ "$verbose" ]; then
echo -ne "\n* creating the stage4 in $stage4Location with the following command:\n\n"$tar_command
fi
# everything is set, are you sure to continue?
echo -ne "\nDo you want to continue? (y/n) "
read answer
while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do
echo -ne "Do you want to continue? (y/n) "
read answer
done
if [ "$answer" == 'y' ]; then
# check whether the file already exists.
if [ "$tar_output" == "--split" ]; then
overwrite="`ls "$stage4Name.$stage4postfix"_* 2>&1 | grep -v 'No such file'`"
else
overwrite="$stage4Name.$stage4postfix"
fi
if [ -a "`echo "$overwrite" | grep "$overwrite" -m1`" ]; then
echo -en "\nDo you want to overwrite $overwrite? (y/n) "
read answer
while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do
echo -en "Do you want to overwrite $overwrite? (y/n) "
read answer
done
if [ "$answer" == 'n' ]; then
echo -e "\n* There's nothing to do ... Exiting"
exit 0;
fi
fi
# if necessary, create the stage4Location
if [ ! -d "$stage4Location" ] ; then
echo "* creating directory $stage4Location"
mkdir -p $stage4Location
fi
echo -e "\n* Please wait while the stage4 is being created.\n"
# do the backup.
sh -c "$tar_command"
# finished, clean up
echo -e "\n* stage4 is done"
echo "* umounting boot"
umount /boot >/dev/null 2>&1
# Integrity check
echo -e "* Checking integrity"
if [ "$zip" == "--gzip" ]; then
zip="gzip"
else
zip="bzip2"
fi
if [ "$tar_output" == "--split" ]; then
if [ "`cat "$stage4Name.$stage4postfix"_*"" | $zip --test 2>&1`" != "" ]; then
echo -e "* Integrity check failed. Re-run the script and check your hardware."
exit -1
fi
else
if [ "`$zip --test $stage4Name.$stage4postfix 2>&1`" != "" ]; then
echo -e "* Integrity check failed. Re-run the script and check your hardware."
exit -1
fi
fi
# everything went smoothly"
echo -e "* Everything went smoothly. You successfully created a stage4."
else
echo -e "\n* There's nothing to do ... Exiting"
fi
# Split the archive into chunks - uncomment the 3 lines if you want to split the stage4
# echo -e "* split $stage4Name.$stage4postfix"
# split $split_options $stage4Name.$stage4postfix "$stage4Name.$stage4postfix"_
# echo "* splitting is done"
|
|
|
Back to top |
|
|
gnychis Veteran
Joined: 23 Mar 2005 Posts: 1004 Location: Pittsburgh, PA
|
Posted: Tue Jan 24, 2006 6:11 am Post subject: |
|
|
problem solved by adding the folders to default_exclude_list
|
|
Back to top |
|
|
mb Guru
Joined: 25 Apr 2002 Posts: 355 Location: Hessen | .de
|
Posted: Mon Jan 30, 2006 4:37 pm Post subject: |
|
|
Hi *,
I've got a small suggestion for your (excellent) script. Maybe you should add the dhcpcd cache files to the default exclude list (/var/lib/dhcdc/dhcpcd*.cache). I made a stage4 file on my test server at home, transfered it to my root server, rebooted.. and.. lost connection, because the root server took the (old) ip from the cache file...
#mb _________________ Linux ares 2.6.15-gentoo-r1 #4 SMP Mon Jan 16 17:38:31 CET 2006 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4600+ AuthenticAMD GNU/Linux |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Mon Jan 30, 2006 5:15 pm Post subject: Check out app-arch/star |
|
|
My apologies if this has already been mentioned, but also check out app-arch/star for an interesting alternative to "tar". This alternative app has support for backing up and restoring selinux style extra attributes (which I need in my case) - It may also have the overlapping exclude/include support that tar is also missing?
I mentioned kdar already, but that's another interesting alternative as well.
Nice script though |
|
Back to top |
|
|
BlinkEye Veteran
Joined: 21 Oct 2003 Posts: 1046 Location: Gentoo Forums
|
Posted: Wed Feb 01, 2006 10:27 am Post subject: |
|
|
thanks for the hint about star. as any page about star has been moved i cannot test if it is fully compatible with the tar options used in my script. _________________ Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Wed Feb 01, 2006 10:47 am Post subject: |
|
|
Top hit in google
http://cdrecord.berlios.de/old/private/man/star.html
I am playing with it right now, and at first sight it's not quite right. Firstly it clobbers your existing tar binary when it installs and the options are not similar enough that this is acceptable. The only thing I can see which might actually help in the end is:
- Multi-archive
- Advanced regexp for excludes
- Archiving of linux attributes
So kdar still looks like a better solution, but the negative there was getting a binary when you needed to restore the system in a hurry (from scratch).
Unsure... |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Wed Feb 01, 2006 10:59 am Post subject: |
|
|
Hmm, just noticed an interesting feature at:
http://www.gnu.org/software/tar/manual/html_mono/tar.html#SEC57
If you "cat" together two archives then try and list the files you will only see the files from the first archive. However, if you add the "-i" option then you can "workaround this" and extract the entire archive. I just tried creating two tar files, then cat'ing them together, then gziping them, and then I could read the whole thing with "tar -tif test.tar.gz"
It's a bit of a hack, but interesting...
Ed |
|
Back to top |
|
|
BlinkEye Veteran
Joined: 21 Oct 2003 Posts: 1046 Location: Gentoo Forums
|
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Wed Feb 01, 2006 12:02 pm Post subject: |
|
|
The link *I* posted works fine. The one *you* posted has an extra "." at the end?
I dont see that star actually solves the problem in this case, but it does offer a possible solution for selinux backups... Also the concatenating two tar archives together thing nearly works, but I can't quite see how to efficiently stream the output of two tar commands into bzip? Any thoughts?
Ed |
|
Back to top |
|
|
Matteo Azzali Retired Dev
Joined: 23 Sep 2004 Posts: 1133
|
Posted: Wed Feb 01, 2006 1:11 pm Post subject: |
|
|
Hum,Blinkeye, sorry to bother, but I would like to build a visual frontend in KDE for this wonderful script,
(Kommander script, check AviUtils or Kfirewatcher to see what I've done before) and I was
swearing if there is some better option to add dirs in default_exclude_list other than modifying the
script itself (an option at script startup? a parameter from command line? ) _________________ Every day a new distro comes to birth. Every day a distro "eats" another.
If you're born distro, no matter what, start to run.
---- http://www.linuxprinting.org/ ---- http://tuxmobil.org/ |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Wed Feb 01, 2006 1:16 pm Post subject: |
|
|
It's just a shell variable, so set the variable in the environment before entering the script and tweak the script to only set the defaults if the environment variable doesnt exist. Look at any one of dozens of shell scripts on your system to see this being used in practice
Good luck |
|
Back to top |
|
|
Matteo Azzali Retired Dev
Joined: 23 Sep 2004 Posts: 1133
|
Posted: Thu Feb 02, 2006 9:08 pm Post subject: |
|
|
ewildgoose wrote: | It's just a shell variable, so set the variable in the environment before entering the script and tweak the script to only set the defaults if the environment variable doesnt exist. Look at any one of dozens of shell scripts on your system to see this being used in practice
Good luck |
Thank you, but that implies modifying the original script, thus forking the script for a GUI.I can do, but
doin so I would fork from original project, I'm instead asking Blinkeye to add this/some other kind of option
to make mkstage4 script gui-compatible, thus having cohordinated projects... (and thus having updated
mkstage4 script usable even for GUI users without any need of modification). _________________ Every day a new distro comes to birth. Every day a distro "eats" another.
If you're born distro, no matter what, start to run.
---- http://www.linuxprinting.org/ ---- http://tuxmobil.org/ |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Thu Feb 02, 2006 9:17 pm Post subject: |
|
|
The usual process on an open source project is that you just send him a diff. Remember that the original script is on a "wiki". You can presumably edit the darn thing yourself if you like...
It's hardly a "fork" though. It's a tiny modification and I'm sure it will be gratefully appreciated
Good luck!
Ed |
|
Back to top |
|
|
ssss25 n00b
Joined: 03 Feb 2006 Posts: 73
|
Posted: Mon Feb 06, 2006 3:59 pm Post subject: |
|
|
The script gets stuck! I never get the menu. After some debugging it hangs at:
find /mnt -name .keep
which is in the
Code: | default_include_files=" |
if I remove this, it works! Why would find get stuck!! if I copy and paste the line alone, it gets stuck too... any idea?
My other issues is that in default_include_files=, I added /usr/src but it didn't backup that directoy in full. it only did the linux folder.
How can I make this script tar EVERYTHING on the root partition! I only have one root part and one swap...
Thanks. |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Mon Feb 06, 2006 4:43 pm Post subject: |
|
|
Try running the command from the command line and see if it still breaks? I guess you could have some recursion in there which is killing the script. OR you could have some crookedly mounted filesystem in there with is nobbling the find command. |
|
Back to top |
|
|
ssss25 n00b
Joined: 03 Feb 2006 Posts: 73
|
Posted: Mon Feb 06, 2006 5:41 pm Post subject: |
|
|
command line fails too.... just hangs there. I couldn't even break...
I removed the line to make the script work! |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Mon Feb 06, 2006 5:44 pm Post subject: |
|
|
Well there you go. You have some filesystem mounted in /mnt which can't be enumarated. Probably it's some dead NFS mount or something else like that which hangs forever when you try and read it. Just look in /mnt for stuff that could be mounted and try using "ls" until you find the problem |
|
Back to top |
|
|
ssss25 n00b
Joined: 03 Feb 2006 Posts: 73
|
Posted: Tue Feb 07, 2006 12:32 am Post subject: |
|
|
Find is useless for me! I did this at the root to find where a file is, and I got segmentation fault
Code: | Ducky / # find . -name kdmrc -print
Segmentation fault
| [/code] |
|
Back to top |
|
|
ewildgoose Tux's lil' helper
Joined: 02 Mar 2003 Posts: 75
|
Posted: Tue Feb 07, 2006 8:08 am Post subject: |
|
|
Look, this is wayyy off topic now. The script is fine, its your system which is bolloxed. Ask in a new thread how to fix your problem. |
|
Back to top |
|
|
gnychis Veteran
Joined: 23 Mar 2005 Posts: 1004 Location: Pittsburgh, PA
|
Posted: Mon Mar 20, 2006 2:12 am Post subject: |
|
|
hi, whenever i run ./mkstage4(unmodified) and do a interactive backup with .tar.gz or .tar.bz2, i get this while trying to extract it:
"Tar: error exit delayed from previous errors"
then it exists, and it never extracts all the files... i've tried making the backup several times and extracting it several times, always the same error right when its about to start extracting /var
anyone else have this problem?
oh... and why doesn't it backup all of /dev, and not just /dev/console and /dev/null ?
Thanks!
George |
|
Back to top |
|
|
opopanax Apprentice
Joined: 30 Aug 2004 Posts: 244
|
Posted: Mon Mar 20, 2006 2:24 am Post subject: |
|
|
gnychis wrote: | oh... and why doesn't it backup all of /dev, and not just /dev/console and /dev/null ? |
Because it doesn't have to, with udev. you have the option to set up your system so that it has a back-up of the /dev filesystem, just in case you need it. /dev is dynamically created at boot, and generally you don't need it--unless you're still using devfs.
Please correct me if I'm wrong, I do tend to talk from the heinie once in a while. |
|
Back to top |
|
|
gnychis Veteran
Joined: 23 Mar 2005 Posts: 1004 Location: Pittsburgh, PA
|
Posted: Mon Mar 20, 2006 3:25 am Post subject: |
|
|
I followed the instructions on the first stage to restore from a stage4, before i chroot into the environment, i only have /dev/console and /dev/null ... when i chroot, /dev is filled up with everything. However when I reboot and try to boot into gentoo, it loads the kernel fine, it starts to boot gentoo, but the only things in dev are /dev/null and /dev/console again... so when it tries to load it can't because /dev/sda3 (my root partition), doesn't exist... what could be happening? |
|
Back to top |
|
|
BlinkEye Veteran
Joined: 21 Oct 2003 Posts: 1046 Location: Gentoo Forums
|
Posted: Mon Mar 20, 2006 6:39 am Post subject: |
|
|
gnychis wrote: | I followed the instructions on the first stage to restore from a stage4, before i chroot into the environment, i only have /dev/console and /dev/null ... when i chroot, /dev is filled up with everything. However when I reboot and try to boot into gentoo, it loads the kernel fine, it starts to boot gentoo, but the only things in dev are /dev/null and /dev/console again... so when it tries to load it can't because /dev/sda3 (my root partition), doesn't exist... what could be happening? |
I don't know exactly what the live cd does but it certainly creates some devices in /dev/ to provide a generic installation possibility. Udev should create the necessary devices upon booting - except you set in /etc/conf.d/rc that you want to use a device tarball. So, please check that file and make sure you use udev and NO device tarball. _________________ Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick |
|
Back to top |
|
|
|