HOWTO: Easily make a full system backup (stage4)

[BlinkEye]

That seems to be working now, but I thought of another thing...

Perhaps there could be a progress bar incorporated into the script... It could just be along the lines of files left to do / total files to get the percentage done. Obviously a better way would be the size of files, but it'd be a good start at least.

Is that do-able do you think?

yes, i think so. if you use the script with the verbose option ( -v/ --verbose ) you see the command it does. removing the tar command and combining it with

Code: Select all

 | wc -l 

gives you the amount of files the command will process to tar. for one box of mine this would be about 166000 files (counted within a few seconds) - so, still no loss. i think the number of files suffice for the progress bar. now, using the vebose option will give you the files processed by tar - and from there one could create this progress bar. i need a command that can say, print a '#' for every 100 lines being output on stdout. any ideas?

[BlinkEye]

Seems to have started working but still have one issue, /var/log/portage i created this and given it lots chmod options, it is currently 777 but still emerges complain it is not writable? i set it to owner/group 'portage' as is on other systems but no joy?

PS great script otherwise

Cheers

could it be you you created a file instead of a directory?

Code: Select all

drwxrws---  2 portage portage   114136 Jan 16 00:28 portage

[gnychis]

for some reason my custom exclude folders do not seem to be working

I try to exclude /160gb and /200gb, however whenever I run the script in verbose mode it is recursively backing up all of /160gb and /200gb

when i start the script i use option 4

here is my file:

Code: Select all

#!/bin/bash 
# Backup script for Gentoo Linux 
# Copyright Reto Glauser aka Blinkeye 
# Distributed under the terms of the GNU General Public License v2
# Mailto: stage4 at blinkeye dot ch
# Forum post: http://forums.gentoo.org/viewtopic-t-312817.html 
# Date: 2005-06-30

version=v3.5
basename=`basename $0`

find=/usr/bin/find
tar=/bin/tar

# these are the commands we actually need for the backup
command_list=(cut date echo $find grep hostname mount sh split $tar umount uname which)

# verify that each command we use exists. if one can't be found use $PATH and make a suggestion if possible.
for command in ${command_list[@]}; do
	if [ ! -x "`which $command 2>&1`" ]; then
		echo -e "\nERROR: $command not found! "
		base=`basename $command`
 		if [ "`which $base 2>&1 | grep "no \`basename $command\` in"`" != "" ]; then
 			echo -e "ERROR: $base is not in your \$PATH."
 		fi
 		exit -1 
	fi
done

help="\nUsage:\n\nsh `basename $0` [[-v]|[--verbose]] [[-s]|[--split]] \n\nTo run the script NOT in verbose mode comes in handy if you want to see only the errors that occur during the backup.\n"

# Defaults to creating one tarball
tar_output="--file"

# split command
split_options="--suffix-length=1 --bytes=685m"

# options for the tar command 
tarOptions=" --preserve-permissions --create --absolute-names --totals --ignore-failed-read" 

# where to put the stage4
stage4Location=/mnt/backups/stage4

# name prefix
stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d`

# patterns which should not be backed up (like iso files).
# example: default_exclude_pattern="*.iso *.divx"
# These pattern count only for files NOT listed in the $custom_include_list.
default_exclude_pattern=""

# these files/directories are always excluded. don't add trailing slashes.
# don't touch it unless you know what you are doing!
# /var/db and /var/cache/edb are intentionally added here. they are listed
# in $default_include_folders
default_exclude_list="
/dev
/lost+found
/mnt
/proc
/sys
/tmp
/usr/portage
/usr/src
/var/log
/var/tmp
/var/db
/var/cache/edb
$stage4Location
`echo $CCACHE_DIR`"

# files/devices/folders, which need to be backed up (preserve folder structure). 
# don't touch it unless you know what you are doing! no recursive backup of folders. 
# use $default_include_folders instead.
default_include_files="
/dev/null
/dev/console
/home
/mnt
/proc
/sys
/tmp
/usr/portage
/usr/src
/var/log/emerge.log"

# folders, which need to be backed up recursively on every backup. 
# don't touch it unless you know what you are doing! the reason for this
# variable is that some users add /var to the $default_exclude_list. here 
# we ensure that portage's memory is backed up in any case.
default_include_folders="
/var/db"

# IMPORTANT: A minimal backup will EXCLUDE files/folders listed here. A custom backup will
# include/exclude these files/folders depening on your answer.
custom_include_list="
/home/*
/usr/src/linux-`uname -r`"

# add files/folders here which are subfolders of a folder listed in $custom_include_list which should NOT
# be backed up. eg. 
#custom_exclude_list="/home/foo/mp3 /home/foo/downloads /home/foo/.*"
custom_exclude_list="/home/hedpe/server /home/hedpe/ut2004 /home/hedpe/downloads /home/hedpe/.rstw16 /200gb /160gb"

# Only files/folders within the $custom_include_list are checked against these patterns
# custom_exclude_pattern="*.mp3 *.iso"
custom_exclude_pattern="*.img *.VOB *.BUP *.mp3 *.iso *.gcm"

# the find_command
find_command="$find /*"

# don't backup anything which matches pattern listed in $default_exclude_pattern
for pattern in $default_exclude_pattern; do
	find_command="$find_command -not -name $pattern"
done

# assemble the find_command 
function find_files()
{
	for folder in $default_exclude_list; do
		find_command="$find_command -path $folder -prune -o"
	done

	find_command="$find_command -print"

	for i in $default_include_files; do
		find_command="echo $i; $find_command"
	done	

	for i in $default_include_folders; do
		if [ -d $i ]; then 
			find_command="$find $i; $find_command"
		else
			find_command="echo $i; $find_command"
		fi		
	done
}

# check the exclude/include variables for non-existing entries
function verify()
{
	for i in $1; do
		if [ ! -e "`echo "$i" | cut -d'=' -f2 | cut -d'*' -f1`" -a "$i" != "/lost+found" -a "$i" != "$stage4Location" ]; then
			echo "ERROR: `echo "$i" | cut -d'=' -f2` not found! Check your "$2
			exit 0
		fi
	done	
}

# check input parameters
while [ $1 ]; do
	case  $1 in
	"-h" | "--help")
		echo -e $help
		exit 0;;
	"-v" | "--verbose")
		verbose=$1;;
	"-s" | "--split")
		tar_output="--split";;
	"");;
	*) 
		echo -e $help
		exit 0;;
	esac
	shift
done

echo ""

# check folder/files listed in $default_exclude_list exist
verify "$default_exclude_list" "\$default_exclude_list"

# check files listed in $default_include_files exist
verify "$default_include_files" "\$default_include_files"

# check folder listed in $default_include_folders exist
verify "$default_include_folders" "\$default_include_folders"

#check folder listed in $custom_include_list exist
verify "$custom_include_list" "\$custom_include_list"

#check folder listed in $custom_exclude_list exist
verify "$custom_exclude_list" "\$custom_exclude_list"
 
# print out the version
 echo -e "\nBackup script $version"
 echo -e "=================="

# how do you want to backup?
echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n
Fast (tar.gz):
 (1) Minimal backup
 (2) Interactive backup


Best (tar.bz2):
 (3) Minimal backup
 (4) Interactive backup\n"

while [ "$option" != '1' -a "$option" != '2' -a "$option" != '3' -a "$option" != '4' ]; do
	echo -en "Please enter your option: "
	read option
done

case $option in
[1,3])
	stage4Name=$stage4Location/$stage4prefix-minimal.tar;;

[2,4])
	stage4Name=$stage4Location/$stage4prefix-custom.tar

	for folder in $custom_include_list; do 
		echo -en "\nDo you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) " 
		read answer 
		while [ "$answer" != 'y' -a "$answer" != 'n' ]; do 
			echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) " 
			read answer 
		done 
		if [ "$answer" == 'n' ]; then 
			find_command="$find_command -path $folder -prune -o"
		else
			custom_find="$find $folder"
			for i in $custom_exclude_pattern; do
				custom_find="$custom_find -name $i -o"
			done
			for i in $custom_exclude_list; do
				custom_find="$custom_find -path $i -prune -o"
			done
			find_command="$custom_find -print; $find_command"
		fi
	done ;;
esac

# add $custom_include_list to the $default_exclude_list as we assembled
# $custom_find with $custom_include_list already.
default_exclude_list="$default_exclude_list $custom_include_list"

case $option in
[1,2])
	stage4postfix="gz"
	zip="--gzip";;
	
[3,4])
	stage4postfix="bz2" 
	zip="--bzip2";;
esac

# mount boot
echo -e "\n* mounting boot" 
mount /boot >/dev/null 2>&1

# find the files/folder to backup
find_files
find_command="($find_command)"

# create the final command
if [ "$tar_output" == "--file" ]; then
	tar_command="$find_command | $tar $zip $tarOptions $verbose --file $stage4Name.$stage4postfix --no-recursion -T -"
else
	tar_command="$find_command | $tar $zip $tarOptions $verbose --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"
fi

if [ "$verbose" ]; then
	echo -ne "\n* creating the stage4 in $stage4Location with the following command:\n\n"$tar_command
fi

# everything is set, are you sure to continue?
echo -ne "\nDo you want to continue? (y/n) " 
read answer 
while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do 
	echo -ne "Do you want to continue? (y/n) " 
	read answer 
done 

if [ "$answer" == 'y' ]; then 
	# check whether the file already exists.
	if [ "$tar_output" == "--split" ]; then
		overwrite="`ls "$stage4Name.$stage4postfix"_* 2>&1 | grep -v 'No such file'`"
	else
		overwrite="$stage4Name.$stage4postfix"
	fi
	
	if [ -a "`echo "$overwrite" | grep "$overwrite" -m1`" ]; then  
		echo -en "\nDo you want to overwrite $overwrite? (y/n) "
		read answer
		while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do
			echo -en "Do you want to overwrite $overwrite? (y/n) "
			read answer
		done
		if [ "$answer" == 'n' ]; then
			echo -e "\n* There's nothing to do ... Exiting" 
			exit 0;
		fi
	fi
	
	# if necessary, create the stage4Location 
	if [ ! -d "$stage4Location" ] ; then 
		echo "* creating directory $stage4Location" 
		mkdir -p $stage4Location 
	fi

	echo -e "\n* Please wait while the stage4 is being created.\n"

	# do the backup.	
	sh -c "$tar_command"
	
	# finished, clean up
	echo -e "\n* stage4 is done" 
	echo "* umounting boot" 
	umount /boot >/dev/null 2>&1

	# Integrity check
	echo -e "* Checking integrity"
	if [ "$zip" == "--gzip" ]; then
		zip="gzip"
	else
		zip="bzip2"
	fi	
	
	if [ "$tar_output" == "--split" ]; then
		if [ "`cat "$stage4Name.$stage4postfix"_*"" | $zip --test 2>&1`" != "" ]; then
			echo -e "* Integrity check failed. Re-run the script and check your hardware."
			exit -1
		fi
	else
		if [ "`$zip --test  $stage4Name.$stage4postfix 2>&1`" != "" ]; then
			echo -e "* Integrity check failed. Re-run the script and check your hardware."
			exit -1
		fi
	fi
	
	# everything went smoothly"
	echo -e "* Everything went smoothly. You successfully created a stage4."
	
else
	echo -e "\n* There's nothing to do ... Exiting" 
fi 

# Split the archive into chunks - uncomment the 3 lines if you want to split the stage4
# echo -e "* split $stage4Name.$stage4postfix" 
# split $split_options $stage4Name.$stage4postfix "$stage4Name.$stage4postfix"_ 
# echo "* splitting is done"

[gnychis]

problem solved by adding the folders to default_exclude_list

[mb]

Hi *,

I've got a small suggestion for your (excellent) script. Maybe you should add the dhcpcd cache files to the default exclude list (/var/lib/dhcdc/dhcpcd*.cache). I made a stage4 file on my test server at home, transfered it to my root server, rebooted.. and.. lost connection, because the root server took the (old) ip from the cache file...

#mb

[ewildgoose]

My apologies if this has already been mentioned, but also check out app-arch/star for an interesting alternative to "tar". This alternative app has support for backing up and restoring selinux style extra attributes (which I need in my case) - It may also have the overlapping exclude/include support that tar is also missing?

I mentioned kdar already, but that's another interesting alternative as well.

Nice script though

[BlinkEye]

thanks for the hint about star. as any page about star has been moved i cannot test if it is fully compatible with the tar options used in my script.

[ewildgoose]

Top hit in google
http://cdrecord.berlios.de/old/private/man/star.html

I am playing with it right now, and at first sight it's not quite right. Firstly it clobbers your existing tar binary when it installs and the options are not similar enough that this is acceptable. The only thing I can see which might actually help in the end is:

- Multi-archive
- Advanced regexp for excludes
- Archiving of linux attributes

So kdar still looks like a better solution, but the negative there was getting a binary when you needed to restore the system in a hurry (from scratch).

Unsure...

[ewildgoose]

Hmm, just noticed an interesting feature at:
http://www.gnu.org/software/tar/manual/ ... html#SEC57

If you "cat" together two archives then try and list the files you will only see the files from the first archive. However, if you add the "-i" option then you can "workaround this" and extract the entire archive. I just tried creating two tar files, then cat'ing them together, then gziping them, and then I could read the whole thing with "tar -tif test.tar.gz"

It's a bit of a hack, but interesting...

Ed

[BlinkEye]

Top hit in google
http://cdrecord.berlios.de/old/private/man/star.html.

i doubt that. and the link doesn't work anyway (only cached). neither does http://www.fokus.gmd.de/research/cc/glo ... /star.html which is the homepage referenced by portage. but thanks for checking it out.

[ewildgoose]

The link *I* posted works fine. The one *you* posted has an extra "." at the end?

I dont see that star actually solves the problem in this case, but it does offer a possible solution for selinux backups... Also the concatenating two tar archives together thing nearly works, but I can't quite see how to efficiently stream the output of two tar commands into bzip? Any thoughts?

Ed

[Matteo Azzali]

Hum,Blinkeye, sorry to bother, but I would like to build a visual frontend in KDE for this wonderful script,
(Kommander script, check AviUtils or Kfirewatcher to see what I've done before) and I was
swearing if there is some better option to add dirs in default_exclude_list other than modifying the
script itself (an option at script startup? a parameter from command line? )

[ewildgoose]

It's just a shell variable, so set the variable in the environment before entering the script and tweak the script to only set the defaults if the environment variable doesnt exist. Look at any one of dozens of shell scripts on your system to see this being used in practice

Good luck

[Matteo Azzali]

It's just a shell variable, so set the variable in the environment before entering the script and tweak the script to only set the defaults if the environment variable doesnt exist. Look at any one of dozens of shell scripts on your system to see this being used in practice

Good luck

Thank you, but that implies modifying the original script, thus forking the script for a GUI.I can do, but
doin so I would fork from original project, I'm instead asking Blinkeye to add this/some other kind of option
to make mkstage4 script gui-compatible, thus having cohordinated projects... (and thus having updated
mkstage4 script usable even for GUI users without any need of modification).

[ewildgoose]

The usual process on an open source project is that you just send him a diff. Remember that the original script is on a "wiki". You can presumably edit the darn thing yourself if you like...

It's hardly a "fork" though. It's a tiny modification and I'm sure it will be gratefully appreciated

Good luck!

Ed

[ssss25]

The script gets stuck! I never get the menu. After some debugging it hangs at:

find /mnt -name .keep

which is in the

Code: Select all

default_include_files="

if I remove this, it works! Why would find get stuck!! if I copy and paste the line alone, it gets stuck too... any idea?

My other issues is that in default_include_files=, I added /usr/src but it didn't backup that directoy in full. it only did the linux folder.

How can I make this script tar EVERYTHING on the root partition! I only have one root part and one swap...

Thanks.

[ewildgoose]

Try running the command from the command line and see if it still breaks? I guess you could have some recursion in there which is killing the script. OR you could have some crookedly mounted filesystem in there with is nobbling the find command.

[ssss25]

command line fails too.... just hangs there. I couldn't even break...

I removed the line to make the script work!

[ewildgoose]

Well there you go. You have some filesystem mounted in /mnt which can't be enumarated. Probably it's some dead NFS mount or something else like that which hangs forever when you try and read it. Just look in /mnt for stuff that could be mounted and try using "ls" until you find the problem

[ssss25]

Find is useless for me! I did this at the root to find where a file is, and I got segmentation fault

Code: Select all

Ducky / # find . -name kdmrc -print
Segmentation fault

[/code]

[ewildgoose]

Look, this is wayyy off topic now. The script is fine, its your system which is bolloxed. Ask in a new thread how to fix your problem.

[gnychis]

hi, whenever i run ./mkstage4(unmodified) and do a interactive backup with .tar.gz or .tar.bz2, i get this while trying to extract it:
"Tar: error exit delayed from previous errors"

then it exists, and it never extracts all the files... i've tried making the backup several times and extracting it several times, always the same error right when its about to start extracting /var

anyone else have this problem?

oh... and why doesn't it backup all of /dev, and not just /dev/console and /dev/null ?

Thanks!
George

[opopanax]

oh... and why doesn't it backup all of /dev, and not just /dev/console and /dev/null ?

Because it doesn't have to, with udev. you have the option to set up your system so that it has a back-up of the /dev filesystem, just in case you need it. /dev is dynamically created at boot, and generally you don't need it--unless you're still using devfs.

Please correct me if I'm wrong, I do tend to talk from the heinie once in a while.

[gnychis]

I followed the instructions on the first stage to restore from a stage4, before i chroot into the environment, i only have /dev/console and /dev/null ... when i chroot, /dev is filled up with everything. However when I reboot and try to boot into gentoo, it loads the kernel fine, it starts to boot gentoo, but the only things in dev are /dev/null and /dev/console again... so when it tries to load it can't because /dev/sda3 (my root partition), doesn't exist... what could be happening?

[BlinkEye]

I followed the instructions on the first stage to restore from a stage4, before i chroot into the environment, i only have /dev/console and /dev/null ... when i chroot, /dev is filled up with everything. However when I reboot and try to boot into gentoo, it loads the kernel fine, it starts to boot gentoo, but the only things in dev are /dev/null and /dev/console again... so when it tries to load it can't because /dev/sda3 (my root partition), doesn't exist... what could be happening?

I don't know exactly what the live cd does but it certainly creates some devices in /dev/ to provide a generic installation possibility. Udev should create the necessary devices upon booting - except you set in /etc/conf.d/rc that you want to use a device tarball. So, please check that file and make sure you use udev and NO device tarball.