HOWTO: Easily make a full system backup (stage4)

Message

BlinkEye · Post by **BlinkEye** » Sat Jul 02, 2005 12:08 pm

pseud answered the logic. and it's not the logic of the script but the one of tar:

man tar wrote:--totals
print total bytes written with --create

if you don't like it remove that parameter from the $tarOptions and add

du -sch $stage4Name.$stage4postfix

at the end of the script.

dundas · Post by **dundas** » Tue Jul 05, 2005 3:50 pm

[question]
Dear Blinkeye:

I just ran the script version=v3.5 as root on my gentoo,

Code: Select all

./mkstage4.sh -v -s

I chosed:

Code: Select all

Best (tar.bz2):
 (4) Interactive backup (added some small folders in the /home seperate partition, including /smb for samba)

and got the following ending errors:

3840215040 (3.6GiBï¼1.1MiB/s)

* stage4 is done
* umounting boot
* Checking integrity
* Integrity check failed. Re-run the script and che

And I supposed that the stage4 creation is successfull, my root partition got 300M left, but I suppose it is still not quite running out of space.

So is there any way for me to manually check the integrity again?
--just add ?

Code: Select all

echo "* md5sum archive:  `md5sum $stage4Name.$stage4postfix`"

and then use md5sum to check it?

Or, if u happen to know what might be done wrong to cause the integrity failure??

[solution]

hello dundas.

echo "* md5sum archive: `md5sum $stage4Name.$stage4postfix`"

this is just meant to be added to the script if you want to do a md5sum each time you run the script. an md5sum doesn't tell you if your stage4 has been built correctly - it just helps to verify upon burning and copying the stage4 if it is still the same as the original.

to test your stage4 manually execute (replace stage4name with your actual stage4 path):

cat stage4name.tar.bz2_* > stage4name.tar.bz2

# bzip2 --test stage4name.tar.bz2

or

# tar -tvf filename.tar.bz2

I tested using sheepdogj15 and blinkeye's suggestion as above, it goes ok, but returned this warning:

bzip2: (stdin): trailing garbage after EOF ignored

after discussing wiz them, and googling, we believe this is just a trivial warning, and my stage4 was made successfully, thx to you all!

ktech · Post by **ktech** » Tue Jul 05, 2005 6:11 pm

A little bit of off-topic:

do you guys know if I can restore a backup made in an Athlon XP 32 Bits into an Athlon 64 computer? I think I'm using the most restrictive of -marc/-mcpu (don't remember why as I'm not in that computer). Are they "binary compatible"?

Once I restore my backup I'll be compiling all of my apps into 64 bits, but I don't know if the backup will be running after restored...

Thanks!

Sheepdogj15 · Post by **Sheepdogj15** » Tue Jul 05, 2005 9:36 pm

dundas wrote:and got the following ending errors:

3840215040 (3.6GiBï¼1.1MiB/s)

* stage4 is done
* umounting boot
* Checking integrity
* Integrity check failed. Re-run the script and che
And I supposed that the stage4 creation is successfull, my root partition got 300M left, but I suppose it is still not quite running out of space.

So is there any way for me to manually check the integrity again?
--just add ?
Code: Select all
echo "* md5sum archive:  `md5sum $stage4Name.$stage4postfix`"
and then use md5sum to check it?

no, that wouldn't do you any good. md5sum doesn't actually error check the file itself: it just gives you a value based on some sort of algorithm ran against the file's binary value. this is good for ensuring there was no corruption from a file transfer (as the file you received should be exactly the same as the original, and thus will have the same md5sum value).

if you want to manually check for integrity, i suggest this command:

Code: Select all

# tar -tvf filename.tar.bz2

this will act like it is extracting the tarball, but it will actually only output names of files, and if you run into a corruption, it will die with an error.

my suspicion, though, is that the spliting process interfered with the error checking. i dunno though, i'll leave it to Blinkeye to figure out. what you could do is delete the resulting tarball(s), and run the script again.

Sheepdogj15 · Post by **Sheepdogj15** » Tue Jul 05, 2005 9:44 pm

ktech wrote:A little bit of off-topic:

do you guys know if I can restore a backup made in an Athlon XP 32 Bits into an Athlon 64 computer? I think I'm using the most restrictive of -marc/-mcpu (don't remember why as I'm not in that computer). Are they "binary compatible"?

Once I restore my backup I'll be compiling all of my apps into 64 bits, but I don't know if the backup will be running after restored...

yes. amd64 is compatible with x86 compiled binaries. in fact, if one wanted to use a pure 32bit environment on an Athlon 64, the recommendation would be to use -march=athlon-xp (there was a thread on this in the amd64 forum all about what CFLAGS and such to use).

the thing is, if you use the whole backup, you will have a 32bit Gentoo. if you want to compile in 64bit, you need to install from an amd64 stageX tarball. and it would be messy to just dump 32bit stuff on a 64bit system. you CAN run 32bit apps on 64bit Gentoo, it's just that you don't want to get things too mingled together (maybe what you could do is extact the backup to a separate partition, and then chroot in. the amd64 hardware docs explain this process pretty good).

Pseud · Post by **Pseud** » Sun Jul 10, 2005 10:07 am

BlinkEye,

I faced one small issue when I used the backup script to move an install from my desktop to my laptop: my hardware clock on the laptop was drifting, gaining something like 10 minutes every hour! (everything else went perfectly; thanks once again for the script!)
I searched the forums and found that the thing causing problems was the /etc/adjtime file. When I deleted this file and rebooted, the issue resolved itself (if the file is not found, it is regenerated upon reboot, and that fixes everything)

So, this is my suggestion:
Add /etc/adjtime to the default exclude list.
Reason: this is necessary if you plan on using your backup to copy your install to another machine; its not necessary to back this up because a reboot regenerates it anyway.

Thanks!

smurfd · Post by **smurfd** » Sun Jul 17, 2005 12:40 am

Wow, this script is awsome!! Thanks for sharing

funkoolow · Post by **funkoolow** » Sun Jul 17, 2005 6:57 pm

ehm, can't understand why, but actually the script won't let me pass the integrity check. that's the end of the output, mybe it's useful:

Code: Select all

/var/www/localhost/htdocs/the54.com-OLDSITE/modules/gallery/netpbm/ppmtojpeg
sh: line 1:  8883 Broken pipe             /usr/bin/find /* -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /mnt/p4-camera -prune -o -path /home/ftp -prune -o -path /home/funkoolow -prune -o -path /home/funkweb -prune -o -path /home/t0mcat -prune -o -path /usr/src/linux-2.6.11-gentoo-r9 -prune -o -print
sh: line 1:  8843 Done(141)               ( /usr/bin/find /var/db; echo /var/log/emerge.log; echo /usr/src; echo /usr/portage; echo /tmp; echo /sys; echo /proc; echo /mnt; echo /home; echo /dev/console; echo /dev/null; /usr/bin/find /usr/src/linux-2.6.11-gentoo-r9 -print; /usr/bin/find /home/t0mcat -print; /usr/bin/find /home/funkweb -print; /usr/bin/find /home/funkoolow -print; /usr/bin/find /home/ftp -print; /usr/bin/find /* -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /mnt/p4-camera -prune -o -path /home/ftp -prune -o -path /home/funkoolow -prune -o -path /home/funkweb -prune -o -path /home/t0mcat -prune -o -path /usr/src/linux-2.6.11-gentoo-r9 -prune -o -print )
      8845 Broken pipe             | /bin/tar --bzip2 --preserve-permissions --create --absolute-names --totals --ignore-failed-read -v --file /mnt/p4-camera/funkserver-stage4-2005.07.17-custom.tar.bz2 --no-recursion -T -

* stage4 is done
* umounting boot
* Checking integrity
* Integrity check failed. Re-run the script and check your hardware.

oh, i'm using the v3.5 in the best and custom modality (option 4)

smurfd · Post by **smurfd** » Tue Jul 19, 2005 6:20 pm

hmm, where would i change for it to Not add my /home to the .tar ?

scoobydu · Post by **scoobydu** » Wed Jul 20, 2005 11:20 am

Great script! Thanks

Just backed up my new powerbook install and restored it onto my mac mini!

Works a treat!

carpman · Post by **carpman** » Fri Jul 22, 2005 10:24 am

Love the script thanks

Just one small thing that could be added to recovery section, that is the use a portage snapshot before emerge sync. This would save some time plus reduce bandwidth usage on rsync servers.

dundas · Post by **dundas** » Sat Jul 23, 2005 3:14 pm

last updated Aug 4, 2005 wiz "modify /etc/resolv.conf"

Hi, I just thought maybe this is helpful to someone testing stage4 under vmware, I got it up wiz the following notes.

followed some of http://gentoo-wiki.com/HOWTO_Install_Ge ... s_NT/2K/XP

In my case I set up the vmware wiz similar partition like my stage4 backup.

/dev/sda1 /boot
/dev/sda2 swap
/dev/sda3 /
/dev/sda4 /home

1. boot off a live-cd and repartition and create filesystems as necessary
(remember to set active boot flag to /boot as well)
2. eventually reboot, using option: gentoo docache
3. umount /mnt/cdrom
4. remove the live-cd and insert the cd with the stage4
5. mount /dev/cdrom1 /mnt/cdrom
6. mount /dev/sdaX /mnt/gentoo #X stands for the number of primary partition u set for your root.
7. mkdir /mnt/gentoo/boot
8. mount /dev/sda1 /mnt/gentoo/boot
9. tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/
or
9. tar xjvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.bz2 -C /mnt/gentoo/
10. mkdir /mnt/gentoo/proc
11. mount -t proc none /mnt/gentoo/proc
12. mkdir /mnt/gentoo/dev
13. mount -o bind /dev /mnt/gentoo/dev
14. chroot /mnt/gentoo /bin/bash
15. env-update
16. source /etc/profile

if in need adjust necessary files (/etc/fstab, /boot/grub/grub.conf) and/or

install grub)

[EDIT Aug 4, 2005]modify /etc/resolv.conf and change the nameserver to the ones of this new network, if u dont do this, emerge --sync will report errors that:
(-2, 'Name or service not known')

(activate swap, not sure if this is necessary)
# mkswap /dev/hda2
# swapon /dev/hda2

17. emerge --sync (rebuild portage tree)
( may need to emerge grub)
# emerge grub
# cp /proc/mounts /etc/mtab

Now we can install GRUB using grub-install: I was stuck here while booting up wiz black screen, that's because I didn't install grub into mbr yet
# grub-install /dev/sda

then if grub-install /dev/sda doesn't work try

Code: Select all

# grub
grub> root (hd0,0) (since fedora /boot is on /sda1)
grub> setup (hd0) (still the mbr of the first hd)
grub> quit

maybe at this point we need to configure the kernel a bit to suit the new

system. u may refer to kernel config how-to.

18. exit
19. cd /
20. umount /mnt/cdrom
21. remove backup cd
22. umount /mnt/gentoo/boot
23. umount /mnt/gentoo/dev

[not sure why, but sometimes I just can't umount /mnt/gentoo/dev,

but...doesn't hurt much]

24. umount /mnt/gentoo/proc
25 umount /mnt/gentoo
26. Reboot

Other things are fine, need a bit configure to startx

[EDIT Jul 26, 2005][/url]
u may refer to mastabog's nice workthrough to do the rest.
http://forums.gentoo.org/viewtopic-t-24 ... art-0.html

p.s. if u had log for portage, u need to mkdir /var/log/portage again. And u should be careful that some file permission for your home directory (if u did backup them), and chown chmod accordingly.

so again, thx BlinkEye

Xerxes83 · Post by **Xerxes83** » Sat Jul 30, 2005 6:51 pm

Very nice script indeed! For my own backups I am using a modified version of a really old version of this script which uses DAR and does some other stuff as well. I was wondering which directories are absolutely necessary to get a system to work again after the bad stuff has happened. I hope it hasn't been asked already... but 10 pages of posts is just too much to read.

Are the following files really needed? And why?
/dev/null
/dev/console
/proc
/sys
/usr/portage

All the files in /dev and /proc can be re-created using a Gentoo live CD? Or can something go wrong in that case? And /usr/portage should restore itself with 'emerge sync' and /sys is empty on my system. I understand that it is more convenient to have them included in a backup, but since I am storing my backups on a remote server I prefer to have them as small as possible.

Stephonovich · Post by **Stephonovich** » Sun Jul 31, 2005 12:28 am

Worked great, with only a few problems.

First, portage complained that /var/log/portage doesn't exist. I did a quick mkdir and it fixed it.
Second, on boot, my gensplash image refuses to display, claiming that the filesystem / is mounted on doesn't contain a /dev/tty1. I checked, and it's there. echo "blah" > /dev/tty1 works as well. That probably isn't relevant to this support thread, but I thought I'd mention it anyway.

Matteo Azzali · Post by **Matteo Azzali** » Mon Aug 01, 2005 10:13 pm

Some strange socket ignored message, I never saw these:

Code: Select all

* mounting boot

Do you want to continue? (y/n) y

* Please wait while the stage4 is being created.

/bin/tar: /var/lib/dbus/system_bus_socket: socket ignorato
/bin/tar: /var/run/dbus/system_bus_socket: socket ignorato
/bin/tar: /var/run/acpid.socket: socket ignorato
/bin/tar: /var/run/xdmctl/dmctl/socket: socket ignorato
/bin/tar: /var/run/xdmctl/dmctl-\:0/socket: socket ignorato
Total bytes written: 2529454080 (2,4GiB, 2,9MiB/s)

* stage4 is done
* umounting boot
* Checking integrity
* Everything went smoothly. You successfully created a stage4.

Maybe it's cause my version (previous of this 3.5) wasn't goin in /var/ directories.
Is this ok???

BlinkEye · Post by **BlinkEye** » Tue Aug 02, 2005 5:50 am

yes, this is ok. but you've pointed me to a good idea - one could exclude the /var/run folder ...

Ma3oxuct · Post by **Ma3oxuct** » Thu Aug 18, 2005 3:30 am

I am not sure if you intended this or not, but when to the script are added excluded folders which riside in /home/{user} and the option "y" is chosen to back up /home/{user}, the folders previously excluded are not.

Here is what I am talking about:

Code: Select all

default_exclude_list="
...
/home/biter/.ut2004
/home/biter/.etwolf
...

It includes the excluded folders any way.

Code: Select all

/home/biter/.ut2004/
/home/biter/.ut2004/Cache/
/home/biter/.ut2004/Cache/0F0802C011D8FD9430000C8CB7637684-1.uxx
/home/biter/.ut2004/Cache/3ADC18814EB51B81C1B77E96E6F7CE63-1.uxx
/home/biter/.ut2004/Cache/C68430BC4794813072F223813183E4CA-1.uxx
/home/biter/.ut2004/Cache/8ECBE65540CF6B3C05930F841C17CC11-1.uxx
/home/biter/.ut2004/Cache/cache.ini
/home/biter/.ut2004/Cache/7492CDAD47C002364C6F408C429537FE-1.uxx

Here is precisly what I have been choosing:

Code: Select all

Cago-Ma3oxuct / # sh stage4-3.5.sh -v


Backup script v3.5
==================

What do you want to do? (Use CONTROL-C to abort)

Fast (tar.gz):
 (1) Minimal backup
 (2) Interactive backup


Best (tar.bz2):
 (3) Minimal backup
 (4) Interactive backup

Please enter your option: 2

Do you want to backup /home/biter? (y/n) y

Do you want to backup /home/common? (y/n) n

Do you want to backup /usr/src/linux-2.6.13-rc6? (y/n) y

* mounting boot

* creating the stage4 in /work/stage4-backups with the following command:

(/usr/bin/find /var/db; echo /var/log/emerge.log; echo /usr/src; echo /usr/portage; echo /tmp; echo /sys; echo /proc; echo /mnt; echo /home; echo /dev/console; echo /dev/null; /usr/bin/find /usr/src/linux-2.6.13-rc6 -print; /usr/bin/find /home/biter -print; /usr/bin/find /bin /boot /dev /etc /home /in /lib /mnt /opt /proc /root /sbin /stage4-3.5.sh /stage4.sh /sys /tmp /usr /var /work -path /home/common -prune -o -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /opt/ut2004 -prune -o -path /opt/enemy-territory -prune -o -path /home/biter/.ut2004 -prune -o -path /home/biter/.etwolf -prune -o -path /in -prune -o -path /work -prune -o -path /work/stage4-backups -prune -o -path /home/biter -prune -o -path /home/common -prune -o -path /usr/src/linux-2.6.13-rc6 -prune -o -print) | /bin/tar --gzip --preserve-permissions --create --absolute-names --totals --ignore-failed-read -v --file /work/stage4-backups/Cago-Ma3oxuct-stage4-2005.08.17-custom.tar.gz --no-recursion -T -
Do you want to continue? (y/n) y

abracadabra · Post by **abracadabra** » Tue Aug 23, 2005 4:27 pm

I get this when i try and dl the script:

~ $ wget ftp://blinkeye.ch/gentoo/mkstage4.sh
--17:19:46-- ftp://blinkeye.ch/gentoo/mkstage4.sh
=> `mkstage4.sh'
Resolving blinkeye.ch... 213.202.33.122
Connecting to blinkeye.ch[213.202.33.122]:21... connected.
Logging in as anonymous ...
Login incorrect.

Could somebody please post a copy of an up-to-date script? Thanks

funkoolow · Post by **funkoolow** » Tue Aug 23, 2005 7:21 pm

http://temp.the54.com

the latest v3.5 is the mkstage4.sh, i've just changed the parameter stage4Location=/mnt/p4-camera to fit my needs, the rest is absolutely intact. note that these will not be available during night-times at CET, since i turn off my machine. Just in case, pm me an email and i'll send them you.

NightTwix · Post by **NightTwix** » Tue Aug 23, 2005 7:42 pm

abracadabra, you could also copy&paste the script from the inital posting.
But beware of wrapping and linefeeds

BlinkEye · Post by **BlinkEye** » Tue Aug 23, 2005 7:52 pm

ftp is working again. sorry for the inconvenience - i assembled a new server and had to switch some harddisk but forgot to adjust the ftp paths (partition changes).

slydini · Post by **slydini** » Tue Aug 23, 2005 10:52 pm

Anyone know how to use this script on say RED Hat 9? What would the exclude list look like?

Also I am trying to do a full disk to disk backup can this script do something like that?

BlinkEye · Post by **BlinkEye** » Tue Aug 23, 2005 11:12 pm

well, the script was made for that purpose. i haven't used red hat for a long time, but you probably know where cached/temporary/downloaded files are stored and exlude those instead of the portage specific ones.

riczan · Post by **riczan** » Sat Aug 27, 2005 3:37 pm

Great script! Thanks. I just moved my gentoo to my new 160GB hard drive and it worked flawlessly EXCEPT for:

this might be a bug or my mistake. Please clarify

- I consider my kernel source to be part of minimal so:

- I erased /usr/src from the default_exclude_list

and the result was that everything on /usr/src was backed up except for the source kernel I care:

Code: Select all

original
-rw-r--r--   1 root root    0 Oct 27  2004 .keep
lrwxr-xr-x   1 root root   31 Mar 21 13:24 linux -> /usr/src/linux-2.6.10-gentoo-r6
drwxr-xr-x  19 root root 4096 Mar 26 12:46 linux-2.6.10-gentoo-r6
drwxr-xr-x  18 root root 4096 Mar 26 22:15 linux-2.6.11-gentoo-r4
drwxr-xr-x  19 root root 4096 Mar 21 13:27 linux-2.6.9-gentoo-r1

and the backed up result

Code: Select all

-rw-r--r--   1 root root    0 Oct 27  2004 .keep
lrwxrwxrwx   1 root root   31 Aug 26 08:48 linux -> /usr/src/linux-2.6.10-gentoo-r6
drwxr-xr-x  18 root root 4096 Mar 26 22:15 linux-2.6.11-gentoo-r4
drwxr-xr-x  19 root root 4096 Mar 21 13:27 linux-2.6.9-gentoo-r1

Please let me know what I did wrong or if this is a bug

Thanks

dinomyte66 · Post by **dinomyte66** » Mon Aug 29, 2005 2:09 am

Nice program, works well, but---since running it I have lost my /boot/grub/grub.conf file. Has anything like this happened to anyone before?
I just happened to be searching for the new backup tar and when I went into my boot directory I noticed my grub config was gone! I know it was there earlier because I had modified it earlier in the day with a reboot parameter. I rebooted my machine several times today to get xorg working so it had to be there all along---is it possible that more than just this file is gone?

Please help---It's probably something I did but it seemed like everything went as it was suppose to.

Should I just recreate my grub.conf in the meantime?

Thanks in advance for all of your help,
Dino

HOWTO: Easily make a full system backup (stage4)

trivial warning [solved]

restore under vmware