[Howto] Creation of a Stage 5 archive

[d2_racing]

Last Update : 2009-02-04

Howto Creation of a Stage 5

Table of Contents

1. Context
2. Differences between the Stage 4 and the Stage 5
3. Creation of a Stage 5
   1. Creation of a Stage 5 Standard
   2. Creation of a Stage 5 Custom
      • Advantages & Inconveniences
      • Possible Options
4. Copy your Stage 5 archive
5. Restoration of a Stage 5
   1. Restoration of a Stage 5 Standard
   2. Restoration of a Stage 5 Custom

1) Context

Hello everyone, because of a awesome crash after a restoration of a Stage 4 archive, I decided to write this howto for those who don't wont problems.

The Stage 4 works perfectly in general, but if you are lucky like me, it will crash your Gentoo
So, if you have a Gensplash, a /mnt directory, you will have a problem with the Stage 4, because he doesn't backup these directory by default.

Here's my configuration :

Code: Select all

#
# Sample boot menu configuration file
#

# Boot automatically after 30 secs.
timeout 5
# By default, boot the first entry.
default 1
splashimage=(hd0,8)/boot/grub/splash.xpm.gz

title=GentooLinux-2.6.18-r3
root=(hd0,8)
kernel /boot/kernel-2.6.18-gentoo-r3 root=/dev/sda9 video=vesafb:mtrr,ywrap,1024x768-32@85 splash=silent,fadein,theme:livecd-2006.1 quiet CONSOLE=/dev/tty1
initrd (hd0,8)/boot/fbsplash-livecd-2006.1
#kernel /boot/kernel-2.6.18-gentoo-r3 root=/dev/sda9 video=radeonfb:mtrr,ywrap,1024x768-32@75


title=WindowsXP
rootnoverify (hd0,0)
makeactive
chainloader +1

title=GentooLinux-2.6.18-r2
root=(hd0,8)
kernel /boot/kernel-2.6.18-gentoo-r2 root=/dev/sda9 video=radeonfb:mtrr,ywrap,1024x768-32@75 

Code: Select all

# This file is edited by fstab-sync - see 'man fstab-sync' for details
# /etc/fstab: static file system information.
# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/fstab,v 1.14 2003/10/13 20:03:38 azarah Exp $
#
# noatime turns off atimes for increased performance (atimes normally aren't
# needed; notail increases performance of ReiserFS (at the expense of storage
# efficiency).  It's safe to drop the noatime options if you want and to
# switch between notail and tail freely.

# <fs>             <mountpoint>    <type>     <opts>            <dump/pass>
/dev/sda9      /      ext3      defaults,noatime   0 1
/dev/sda10      none      swap      sw         0 0
none         /proc      proc      defaults      0 0   
none         /dev/shm   tmpfs      nodev,nosuid,noexec   0 0
/dev/cdroms/cdrom0   /mnt/cdrom0   auto      noauto,user      0 0
/dev/cdroms/cdrom1   /mnt/cdrom1   auto      noauto,user      0 0
/dev/sda8      /mnt/pont   vfat      defaults,rw,user,umask=0 0 0
/dev/sda1      /mnt/win_c   ntfs      defaults,ro,user,nls=iso8859-1,nls=utf8,umask=0 0 0
/dev/sda5      /mnt/win_d   ntfs      defaults,ro,user,nls=iso8859-1,nls=utf8,umask=0 0 0   
/dev/sda6      /mnt/win_e   ntfs      defaults,ro,user,nls=iso8859-1,nls=utf8,umask=0 0 0
/dev/sda7      /mnt/win_f   ntfs      defaults,ro,user,nls=iso8859-1,nls=utf8,umask=0 0 0
# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
#/dev/BOOT      /boot      ext2      noauto,noatime      1 1
#/dev/ROOT      /      xfs      noatime         0 0
#/dev/SWAP      none      swap      sw         0 0
#/dev/cdroms/cdrom0   /mnt/cdrom   iso9660      noauto,ro      0 0
#/dev/fd0      /mnt/floppy   auto      noauto         0 0
#
# NOTE: The next line is critical for boot!
#none         /proc      proc      defaults      0 0
#
# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
#  use almost no memory if not populated with files)
# Adding the following line to /etc/fstab should take care of this:
#
#none         /dev/shm   tmpfs      defaults      0 0
#
#/dev/hdd                /media/cdrom            auto    user,exec,noauto,managed 0 0
#/dev/hdc                /media/cdrecorder       auto    user,exec,noauto,managed 0 0
#/dev/floppy/0           /media/floppy           auto    user,exec,noauto,managed 0 0
#/dev/sdb1               /media/sdb1             auto    user,exec,noauto,managed 0 0 

So, because of a catastrophic restoration, I decided to create a Stage 5 with the command tar and this will give a valid Stage 5 and a user will be able to restore in no time
and it will boot perfectly without any errors.

For the record, I will list the problems that I had after a restore of Stage 4 backup:
-A lot of messages about udev.rule, UDEV-event #.
-The script didn't backup /mnt, so in my case, I had a warning about the FSTAB.
-For a unknown reason, my Gensplash didn't start, because of a missing /dev/TTY1...so my Gensplash was disable.

2) Differences between the Stage 4 and the Stage 5

First of all, you can find the excellent Stage 4 script here : http://blinkeye.ch/mediawiki/index.php/ ... 8stage4%29

The Stage 4, is a script that backup only the necessary directories to have a minimal functionable Gentoo.So, the Stage 4 doesn't take a ghost of a partition.Also, the Stage 4 never backup the directory /dev and if you have a Gensplash, you won't have /dev/console and /dev/tty1, and it will crash your Gensplash.

The Stage 5, is only uses the tar command, so you can do what you want.In fact, you can take a ghost of a partition or only take a backup of certain directory.

Like the Stage 4, the Stage 5 have a integrity check and the end of the process.

Finally, I wrote the Stage 5 for that purpose and I choosed the next number for the Stage.

3) Creation of Stage 5

Creation of a Stage 5 Standard

Code: Select all

# su -
# rm /usr/portage/distfiles/*
# mkdir /mnt/backup
# mount -o bind / /mnt/backup
# cd /mnt/backup
# tar --exclude  stage5.tar.bz2 -cvvjpf stage5.tar.bz2 .
# time bzip2 -tv stage5.tar.bz2 
# umount /mnt/backup 

By the way, I saw a little bug inside my current box.

If you have a /boot, before starting your tar command, double check inside /mnt/back/boot if you have something.

If not, run this command :

Code: Select all

# mount -o bind  /boot /mnt/backup/boot 

The bind command is only working for a specific partition at once.

So, I had a / for sda8 and a /boot for sda7 for example.

The "." after the .bz2 is very important
The bzip2 command will check the integrity of the archive.
The creation of the .tar.bz2 takes some time, so be patient.

The Stage 5 act like Ghost or Acronis True Image.In fact, you can backup a complete partition.With this stage, you can backup a complete HDD and transfer it to a new one without any problems.The technique can be use by administrator, because in case of a crash, they can restore only the partition that failed.

Creation of a Stage 5 Custom

The Stage 5 custom, is a Stage 5 with parameters.

Advantages & Inconveniences

1. This can backup a Gentoo installation or a complete partition and it can be use to backup only certain directories.
2. It will be much faster, because it doesn't backup all the directories.
3. You have to be very careful when you exclude directories, because it can crash your restoration, because you may have remove some files or directories that were essential for your Gentoo.

Possible options

Here a example of a command that exclude /tmp,/var/tmp & /usr/portage/distfiles
Note that you don't have to put / before the directory, because you have binded the /

Code: Select all

# tar --exclude tmp --exclude var/tmp --exclude usr/portage/distfiles --exclude stage5.tar.bz2 -cvvjpf stage5.tar.bz2 .

Also, certain users wont to have a Stage 5 that contain only the minimum directory, so for that here an example :

Code: Select all

# rm -rf /usr/portage/distfiles/* && rm -rf /var/tmp/portage/* && rm -rf /var/log/portage/* 
# tar cvvjpf --exclude=dev --exclude=proc --exclude=sys --exclude=tmp --exclude=stage5.tar.bz2  stage5.tar.bz2 .

But, with this command, you will have to do some extra command in the restoration, because you will have to recreate certain directories.

Code: Select all

# mkdir {dev,proc,tmp,sys} && chmod 1777 tmp && cd dev && MAKEDEV generic

Also, many users wont to backup only their /home directory.So here's the command :

Code: Select all

# mount -o bind /home/$user /mnt/backup
# cd /mnt/backup
# tar --exclude stage5.tar.bz2 -cvvjpf stage5.tar.bz2 .

You can also backup everything except the /home directory :

Code: Select all

# mount -o bind / /mnt/backup
# cd /mnt/backup
# tar --exclude stage5.tar.bz2 --exclude home/$user -cvvjpf stage5.tar.bz2 .
# time bzip2 -tv stage5.tar.bz2 

# umount /mnt/backup
# mount -o bind /home/$user /mnt/backup
# cd /mnt/backup
# tar --exclude stagehome.tar.bz2 -cvvjpf stagehome.tar.bz2 .
# time bzip2 -tv stagehome.tar.bz2 
# umount /mnt/backup
# mkdir /mnt/backup/full
# cd /
# cp stage5.tar.bz2 /mnt/backup/full/
# cp /home/$user/stagehome.tar.bz2 /mnt/backup/full
# tar -cvf stagefull.tar stage5.tar.bz2 stagehome.tar.bz2

4)Copy your Stage 5 archive

Now, you can burn the .tar.bz2 with K3B or you can leave this image on a separate partition.

5)Restoration of a Stage 5


Restoration of a Stage 5 Standard

The first thing to do is to boot with the Gentoo LiveCD and use the docache option.This will be useful, because we need
to umount the LiveCD

Boot with the Gentoo LiveCD.

Code: Select all

#gentoo docache

Now you have to format the destination partition

Code: Select all

# mke2fs -j /dev/sda9 dans mon cas.

#umount /mnt/cdrom

Remove the Gentoo LiveCD and put on your Stage 5 DVD backup.

Code: Select all

#mount /dev/cdrom /mnt/cdrom

#mount -t ext3 /dev/sda9 /mnt/gentoo dans mon cas

# cd /mnt/cdrom
# ls -la

# tar xjvpf /mnt/cdrom/stage5.tar.bz2 -C /mnt/gentoo/
# mount -t proc none /mnt/gentoo/proc
# mount -o bind /dev /mnt/gentoo/dev

# chroot /mnt/gentoo /bin/bash
# env-update
# source /etc/profile
# emerge --sync
# exit
# cd /
# umount /mnt/cdrom
# umount /mnt/gentoo/dev
# umount /mnt/gentoo/proc
# umount /mnt/gentoo
# reboot

Now you will have a Gentoo back on track

Restoration of a Stage 5 Custom

The procedure of restoration of a Stage 5 Custom is almost the same as the Stage 5 Standard except some details.

But, you will have to insert your own command to be sure that everything is ok, because at the tar command you will have to adapt your command.

Exemple :

Code: Select all

#gentoo docache 
# mke2fs -j /dev/sda9 dans mon cas.
#umount /mnt/cdrom 
#mount /dev/cdrom /mnt/cdrom
#mount -t ext3 /dev/sda9 /mnt/gentoo dans mon cas
# cd /mnt/cdrom
# ls -la 
# tar xjvpf /mnt/cdrom/stage5.tar.bz2 -C /mnt/gentoo/
# mkdir {dev,proc,tmp,sys} 
# chmod 1777 tmp 
# cd dev 
# MAKEDEV generic 
# mount -t proc none /mnt/gentoo/proc 
# mount -o bind /dev /mnt/gentoo/dev
# chroot /mnt/gentoo /bin/bash
# env-update
# source /etc/profile
# emerge --sync
# exit
# cd /
# umount /mnt/cdrom
# umount /mnt/gentoo/dev
# umount /mnt/gentoo/proc
# umount /mnt/gentoo
# reboot

[d2_racing]

Any comments ???

[Conan]

Stage4 works just fine... the only difference is you are bindmounting to get the static devnodes, whereas they are excluded in a stage4 because most of /dev is dynamic (and therefore a waste of time to add.) You could achieve the same results by copyng /dev/null and /dev/console from the livecd, or mknoding them. The lack of these two devices is what caused you problems

[d2_racing]

So, why the Stage 4 doesn't copy these 2 files ???

[Conan]

because the wiki hasn't been updated... you could do -X stage4.excl --include /dev/null --include /dev/console I think, and that'd work just fine.

[likewhoa]

Stage4 works just fine... the only difference is you are bindmounting to get the static devnodes, whereas they are excluded in a stage4 because most of /dev is dynamic (and therefore a waste of time to add.) You could achieve the same results by copyng /dev/null and /dev/console from the livecd, or mknoding them. The lack of these two devices is what caused you problems

you don't even have to copy /dev/null or /dev/console from livecd, what you do is mount your partition as normal but skip the 'mount -o bind /dev /mnt/gentoo/dev' part, then you just
cd into /dev and type 'MAKEDEV generic' and it will generate the require nodes for you

[hirakendu]

frankly, i dont understand why so much effort. i do understand that the intricacy is only because it is being done from inside the working gentoo? what i do is boot using a live cd and simply do tarring along with gzip/bzip2/lzma on the root partition. and a simple extraction on a freshly formatted drive is enough for restoring.

so, for creating image :
$ cd /gentoo/ ( /gentoo/ being the mounted root partition containing bin, boot, usr, etc after booting with a livecd.)
$ tar -cvps --atime-preserve * | lzma e -si -so > /data/gentoo-img.tar.lzma (for a lzma archive)
(or)
$ tar -cvzps --atime-preserve -f /data/gentoo-img.tar.gz * (for gzip archive)

and for restoring from image :
$ mkreiserfs /dev/sdaX
$ mount /dev/sdaX /gentoo
$ cat /data/gentoo-img.tar.lzma | lzma d -si -so | tar -xps --atime-preserve -C /gentoo/ ( from lzma archive)
(or)
$ tar -xvzps --atime-preserve -f /data/gentoo-img.tar.gz -C /gentoo/ (from gzip archive)

and after that, the usual (re)installation of grub and editing some configs (fstab, grub.conf, net , hostname, xorg.conf) to one's liking, just as is done in case of stage3.

btw, i created an archive of a system buit with cflags "o2, prescott" for cpu's with sse3 +. and the image works fine on all machines, inlcuding athlon 64x2's. (yeah all those performance loss fundaes dont work - its 'the' same - atleast with tests like openssl and superpi. ~infinite uptimes, no crashes, stable , fast and responsive.)

my system is about 5.0gb and it takes 15 minutes with gzip and 1hour,30mins with lzma to compress. while decompressing, gzip takes 15 minutes while lzma takes 7 minutes (flat!). btw, gzip compresses it to about 1.6 gb while lzma does it to about 1.1 gb. (this is on a athlon 64x2 3800+ with sata drive @ 67 MB/s (hdparm -tT) and a 16x dvd drive.)

hirakendu.

[likewhoa]

there is no need to perform the tar up from a livecd.

[hirakendu]

yes, i understand that its not necessary, but if its as simple as that from a livecd (and we dont mind rebooting the machine for a 'small' downtime of about 15 minutes for gzip or 2 hrs for lzma) rather than a painstaking script from the running system, i go for the livecd one. anyway, just a poor man's solution, never mind.

[steveL]

So does that work is simply cd / and run those cmds likewhoa? What about all the /dev stuff?

I just want a simple way to backup my existing gentoo before I transfer it to a new HDD. I'm guessing the Stage4 is the way to do it?

[d2_racing]

Yes, use theses commands and you will have a Stage 5 archive and you will be able to transfert to your new harddrive with less problems then a Stage 4.

[Conan]

Your "less problems" is just talking out of your hind end.

[steveL]

Let's not get rude here, yeah? Simply saying `that's not true' and (hopefully) giving reasons is better imo.

I still don't see how: tar -cvzps --atime-preserve -f /data/gentoo-img.tar.gz *
..avoids /dev and /proc for example.

Thanks.

[Earthwings]

Your "less problems" is just talking out of your hind end.

Conan, second warning from me now. Please improve your language or it will end in a ban sooner or later.

[steveL]

Please improve your language or it will end in a ban sooner or later.

Now why can't you guys moderate the dev list and bugzilla? That would make things a lot nicer

[onelife151]

Thanks for posting the link for the stage 4 backup and your detailed explanation of your exotic back up strategy. I want to thank you to all of the gentoo community. Even though we have out bouts here and there, we are all here to learn and share information is this slighly less microsoft owned world.

[d2_racing]

Thanks for posting the link for the stage 4 backup and your detailed explanation of your exotic back up strategy. I want to thank you to all of the gentoo community. Even though we have out bouts here and there, we are all here to learn and share information is this slighly less microsoft owned world.

No problem

[alexdu]

Now you will have a Gentoo back on track.

Hi.

I don't understand
if someone needs to make/restore a *nix snapshot why do not use simple low level coping:

Code: Select all

# cp /dev/hda1 ...
; or even
# cp /dev/hda ...
; later mount LiveCD and do back
# cp ... /dev/hda1
; or
# cp ... /dev/hda

you can pipe it any way you want - tar, scp, bzip...

Also, playing with tools like resize_reiserfs you can shrink or expand your partition.
You can also change the physical order of restored partitions - all you need is to fix /etc/fstab after restore. (and may be somewhere else if you use raw hard disk access somewhere - it's unusual case)
If you want to change something in the saved binary image, you can mount it as a loop device: mount -o loop file... mountpoint.... , while it is a single partition and not compressed or splitted.

[d2_racing]

Now you will have a Gentoo back on track.

Hi.

I don't understand
if someone needs to make/restore a *nix snapshot why do not use simple low level coping:

Code: Select all

# cp /dev/hda1 ...
; or even
# cp /dev/hda ...
; later mount LiveCD and do back
# cp ... /dev/hda1
; or
# cp ... /dev/hda

you can pipe it any way you want - tar, scp, bzip...

Also, playing with tools like resize_reiserfs you can shrink or expand your partition.
You can also change the physical order of restored partitions - all you need is to fix /etc/fstab after restore. (and may be somewhere else if you use raw hard disk access somewhere - it's unusual case)
If you want to change something in the saved binary image, you can mount it as a loop device: mount -o loop file... mountpoint.... , while it is a single partition and not compressed or splitted.

Yes, it's only an another methode

It's what I love the most about Linux and Unix, you can choose your own path and you will arrive at destination every time

[steveL]

f someone needs to make/restore a *nix snapshot why do not use simple low level coping:

Code: Select all

# cp /dev/hda1 ...
; or even
# cp /dev/hda ...
; later mount LiveCD and do back
# cp ... /dev/hda1
; or
# cp ... /dev/hda

Are you saying I can backup the whole of my system (on /dev/hdb) with:
# cp /dev/hda ~/backup/today/

you can pipe it any way you want - tar, scp, bzip...

Also, playing with tools like resize_reiserfs you can shrink or expand your partition.
You can also change the physical order of restored partitions - all you need is to fix /etc/fstab after restore. (and may be somewhere else if you use raw hard disk access somewhere - it's unusual case)
If you want to change something in the saved binary image, you can mount it as a loop device: mount -o loop file... mountpoint.... , while it is a single partition and not compressed or splitted.

Can you give examples of these?

[d2_racing]

f someone needs to make/restore a *nix snapshot why do not use simple low level coping:

Code: Select all

# cp /dev/hda1 ...
; or even
# cp /dev/hda ...
; later mount LiveCD and do back
# cp ... /dev/hda1
; or
# cp ... /dev/hda

Are you saying I can backup the whole of my system (on /dev/hdb) with:
# cp /dev/hda ~/backup/today/

you can pipe it any way you want - tar, scp, bzip...

Also, playing with tools like resize_reiserfs you can shrink or expand your partition.
You can also change the physical order of restored partitions - all you need is to fix /etc/fstab after restore. (and may be somewhere else if you use raw hard disk access somewhere - it's unusual case)
If you want to change something in the saved binary image, you can mount it as a loop device: mount -o loop file... mountpoint.... , while it is a single partition and not compressed or splitted.

Can you give examples of these?

Yes, send an example with the tar or something else plz

[likewhoa]

So does that work is simply cd / and run those cmds likewhoa? What about all the /dev stuff?

I just want a simple way to backup my existing gentoo before I transfer it to a new HDD. I'm guessing the Stage4 is the way to do it?

my method to backup/clone a system for transfering to another system of the same -march is simple..

Code: Select all

tar cvjp --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/stage4.tar.bz2 --file stage4.tar.bz2 /

then simply scp that tarball to your new system,.. untar it, then simply cd into the /mnt/gentoo/dev directory and run

Code: Select all

MAKEDEV generic

this will create the necessary nodes. after mount proc & dev then finally chroot to it. that's it; you should know the rest.

there is really no need to tar/copy /dev,/sys,/proc and you should also delete files from /tmp,/usr/portage/distfiles,/var/tmp/portage, & /var/log/portage to keep the tarball small.
Hope this helps. This method is what i use to setup systems of the same -march and it works well.

[d2_racing]

there is really no need to tar/copy /dev,/sys,/proc and you should also delete files from /tmp,/usr/portage/distfiles,/var/tmp/portage, & /var/log/portage to keep the tarball small.
Hope this helps. This method is what i use to setup systems of the same -march and it works well.

I don't have any problem to exclude what you have said, but the /dev is necessary, because if you have a framebuffer it will crash because of the messing node in /dev.

[likewhoa]

there is really no need to tar/copy /dev,/sys,/proc and you should also delete files from /tmp,/usr/portage/distfiles,/var/tmp/portage, & /var/log/portage to keep the tarball small.
Hope this helps. This method is what i use to setup systems of the same -march and it works well.

I don't have any problem to exclude what you have said, but the /dev is necessary, because if you have a framebuffer it will crash because of the messing node in /dev.

While it's ok to copy /sys & /proc you still don't have to and it's only optional as these folders will be regenerated on the new system. I stick to excluding them from my backups.

[nth10sd]

pardon my ignorance, i've been away awhile but what is stage 5?

stage 1 = everything incl. bootstrap
stage 2 = no bootstrap
stage 3 = precompiled binaries
stage 4 = ?
stage 5 = ??

Is it something to do with liveCDs?