HOWTO: Easily make a full system backup (stage4)

[Pseud]

Coolio! Thank you! Will check out asap.

[Pseud]

BlinkEye,

I haven't tried the new script yet, but I just glanced through the code and have a small concern. As I understand it, the script accepts the -s option but still resorts back to using the split command. Wouldn't this mean that there's still double the hard-disk space usage? (because it's an intrinsic problem with the split command itself: when it splits a file into pieces, there's no way to avoid keeping the to-be-split file.) I gave it some thought and couldn't even think of a way of writing a split program (in C, for instance) that will avoid this doubling of storage.

What I suggested earlier was something like this:
The script should generate split tarballs on the fly: not by creating a 2G tarball and then splitting it into three 600m pieces and one 200m piece (for example), but by creating a second tarball as soon as there is more data than will fit in a 600m tarball (and so on for more than two tarballs, etc.)
This won't be trivial though, because there's no way of guessing how much data will fit into a 600m tarball because the amount of compression depends on the data itself! Also, this might cost us in terms of lower compression overall, but that's a trade-off if one doesn't want a mammoth tarball.

So, is there an intelligent way out? Sounds like an interesting problem, but I can't think of a solution right away.

[BlinkEye]

it works without first creating a big tarball. i'll explain the accordant command:

Code: Select all

tar_command="$find_command | $tar $zip $tarOptions --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"

tar doesn't write to a file (i removed the --file option from $tarOptions), so it writes to stdout. now, split takes input from stdin because of the "-" after the $split_options. this results in chunks the way you set in $split_options.

[EDIT] fixed language [/EDIT]

[Pseud]

Heh,
Rather dumb of me, not thinking about piping the commands! (and not looking carefully at the script ... I got misled by the commented split statements you've still left at the end, thinking that one would have to uncomment those as earlier.)
This just set me wondering though, could there be a memory choke when (RAM + swap) < archive size ?

[profy]

Helllo,

with version=v3.4

Code: Select all

PROXYGENTOO ~ # ./mkstage4.sh 
: No such file or directory
PROXYGENTOO ~ # sh mkstage4.sh 
: command not found: 
: command not found0: 
: command not found3: 
: command not found6: 
'kstage4.sh: line 18: syntax error near unexpected token `do
'kstage4.sh: line 18: `for command in ${command_list[@]}; do
PROXYGENTOO ~ # find /bin/tar 
/bin/tar

Any suggestion to help me ?

Sorry if it's an easy error i'm a beginner.

[NightTwix]

Helllo,

with version=v3.4

Code: Select all

PROXYGENTOO ~ # ./mkstage4.sh 
: No such file or directory
PROXYGENTOO ~ # sh mkstage4.sh 
: command not found: 
: command not found0: 
: command not found3: 
: command not found6: 
'kstage4.sh: line 18: syntax error near unexpected token `do
'kstage4.sh: line 18: `for command in ${command_list[@]}; do
PROXYGENTOO ~ # find /bin/tar 
/bin/tar

Any suggestion to help me ?

Sorry if it's an easy error i'm a beginner.

looks like your script is broken.
did you copy&paste it oder did u download it?

try to download it directly here: ftp://blinkeye.ch/gentoo/mkstage4.sh

[profy]

Yes it works

[torchZ06]

got a little problem. after using the script to create the tarball and following the instructions to extract it to a new partition, everything worked fine, except on my new partition i can only login as root! when i try another user, it says

Code: Select all

No directory /home/username!

but the directory DOES exist and has the correct permissions, so where did i go wrong?

i even tried adding another user to the new partitions copy using the useradd method as described in the handbook, but that produced the same results as above.

[BlinkEye]

does your user exist in

Code: Select all

cat /etc/passwd

and does the group "users" exist in

Code: Select all

cat /etc/group

are you able to login (just can't change to your home) or does the login itself fail?

[Jedidiah]

So its been several days, but i finally got it to work on a Mac - almost. i can now sucessfully back up and untar the stage4 without getting any errors (this is on a running computer, where i just untar the stage4 into a new directory, with no intention of booting to it). however, when i try to copy it to a clean hard drive, with the intentions of booting from it, i get an error. here is what i am doing.
1 - i run the stage4 script on a mac that is up and running, have it save the file to an external USB harddisk
2 - i put the MacOSX cd into the computer that i want to copy to, boot to it, erase the harddisk, and then open up the command prompt - all of this is supposedly runing from the mac install cd
3 - when i try to untar the file, it gives me the following error

Code: Select all

tar (child): gzip: Cannot exec: No Such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error exit delayed from previous errors

this error is returned to me wether i cd into the external hardrive and tell it to untar and copy it to the local hd, copy the file to the local hd via cp command. i have tried a variety of different methods for untarring this archive onto the local hd and they all return some varriation on this message. i have succesfully untarred the archive to a directory on the external harddrive (and a local harddisk) while i was booted into macosx.

does anyone have any ideas?

the only other idea i had was to boot from a gentoo live cd for the sake of untarring the archive, but i wasn't sure if there was a livecd that supported writing to a macosx formated partition that is journaled. any suggestions for which live cd i should use? i am on the new imac G5s http://store.apple.com/1-800-MY-APPLE/W ... .1.0?44,20, and becuase this is for the college i am working for, installing gentoo is not an option. so unless someone has a better idea, i really just need to know which live cd (that has tar on it) supports the imac G5s that we have

thanks a ton

jedidiah

[dundas]

[Solved] & [Repeated Question, sorry, feel free to delete this post]

Hi

Impressive script, however when I tried to execute

Code: Select all

./mkstage4.sh

I got error:

/var/log/portage not found! Check your $default_include_files

[solution] just remove /var/log/portage from the $default_include_files

thx man.

[BlinkEye]

does anyone have any ideas?

the only other idea i had was to boot from a gentoo live cd for the sake of untarring the archive, but i wasn't sure if there was a livecd that supported writing to a macosx formated partition that is journaled. any suggestions for which live cd i should use? i am on the new imac G5s http://store.apple.com/1-800-MY-APPLE/W ... .1.0?44,20, and becuase this is for the college i am working for, installing gentoo is not an option. so unless someone has a better idea, i really just need to know which live cd (that has tar on it) supports the imac G5s that we have

thanks a ton

jedidiah

it's obvious that the problem is the tar version or the tar capabilities of the mac boot disk. either you gunzip/bunzip your stage4 (if you can handle the size of hte uncompressed tar file) or you try out a ppc livecd (i assume a 64bit version): ftp://mirror.switch.ch/mirror/gentoo/re ... /installcd. i don't own a mac.

what fs are you using? HFS+ or UFS? do a google on the writing capabilities or check the forums. this is what i've found: http://johnleach.co.uk/documents/powerpc/

[alsuren]

Okay, I've already used this script to do a clone of my AMD box onto the HDD of an old Pentium3 that I decapitated. I did it by plugging the old HDD into the new comp, and it got my old box bootable again, and after some tapping away, I managed to test SSH on it. This was quite a while back and I got an error when I fired up KDE about inter-process communication, which I've just established is because of my n00bishness in forgetting to create my ~ directory before testing. By the time I re-tested it, I had both upgraded to the latest KDEbase and re-created ~ so I foolishly assumed that it was the update that fixed it. OOOOOOOPS

Just a hint that I picked up elsewhere: The rsync servers are quite slow if you're trying to grab your whole portage tree from them (which is why the Gentoo universal install CD contains a portage snapshot to get you started). It might be nice to get the script to create a snapshot of portage as well (not including distfiles) that you can put on the DVD/multiple CDs when you're burning it/them so it's there if you want it. My backup is just over 1GB so I would need to put it on 2 CDs, with room to spare. A portage snapshot would be quite nice for when I go around being evangelical.

I saw a couple of little links on google and stuff that *might* be useful for people who want to make a bit more of an incremental backup every few days than a stage4 tarball (I know it's not strictly on topic, but they might still be useful)

http://apfelboymchen.homeunix.net/gnu/scripte/backup/ (written by Irom, script using "tar" for incremental backups. I think someone suggested incremental tarballs on this thread, so this could be good for you)

http://dar.linux.free.fr/tutorial.php3 (found on google, tutorial on using "dar". Seems quite old because it refers to zip drives, but might also be useful)

What would be nice is if the script automatically makes a script that you can use for restoring the backup (I'm going to have some time over the summer, so I've posted those links as much for myself as anything else. I'll try hacking together some scripts over the summer and if anything interesting becomes of them, I'll re-post)

[alsuren]

While I'm waiting for the tarball to be made, I just had an idea. Is there any way to create a pipe (or entire plumbing system, in this case) so that the data is passed through md5sum at the same time as it's being written to disk? That way, you don't have to read the whole thing off the disk again to md5sum it if you want before you put it onto a CD

EDIT:
oh..... ignore me! it says it's "checking integrity" which suggests some kind of checksumming. It's just that I did a quick find in Kdevelop for "sum" and it came back with nothing...... So if my tarball is corrupt after however long sitting on a CD, will tar simply fail? :O I never knew that..... or does it just continue and tell me which file is corrupt? *hopes it's the second one*

[Pseud]

BlinkEye,

Just two more (non-urgent) things which you might want to fix up. I did a backup yesterday using the split option and things went fine, except for these:

1. mkstage4 -v -s does't work (can't both split and be verbose ... the script only takes the first option and ignores the next) ... not a big deal I guess, but would be nice (since -v and -s are unrelated options, they shouldn't be mutually exclusive).
2. the integrity check "fails" when the script is called with the split option ... so the script should either skip the integrity check when called with the split option, or should do something different to check integrity.

[BlinkEye]

thanks for pointing that out.

new version out: v3.5
* Fixed issue with command line parameters and mutual exclusion. --split and --verbose or -s and -v may now be combined in any order.
* Fixed integrity check for splitted files - it's now possible.
* Removed /var/cache/edb as this is only cache and portage does not need it to be backed up.
* Removed /var/log/portage per a default as new users get confused about adjusting the script.
* Added used commands to be checked at the beginning.

[ktech]

BlinkEye, thanks a lot for your efforts. This script is highly appreciated here

One suggestion: Perhaps you can do an option for the script to print on the screen the size of the files being backed-up. Something like du {list of all directories to backup}. That way, you can guess if you're backing up all the directories you want... Not too difficult to implement and very useful to me

[JeffBlair]

slydini I am running into the same problem. Could you post what you typed to get past that? I thought it had worked as well, but it looks like some of the /var dir did not get extracted out. I couldn't start webmin/mysql/etc... Otherwise nice little script.

This is about it stoping when it hits .reiserfs_priv directory.

would I type in something like this?

Code: Select all

tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/  --exclude=.reiserfs_priv

Thanks again.

[JeffBlair]

Ok, I answered my own question. That command will do it, it got all the way thru without any errors. Now to reboot.

[ktech]

I have a problem:

Code: Select all

Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)

Then I look at the generated file:

Code: Select all

evanescence stage4 # ls -ll -h
total 2,0G
-rw-r--r--  1 root root 2,0G jul  2 01:38 evanescence-stage4-2005.07.02-custom.tar.gz

so, what's the deal? The filesystem where the backup is, is reiser3.

[DrWoland]

I have a problem:

Code: Select all

Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)

Then I look at the generated file:

Code: Select all

evanescence stage4 # ls -ll -h
total 2,0G
-rw-r--r--  1 root root 2,0G jul  2 01:38 evanescence-stage4-2005.07.02-custom.tar.gz

so, what's the deal? The filesystem where the backup is, is reiser3.

Maybe that's how much was backed up?

[Pseud]

I have a problem:

Code: Select all

Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)

Then I look at the generated file:

Code: Select all

evanescence stage4 # ls -ll -h
total 2,0G
-rw-r--r--  1 root root 2,0G jul  2 01:38 evanescence-stage4-2005.07.02-custom.tar.gz

so, what's the deal? The filesystem where the backup is, is reiser3.

4.2G raw data; 2.0 G after compression

[ktech]

So the words "Total bytes written:" should be "Total bytes readed"?

How strange...

[BlinkEye]

rather do a

Code: Select all

du -sch filename

than

Code: Select all

ls -l filename

du means diskusage and shows the exact space occupied by the filename on your filesystem.

[ktech]

it's still 2.0 GB.

BlinkEye, there is any logic in the script telling me that it has writen 4 MB and I having only a 2 MB file?