Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: Easily make a full system backup (stage4)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 8, 9, 10 ... 16, 17, 18  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Pseud
Apprentice
Apprentice


Joined: 19 Mar 2004
Posts: 273
Location: Bangalore, India

PostPosted: Wed Jun 22, 2005 3:34 am    Post subject: Reply with quote

Coolio! Thank you! Will check out asap.
_________________
eschew obfuscation
Back to top
View user's profile Send private message
Pseud
Apprentice
Apprentice


Joined: 19 Mar 2004
Posts: 273
Location: Bangalore, India

PostPosted: Wed Jun 22, 2005 4:18 am    Post subject: Reply with quote

BlinkEye,

I haven't tried the new script yet, but I just glanced through the code and have a small concern. As I understand it, the script accepts the -s option but still resorts back to using the split command. Wouldn't this mean that there's still double the hard-disk space usage? (because it's an intrinsic problem with the split command itself: when it splits a file into pieces, there's no way to avoid keeping the to-be-split file.) I gave it some thought and couldn't even think of a way of writing a split program (in C, for instance) that will avoid this doubling of storage.

What I suggested earlier was something like this:
The script should generate split tarballs on the fly: not by creating a 2G tarball and then splitting it into three 600m pieces and one 200m piece (for example), but by creating a second tarball as soon as there is more data than will fit in a 600m tarball (and so on for more than two tarballs, etc.)
This won't be trivial though, because there's no way of guessing how much data will fit into a 600m tarball because the amount of compression depends on the data itself! Also, this might cost us in terms of lower compression overall, but that's a trade-off if one doesn't want a mammoth tarball.

So, is there an intelligent way out? Sounds like an interesting problem, but I can't think of a solution right away.
_________________
eschew obfuscation
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Wed Jun 22, 2005 8:02 am    Post subject: Reply with quote

it works without first creating a big tarball. i'll explain the accordant command:
Code:
tar_command="$find_command | $tar $zip $tarOptions --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"

tar doesn't write to a file (i removed the --file option from $tarOptions), so it writes to stdout. now, split takes input from stdin because of the "-" after the $split_options. this results in chunks the way you set in $split_options.

[EDIT] fixed language [/EDIT]
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick


Last edited by BlinkEye on Fri Jul 01, 2005 5:24 am; edited 1 time in total
Back to top
View user's profile Send private message
Pseud
Apprentice
Apprentice


Joined: 19 Mar 2004
Posts: 273
Location: Bangalore, India

PostPosted: Wed Jun 22, 2005 9:02 am    Post subject: Reply with quote

Heh,
Rather dumb of me, not thinking about piping the commands! :oops: (and not looking carefully at the script ... I got misled by the commented split statements you've still left at the end, thinking that one would have to uncomment those as earlier.)
This just set me wondering though, could there be a memory choke when (RAM + swap) < archive size ?
_________________
eschew obfuscation
Back to top
View user's profile Send private message
profy
Tux's lil' helper
Tux's lil' helper


Joined: 08 Jun 2005
Posts: 96
Location: Sophia Antipolis

PostPosted: Thu Jun 23, 2005 8:33 am    Post subject: Reply with quote

Helllo,

with version=v3.4

Code:

PROXYGENTOO ~ # ./mkstage4.sh
: No such file or directory
PROXYGENTOO ~ # sh mkstage4.sh
: command not found:
: command not found0:
: command not found3:
: command not found6:
'kstage4.sh: line 18: syntax error near unexpected token `do
'kstage4.sh: line 18: `for command in ${command_list[@]}; do
PROXYGENTOO ~ # find /bin/tar
/bin/tar


Any suggestion to help me ?

Sorry if it's an easy error i'm a beginner.
_________________
Plus je connais les hommes, plus j'aime mon pc.
Back to top
View user's profile Send private message
NightTwix
n00b
n00b


Joined: 06 Jan 2005
Posts: 58

PostPosted: Thu Jun 23, 2005 10:32 am    Post subject: Reply with quote

profy wrote:
Helllo,

with version=v3.4

Code:

PROXYGENTOO ~ # ./mkstage4.sh
: No such file or directory
PROXYGENTOO ~ # sh mkstage4.sh
: command not found:
: command not found0:
: command not found3:
: command not found6:
'kstage4.sh: line 18: syntax error near unexpected token `do
'kstage4.sh: line 18: `for command in ${command_list[@]}; do
PROXYGENTOO ~ # find /bin/tar
/bin/tar


Any suggestion to help me ?

Sorry if it's an easy error i'm a beginner.


looks like your script is broken.
did you copy&paste it oder did u download it?

try to download it directly here: ftp://blinkeye.ch/gentoo/mkstage4.sh
Back to top
View user's profile Send private message
profy
Tux's lil' helper
Tux's lil' helper


Joined: 08 Jun 2005
Posts: 96
Location: Sophia Antipolis

PostPosted: Thu Jun 23, 2005 10:39 am    Post subject: Reply with quote

Yes it works :)
_________________
Plus je connais les hommes, plus j'aime mon pc.
Back to top
View user's profile Send private message
torchZ06
Apprentice
Apprentice


Joined: 01 Nov 2003
Posts: 175
Location: the front range

PostPosted: Thu Jun 23, 2005 3:02 pm    Post subject: can't login as any user but root Reply with quote

got a little problem. after using the script to create the tarball and following the instructions to extract it to a new partition, everything worked fine, except on my new partition i can only login as root! when i try another user, it says
Code:

No directory /home/username!


but the directory DOES exist and has the correct permissions, so where did i go wrong?

i even tried adding another user to the new partitions copy using the useradd method as described in the handbook, but that produced the same results as above.
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Thu Jun 23, 2005 8:01 pm    Post subject: Reply with quote

does your user exist in
Code:
cat /etc/passwd

and does the group "users" exist in
Code:
cat /etc/group


are you able to login (just can't change to your home) or does the login itself fail?
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Jedidiah
n00b
n00b


Joined: 16 Nov 2004
Posts: 13

PostPosted: Fri Jun 24, 2005 3:18 pm    Post subject: Reply with quote

So its been several days, but i finally got it to work on a Mac - almost. i can now sucessfully back up and untar the stage4 without getting any errors (this is on a running computer, where i just untar the stage4 into a new directory, with no intention of booting to it). however, when i try to copy it to a clean hard drive, with the intentions of booting from it, i get an error. here is what i am doing.
1 - i run the stage4 script on a mac that is up and running, have it save the file to an external USB harddisk
2 - i put the MacOSX cd into the computer that i want to copy to, boot to it, erase the harddisk, and then open up the command prompt - all of this is supposedly runing from the mac install cd
3 - when i try to untar the file, it gives me the following error
Code:
tar (child): gzip: Cannot exec: No Such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error exit delayed from previous errors

this error is returned to me wether i cd into the external hardrive and tell it to untar and copy it to the local hd, copy the file to the local hd via cp command. i have tried a variety of different methods for untarring this archive onto the local hd and they all return some varriation on this message. i have succesfully untarred the archive to a directory on the external harddrive (and a local harddisk) while i was booted into macosx.

does anyone have any ideas?

the only other idea i had was to boot from a gentoo live cd for the sake of untarring the archive, but i wasn't sure if there was a livecd that supported writing to a macosx formated partition that is journaled. any suggestions for which live cd i should use? i am on the new imac G5s [url]http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore.woa /73002/wo/M36xDLuJ1Pif2YYZuP32rh4J0ii/1.0.11.1.0.6.9.1.2.21.3.1.1.0?44,20[/url], and becuase this is for the college i am working for, installing gentoo is not an option. so unless someone has a better idea, i really just need to know which live cd (that has tar on it) supports the imac G5s that we have

thanks a ton

jedidiah
Back to top
View user's profile Send private message
dundas
Guru
Guru


Joined: 16 Dec 2004
Posts: 317
Location: China, Earth

PostPosted: Sat Jun 25, 2005 6:31 pm    Post subject: Reply with quote

[Solved] & [Repeated Question, sorry, feel free to delete this post]

Quote:
Hi

Impressive script, however when I tried to execute

Code:
./mkstage4.sh


I got error:

Quote:
/var/log/portage not found! Check your $default_include_files



[solution] just remove /var/log/portage from the $default_include_files

thx man.


Last edited by dundas on Tue Jun 28, 2005 4:15 am; edited 2 times in total
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Sat Jun 25, 2005 10:38 pm    Post subject: Reply with quote

Jedidiah wrote:

does anyone have any ideas?

the only other idea i had was to boot from a gentoo live cd for the sake of untarring the archive, but i wasn't sure if there was a livecd that supported writing to a macosx formated partition that is journaled. any suggestions for which live cd i should use? i am on the new imac G5s [url]http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore.woa
/73002/wo/M36xDLuJ1Pif2YYZuP32rh4J0ii/1.0.11.1.0.6.9.1.2.21.3.1.1.0?44,20[/url], and becuase this is for the college i am working for, installing gentoo is not an option. so unless someone has a better idea, i really just need to know which live cd (that has tar on it) supports the imac G5s that we have

thanks a ton

jedidiah

it's obvious that the problem is the tar version or the tar capabilities of the mac boot disk. either you gunzip/bunzip your stage4 (if you can handle the size of hte uncompressed tar file) or you try out a ppc livecd (i assume a 64bit version): ftp://mirror.switch.ch/mirror/gentoo/releases/ppc64/2005.0/installcd. i don't own a mac.

what fs are you using? HFS+ or UFS? do a google on the writing capabilities or check the forums. this is what i've found: http://johnleach.co.uk/documents/powerpc/
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
alsuren
n00b
n00b


Joined: 26 Feb 2005
Posts: 19
Location: London

PostPosted: Wed Jun 29, 2005 2:49 pm    Post subject: Reply with quote

Okay, I've already used this script to do a clone of my AMD box onto the HDD of an old Pentium3 that I decapitated. I did it by plugging the old HDD into the new comp, and it got my old box bootable again, and after some tapping away, I managed to test SSH on it. This was quite a while back and I got an error when I fired up KDE about inter-process communication, which I've just established is because of my n00bishness in forgetting to create my ~ directory before testing. By the time I re-tested it, I had both upgraded to the latest KDEbase and re-created ~ so I foolishly assumed that it was the update that fixed it. OOOOOOOPS

Just a hint that I picked up elsewhere: The rsync servers are quite slow if you're trying to grab your whole portage tree from them (which is why the Gentoo universal install CD contains a portage snapshot to get you started). It might be nice to get the script to create a snapshot of portage as well (not including distfiles) that you can put on the DVD/multiple CDs when you're burning it/them so it's there if you want it. My backup is just over 1GB so I would need to put it on 2 CDs, with room to spare. A portage snapshot would be quite nice for when I go around being evangelical.

I saw a couple of little links on google and stuff that *might* be useful for people who want to make a bit more of an incremental backup every few days than a stage4 tarball (I know it's not strictly on topic, but they might still be useful)

http://apfelboymchen.homeunix.net/gnu/scripte/backup/ (written by Irom, script using "tar" for incremental backups. I think someone suggested incremental tarballs on this thread, so this could be good for you)

http://dar.linux.free.fr/tutorial.php3 (found on google, tutorial on using "dar". Seems quite old because it refers to zip drives, but might also be useful)

What would be nice is if the script automatically makes a script that you can use for restoring the backup (I'm going to have some time over the summer, so I've posted those links as much for myself as anything else. I'll try hacking together some scripts over the summer and if anything interesting becomes of them, I'll re-post)
_________________
<alsuren 22:07:55> I thought it was 42
<ladymauv 22:08:19> Maybe it's really 69


Last edited by alsuren on Wed Jun 29, 2005 9:44 pm; edited 1 time in total
Back to top
View user's profile Send private message
alsuren
n00b
n00b


Joined: 26 Feb 2005
Posts: 19
Location: London

PostPosted: Wed Jun 29, 2005 6:24 pm    Post subject: Reply with quote

While I'm waiting for the tarball to be made, I just had an idea. Is there any way to create a pipe (or entire plumbing system, in this case) so that the data is passed through md5sum at the same time as it's being written to disk? That way, you don't have to read the whole thing off the disk again to md5sum it if you want before you put it onto a CD

EDIT:
oh..... ignore me! it says it's "checking integrity" which suggests some kind of checksumming. It's just that I did a quick find in Kdevelop for "sum" and it came back with nothing...... So if my tarball is corrupt after however long sitting on a CD, will tar simply fail? :O I never knew that..... or does it just continue and tell me which file is corrupt? *hopes it's the second one*
_________________
<alsuren 22:07:55> I thought it was 42
<ladymauv 22:08:19> Maybe it's really 69
Back to top
View user's profile Send private message
Pseud
Apprentice
Apprentice


Joined: 19 Mar 2004
Posts: 273
Location: Bangalore, India

PostPosted: Thu Jun 30, 2005 3:50 am    Post subject: Reply with quote

BlinkEye,

Just two more (non-urgent) things which you might want to fix up. I did a backup yesterday using the split option and things went fine, except for these:

1. mkstage4 -v -s does't work (can't both split and be verbose ... the script only takes the first option and ignores the next) ... not a big deal I guess, but would be nice (since -v and -s are unrelated options, they shouldn't be mutually exclusive).
2. the integrity check "fails" when the script is called with the split option ... so the script should either skip the integrity check when called with the split option, or should do something different to check integrity.
_________________
eschew obfuscation
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Thu Jun 30, 2005 10:20 pm    Post subject: Reply with quote

thanks for pointing that out.

new version out: v3.5
* Fixed issue with command line parameters and mutual exclusion. --split and --verbose or -s and -v may now be combined in any order.
* Fixed integrity check for splitted files - it's now possible.
* Removed /var/cache/edb as this is only cache and portage does not need it to be backed up.
* Removed /var/log/portage per a default as new users get confused about adjusting the script.
* Added used commands to be checked at the beginning.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
ktech
Guru
Guru


Joined: 12 Apr 2003
Posts: 340

PostPosted: Fri Jul 01, 2005 2:17 pm    Post subject: Reply with quote

BlinkEye, thanks a lot for your efforts. This script is highly appreciated here :)

One suggestion: Perhaps you can do an option for the script to print on the screen the size of the files being backed-up. Something like du {list of all directories to backup}. That way, you can guess if you're backing up all the directories you want... Not too difficult to implement and very useful to me :)
Back to top
View user's profile Send private message
JeffBlair
Apprentice
Apprentice


Joined: 23 May 2003
Posts: 173
Location: USA, Lone star state

PostPosted: Fri Jul 01, 2005 2:38 pm    Post subject: Reply with quote

slydini I am running into the same problem. Could you post what you typed to get past that? I thought it had worked as well, but it looks like some of the /var dir did not get extracted out. I couldn't start webmin/mysql/etc... Otherwise nice little script.

This is about it stoping when it hits .reiserfs_priv directory.

would I type in something like this?
Code:

tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/  --exclude=.reiserfs_priv


Thanks again.
Back to top
View user's profile Send private message
JeffBlair
Apprentice
Apprentice


Joined: 23 May 2003
Posts: 173
Location: USA, Lone star state

PostPosted: Fri Jul 01, 2005 10:23 pm    Post subject: Reply with quote

Ok, I answered my own question. That command will do it, it got all the way thru without any errors. Now to reboot. :)
Back to top
View user's profile Send private message
ktech
Guru
Guru


Joined: 12 Apr 2003
Posts: 340

PostPosted: Sat Jul 02, 2005 1:22 am    Post subject: Reply with quote

I have a problem:

Code:
Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)


Then I look at the generated file:

Code:

evanescence stage4 # ls -ll -h
total 2,0G
-rw-r--r--  1 root root 2,0G jul  2 01:38 evanescence-stage4-2005.07.02-custom.tar.gz


so, what's the deal? The filesystem where the backup is, is reiser3.
Back to top
View user's profile Send private message
DrWoland
l33t
l33t


Joined: 13 Nov 2004
Posts: 603

PostPosted: Sat Jul 02, 2005 4:28 am    Post subject: Reply with quote

ktech wrote:
I have a problem:

Code:
Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)


Then I look at the generated file:

Code:

evanescence stage4 # ls -ll -h
total 2,0G
-rw-r--r--  1 root root 2,0G jul  2 01:38 evanescence-stage4-2005.07.02-custom.tar.gz


so, what's the deal? The filesystem where the backup is, is reiser3.


Maybe that's how much was backed up?
_________________
I'm not a Guru, I just ask a lot of questions.
Back to top
View user's profile Send private message
Pseud
Apprentice
Apprentice


Joined: 19 Mar 2004
Posts: 273
Location: Bangalore, India

PostPosted: Sat Jul 02, 2005 4:45 am    Post subject: Reply with quote

ktech wrote:
I have a problem:

Code:
Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)


Then I look at the generated file:

Code:

evanescence stage4 # ls -ll -h
total 2,0G
-rw-r--r--  1 root root 2,0G jul  2 01:38 evanescence-stage4-2005.07.02-custom.tar.gz


so, what's the deal? The filesystem where the backup is, is reiser3.


4.2G raw data; 2.0 G after compression
_________________
eschew obfuscation
Back to top
View user's profile Send private message
ktech
Guru
Guru


Joined: 12 Apr 2003
Posts: 340

PostPosted: Sat Jul 02, 2005 9:03 am    Post subject: Reply with quote

So the words "Total bytes written:" should be "Total bytes readed"?

How strange...
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Sat Jul 02, 2005 10:07 am    Post subject: Reply with quote

rather do a
Code:
du -sch filename

than
Code:
ls -l filename

du means diskusage and shows the exact space occupied by the filename on your filesystem.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
ktech
Guru
Guru


Joined: 12 Apr 2003
Posts: 340

PostPosted: Sat Jul 02, 2005 10:14 am    Post subject: Reply with quote

it's still 2.0 GB.

BlinkEye, there is any logic in the script telling me that it has writen 4 MB and I having only a 2 MB file?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 8, 9, 10 ... 16, 17, 18  Next
Page 9 of 18

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum