Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: Easily make a full system backup (stage4)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5 ... 16, 17, 18  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Mon Apr 18, 2005 7:27 am    Post subject: Reply with quote

slydini wrote:
Only trouble is it doesn't work for me.

thanks for the mini-howto. i'll put that in my initial thread. but how about actually trying a suggestion others gave? you may extract this stage4 wherever you want - if you do that INSIDE an existing directory you'll get
Code:
tar: Removing leading `/' from member names

which means it doesn't use absolute pathes anymore. your directions can't be right as you haven't yet copied over the stage4 to /mnt/gentoo. either way,
Code:
cd /mnt/gentoo
tar xzvpf host-stage4-16.04.2005-custom.tar.gz .

simply doesn't work, it should be
Code:
tar xzvpf host-stage4-16.04.2005-custom.tar.gz

if you have your stage4 in /mnt/gentoo and your current dir is /mnt/gentoo
if you're somewhere else use
Code:
tar xzvpf /path/to/stage4/host-stage4-16.04.2005-custom.tar.gz -C /mnt/gentoo

_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
slydini
Tux's lil' helper
Tux's lil' helper


Joined: 30 Oct 2002
Posts: 129
Location: Virginia Beach, VA

PostPosted: Mon Apr 18, 2005 12:38 pm    Post subject: Reply with quote

BlinkEye, thanks for your help. Just so you know I tried what others have suggested including what you have suggested and I and still get the exact same error as posted previously. In fact I removed the --absolute-paths option temporarily so I could test it that way but it still doesn't work. So at this point I simply do not understand what is going on. If the only thing you would change in my restore steps is cp the tarball in the /mnt/gentoo directory first before extracting then there is something very strange going on with my laptop and your script. I don't know what it is I only know that I cannot extract any files past that stupid .reiserfs_priv: Cannot mkdir: Permission denied.

I even chmod 777 gentoo with no luck. I will try to exclude the .reiserfs_priv directory from the restore and see what happens. Ok, now I have gotten past the error and it looks like everything else is restoring OK. :DI will examine the contents of that .reiserfs_priv directory to see whats going on. Wait a minute, while extracting the second tarball I got the same exact error upon restore. This is really weird. Let me see if it even will come up after a reboot. Yes, it seems to have booted OK so whatever that directory is it did not want to be restored.
_________________
_______________________________________
Oh how I love Linux, especially on gentoo.
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Mon Apr 18, 2005 1:13 pm    Post subject: Reply with quote

glad to hear it's working. i thought it was just an issue of where and how you tried extracting the tarball. anyway, i put up a new section: "5. Restore" in the initial thread - thanks to your help.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
saskatchewan46
n00b
n00b


Joined: 08 Jul 2003
Posts: 57
Location: Saskatchewan

PostPosted: Mon Apr 18, 2005 11:26 pm    Post subject: Reply with quote

Thanks for this cool script. I've have been needing something like this; especially after a recent bad ram scare.

I did have problems though when I wanted to exclude a folder with a space in its name (ie: "this folder"). I have tried adding "--exclude=/home/me/this\ folder/*" , "--exclude=/home/me/this folder/*", variations with ' ' and etc but no good. I though maybe there was an issue with the For loop in your verify routine, but even when I hardcoded the folder to the tar_command_append, it still never excluded it. Maybe does tar have issues with excluding "this folder"? Probably just me though..

Anyways, long story short, rather then excluding the folder, I just excluded *.iso files from my tar. Would adding an option like exclude_file_pattern="--exclude=*.iso" be useful to your script?

Thanks again.
Back to top
View user's profile Send private message
Slimm
n00b
n00b


Joined: 15 Dec 2004
Posts: 14

PostPosted: Wed Apr 20, 2005 9:52 am    Post subject: Reply with quote

thanks. thanks to a backup made from this script, i was just able to recover from the aftereffects of a temporary bout of stupidity.
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Wed Apr 20, 2005 8:32 pm    Post subject: Reply with quote

saskatchewan46 wrote:
Would adding an option like exclude_file_pattern="--exclude=*.iso" be useful to your script?

it most certainly would. good idea, thanks.

updated script to v1.4.2
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Lucky B
Apprentice
Apprentice


Joined: 01 Apr 2005
Posts: 235
Location: Florida

PostPosted: Wed Apr 20, 2005 10:14 pm    Post subject: Reply with quote

I did a script to do something similar to yours but... how shall we say... more crude/hackish.

It actually just backs up only the stuff in your system profile, which, arguably, is all I need to recover =P

have a look, maybe you can implement it as a 5th option =P

http://forums.gentoo.org/viewtopic-t-325979.html
_________________
We are Dyslexic of Borg. Fusistance is retile. Your ass will be laminated.
Back to top
View user's profile Send private message
robgrady
Apprentice
Apprentice


Joined: 30 Sep 2004
Posts: 199

PostPosted: Thu Apr 21, 2005 4:50 pm    Post subject: Reply with quote

For all those having difficulties with tar not allowing you to include a file or directory under an excluded directory, I think I found a solution.
Code:
# tar -cvvzf kernel.tgz --exclude=/usr/src/ /usr/src/linux-`uname -r`
tar: Removing leading `/' from member names
drwxr-xr-x root/root         0 2005-04-11 12:48:55 /usr/src/linux-2.6.11-gentoo-r5/
drwxr-xr-x root/root         0 2005-04-11 12:45:23 /usr/src/linux-2.6.11-gentoo-r5/drivers/
drwxr-xr-x root/root         0 2005-04-11 12:45:23 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/
drwxr-xr-x root/root         0 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/
-rw-r--r-- root/root     29845 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/radeon_monitor.c
-rw-r--r-- root/root     11897 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/mach64_accel.c
-rw-r--r-- root/root     20936 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/mach64_gx.c
-rw-r--r-- root/root       479 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/Makefile
<snip>

Let me assure you, the text in the <snip> included only files from "/usr/src/linux-2.6.11-gentoo-r5". Now, if you use "--exclude=/usr/src" or "--exclude=/usr/src/*" it won't work. Admittedly, I have not tried this with more complicated commands, so maybe something else will break it, I'm not sure. Hope that helps.
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Thu Apr 21, 2005 7:49 pm    Post subject: Reply with quote

GNU Tar wrote:
-I include-file Opens include-file containing a list of files, one per line, and treats it as if each file appeared separately on the command line. Be careful of trailing white spaces. Also beware of leading white spaces, since, for each line in the included file, the entire line (apart from the newline) will be used to match against the initial string of files to include. In the case where excluded files (see X function modifier) are also specified, they take precedence over all included files. If a file is specified in both the exclude-file and the include-file (or on the command line), it will be excluded.

important part: If a file is specified in both the exclude-file and the include-file (or on the command line), it will be excluded.
i just tried it again first with some small archives and then with the script. i don't see any logic when some files get backed up or not when a folder is both excluded and included. well, it simply doesn't work properly.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
robgrady
Apprentice
Apprentice


Joined: 30 Sep 2004
Posts: 199

PostPosted: Thu Apr 21, 2005 8:29 pm    Post subject: Reply with quote

Yep, I ran a few more test cases and I'm seeing oddities also. I saw that part of the man page, I just don't believe that there isn't a way around it.
Back to top
View user's profile Send private message
Sheepdogj15
Guru
Guru


Joined: 07 Jan 2005
Posts: 430
Location: Backyard

PostPosted: Sat Apr 23, 2005 8:00 am    Post subject: Reply with quote

very cool. 8) right now i'm using the script and it looks like things are going well.

if i may make a suggestion, i've noticed that the portage directory gets a bit bloated. ATM it is an excess of 1.1GB on my system. it might just compact down well, but for some people where disk space is a concern, if they have to do a restore they might be better off d/ling a portage snapshot and then doing an rsync.

also, you can shave hundreds of MBs (if not GBs) by excluding ccache's ... cache. to my knowledge these files are not necessary for emerge to work properly. i could be wrong though

so, what i suggest is that:
Code:
--exclude=/usr/portage/*
--exclude=/root/.ccache/*


... be added to custom_exclude_list (that way, people can choose whether or not to back these up).

what do you think?
_________________
Sheepdog
Why Risk It? | Samba Howto
Back to top
View user's profile Send private message
thomasa88
Tux's lil' helper
Tux's lil' helper


Joined: 13 Apr 2005
Posts: 143
Location: Sweden

PostPosted: Sat Apr 23, 2005 8:59 am    Post subject: Reply with quote

THX DUDE :D

The script worked perfectly

what I did:

backup: ;)
- bz2 using the script
- cp bz2 to other computer

destroy:
(dont do this if u dont want to restore: :roll:)
-time: 23:52
-try to resize ext3 doing following
-convert to ext2
-cant resize so I decide to extend partition with fdisk (putting on same startblock)
-disk is mountable
-cant run resize, trying fsck, cant find superblocks
-testing diff superblock positions, cant find any
-try to find some way to search for them on google (time: 1:00)
-dont find any, but wait: mke2fs -S -b 4096
-just one error in that thinking: the superblocks are pulled sidewise, it overwrites file data, yes! :S (should have fdisked it back :P)

restore:
-booting with livecd
-fdisk
-no samba :S :P, but it got ssh so I able to use sftp
-putting bz2 on temp-partition (going to be win sometime :P)
-mounting /mnt/gentoo and /mnt/gentoo/boot
-cd /mnt/gentoo
-tar xjvpf bz2
-chroot /mnt/gentoo /bin/bash
-mount -t proc proc /proc (necessary for grub?)
-emerge --sync
-edit /etc/fstab
-configure and install grub - cant find any device - looking into /dev no devs - udev?
(-trying to things, doesnt work)
-restart, in grub: setup (hd0,0)
-boot - ooops, havent changed grub.conf since partition :P
-boot with right =root and change my grub.conf
-WEE :D, up'n'running


conclusion:
- i have a very destructive behavior
- got partitions with better sizes

btw, thx again :D

Edit.

is it any possibility to get this script into portage? (just do emerge mkstage4 and get latest ver)

Edit.

went from 10GB disk usage to 4GB, nice to get rid of tmp-files and stuff :)
_________________
- thomasa88
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Sun Apr 24, 2005 7:07 am    Post subject: Reply with quote

Sheepdogj15 wrote:
so, what i suggest is that:
Code:
--exclude=/usr/portage/*
--exclude=/root/.ccache/*


... be added to custom_exclude_list (that way, people can choose whether or not to back these up).

what do you think?

i do that already :roll: . my ccache goes into /var/tmp/portage which is excluded and i do exclude /usr/portage/* already. maybe i forgot to do that in one version as i'm still playing around with some options.

a new version is on the way. it does need some testing though, but the problem about first creating a tar and compressing it afterwards will be gone.

@thomasa88: i don't know how something goes into portage - and i haven't thought about that yet.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Sheepdogj15
Guru
Guru


Joined: 07 Jan 2005
Posts: 430
Location: Backyard

PostPosted: Sun Apr 24, 2005 7:36 pm    Post subject: Reply with quote

weird. i wonder why my ccache goes in to /root instead. :? and ah yes, i didn't see the line regarding portage before. my bad
_________________
Sheepdog
Why Risk It? | Samba Howto
Back to top
View user's profile Send private message
jtaylor72
Apprentice
Apprentice


Joined: 01 Feb 2004
Posts: 197
Location: "the can", WA

PostPosted: Sun Apr 24, 2005 7:47 pm    Post subject: Reply with quote

partimage works for me
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Sun Apr 24, 2005 7:48 pm    Post subject: Reply with quote

thanks for the input anyway - i'm working on a "suit everybody" solution now. i manually set the ccache variable as i don't want some temporary files in /root. the right location is /etc/make.conf:
Code:
CCACHE_SIZE="2G"
CCACHE_DIR="${PORTAGE_TMPDIR}/ccache"

where my $(PORTAGE_TMPDIR) is /tmp. this isn't ordinary neither, but portages tmp files are really tmp files for me so i don't want them in /var/tmp but in /tmp. of course, suit yourself. i think it didn't work for me though (check with "echo $CCACHE_DIR") so i added the following to /root/.bashrc:
Code:
export CCACHE_DIR=/tmp/portage/ccache
export CCACHE_SIZE=2G

_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Sheepdogj15
Guru
Guru


Joined: 07 Jan 2005
Posts: 430
Location: Backyard

PostPosted: Sun Apr 24, 2005 8:23 pm    Post subject: Reply with quote

i see now. and that isn't a bad idea, i'll probably change my setup for ccache too.
_________________
Sheepdog
Why Risk It? | Samba Howto
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Mon Apr 25, 2005 8:37 am    Post subject: Reply with quote

udev users: please read the section 0.a. if you have any further info/failure/success/hints, please post it here.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
ewildgoose
n00b
n00b


Joined: 02 Mar 2003
Posts: 66

PostPosted: Mon Apr 25, 2005 6:14 pm    Post subject: Reply with quote

Just discovered this thread. Nice script!

Just a couple of comments. I think that the default excludes should definitely list /usr/portage/distfiles, but I would put the whole of /usr/portage as one of the optional excludes. Also I would put /var/log into the optional bucket as well. For me both of these would be useful if I was doing a baremetal re-install, and I would prefer to keep them.

In practice of course one might do an (infrequent) system backup and a (frequent) data backup, and so I agree that they deserve to be kept in the optional bucket.

The lack of compression is a major bind for me though. I don't have twice the diskspace available for the backup...

One possible solution might be switching to "dar". See http://dar.sourceforge.net/

DAR is basically an updated tar, but it has a few other advantages like recovery from corrupted archives (nice) and also backup of extended attributes (which are useful on a hardened server and a few other reasons).

I will be switching to DAR myself now, will drop a new command line here if it all work out.
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Mon Apr 25, 2005 7:44 pm    Post subject: Reply with quote

ewildgoose wrote:
Just a couple of comments. I think that the default excludes should definitely list /usr/portage/distfiles, but I would put the whole of /usr/portage as one of the optional excludes. Also I would put /var/log into the optional bucket as well. For me both of these would be useful if I was doing a baremetal re-install, and I would prefer to keep them.

well, /usr/portage/* is in the default_exludes, meaning everything deeper is excluded too. what i'm trying to guarantee is a minimal working system backup. if you don't agree my policy change the script - it's easy and well documented.

i got a version working which doesn't need to create a tar first and then compress the tar. as there are still some issues with udev i don't change the version yet (or everybody thinks this is a issue of my script). if someone wants the new version right away, pm me.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Sheepdogj15
Guru
Guru


Joined: 07 Jan 2005
Posts: 430
Location: Backyard

PostPosted: Mon Apr 25, 2005 10:20 pm    Post subject: Reply with quote

BlinkEye wrote:
ewildgoose wrote:
Just a couple of comments. I think that the default excludes should definitely list /usr/portage/distfiles, but I would put the whole of /usr/portage as one of the optional excludes. Also I would put /var/log into the optional bucket as well. For me both of these would be useful if I was doing a baremetal re-install, and I would prefer to keep them.

well, /usr/portage/* is in the default_exludes, meaning everything deeper is excluded too. what i'm trying to guarantee is a minimal working system backup. if you don't agree my policy change the script - it's easy and well documented.


yeah... i found it was pretty much as easy as moving --exclude=/usr/portage/* from the default list to the custom list. getting more granular with what all is in portage is similarly as simple.


also on another point, i had a similar problem involving /dev on the restore, except i already had devfs set up as a fall back. what happened was i changed my partition structure, and thus i had to rerun Grub. however, it couldn't find my hardware. well, it took a bit of time before i recalled the script excluded the dev filesystem. i found that the easy solution to getting that working is, from the LiveCD (outside the chroot), mount dev like this:

Code:
mount -t devfs none /mnt/gentoo/dev


and then chroot.

I just thought i would mention it in case other people have trouble with this.
_________________
Sheepdog
Why Risk It? | Samba Howto
Back to top
View user's profile Send private message
bigfunkymo
Apprentice
Apprentice


Joined: 23 Jan 2004
Posts: 237

PostPosted: Tue Apr 26, 2005 12:26 pm    Post subject: Reply with quote

IIIRC, the udev HOWTO mentions needing to have /dev/console and /dev/null. perhaps you don't need to backup all of /dev, maybe just those 2 files
_________________
[No package... Grabbing a set.]
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Tue Apr 26, 2005 1:34 pm    Post subject: Reply with quote

the script does that. i don't backup /dev at all (only /dev/console and /dev/null). it's not that simple.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
bigfunkymo
Apprentice
Apprentice


Joined: 23 Jan 2004
Posts: 237

PostPosted: Tue Apr 26, 2005 7:00 pm    Post subject: Reply with quote

perhaps there are other /dev files that should be part of the specific inclusions?

If we had more information on why the systems using udev are failing, we (the forum community) might be able to provide a little insight.

Do you think it would hurt anything to remove the /dev exclusion?

Also along those same lines, for those who have a "/boot/boot -> ." symlink. Would it help to add:
Code:
--exclude=/boot/boot

_________________
[No package... Grabbing a set.]
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1043
Location: Gentoo Forums

PostPosted: Tue Apr 26, 2005 9:03 pm    Post subject: Reply with quote

i tried that of course. it doesn't work neither. i don't understand udev yet, but it doesn't use a device tarball (at least not if you follow the udev guid on gentoo.org). but i figured it out (at least for my case): to get rid of those nasty errors about not finding the device (/dev/sda1) i added the following three lines to /etc/init.d/checkroot:
/etc/init.d/checkroot wrote:
start() {
local retval=0

if [ ! -f /fastboot -a -z "${CDBOOT}" ]
then

ebegin "Creating /dev/sda"
cd /dev
MAKEDEV /dev/sda &>/dev/null



ebegin "Remounting root filesystem read-only (if necessary)"
mount / -n -o remount,ro &>/dev/null
eend $?

....

this will create the necessary nodes before actually trying to mount non-existing ones. i can't test it with a raid system at the moment, but i booted my laptop and my amd64 raid0 system with my external usb harddisk. i bet you would have to do exactly the same thing if you try to boot a raid system which uses /dev/md*, i.e. just replace /dev/sda above with /dev/md.
after booting the system once with these lines you may afterwards remove them (i don't know why).
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5 ... 16, 17, 18  Next
Page 4 of 18

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum