Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Backup Images too big
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
elomaniak
n00b
n00b


Joined: 27 Jan 2013
Posts: 47

PostPosted: Sat Feb 01, 2014 7:40 pm    Post subject: Backup Images too big Reply with quote

Hi Folks,

I am currently building my system for a small Wandboard Cluster.
So I do often backup all my done things, just to be able to revert without a problem.

I am using 16GB cards, and when I make a backup with dd i get a 16GB img. i compress them with gzip, and the size is about 800MB
but transfering them back on SD is a pain in the ass, even with class10 cards

is there any way to make an image of the used card witht he actual size of data? and not the whole card?

Thanks in advance
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Sat Feb 01, 2014 7:59 pm    Post subject: Re: Backup Images too big Reply with quote

elomaniak wrote:
Hi Folks,

I am currently building my system for a small Wandboard Cluster.
So I do often backup all my done things, just to be able to revert without a problem.

I am using 16GB cards, and when I make a backup with dd i get a 16GB img. i compress them with gzip, and the size is about 800MB
but transfering them back on SD is a pain in the ass, even with class10 cards

is there any way to make an image of the used card witht he actual size of data? and not the whole card?

Thanks in advance


dd is lowlevel so it will read/write every block, it has no knowledge of files. If you want to do that you need something that is file aware like rsync or similar. dd will just read all sectors beginning to end, gzip will compress the empty space nicely but when restoring the image it has to write the empty space too which is why it takes so long.

You could potentially write a bash script that:

1. Mounts partition(s) and rsync's all the data off (diff so it will be fast)
2. dd the bootsector into an img file

When using a bash script to restore you could make it:

1. Re-partition the card (Optionally when you really b0rked it)
2. Mount partition(s) and rsync data back
3. Restore bootsector (Optionally when you really b0rked it)

And when you're scripting skills are up to it you can automate it that when you call the script it can detect the state of the SD card and validate the partitions, repartition if it doesn't look allright etc and using parameters like --backup or --restore rather than making seperate scripts for the jobs..

idea :?:
Back to top
View user's profile Send private message
elomaniak
n00b
n00b


Joined: 27 Jan 2013
Posts: 47

PostPosted: Sat Feb 01, 2014 8:19 pm    Post subject: Reply with quote

currently i am not very familiar with bash scripting

could you help me out with the dd on the rsync, diff and dd on bootsector?
thanks in advance
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Sat Feb 01, 2014 8:53 pm    Post subject: Reply with quote

Think you're mixing some things up here,

You us DD to read/write blocks from/to a device. (this is not aware of any file structure)
You use rsync to synchronise files.

Let's say your SD Card has a root partition you are concerned about, /dev/sdb3 for instance. You create a folder sdb3 under /mnt and mount the root partition of the SD Card there. You can then synchronize the files to a folder in your homefolder with something like this:

rsync -avz --delete /mnt/sdb ~/backup/

This will take all files from /mnt/sdb and copy them to the backup folder in your home folder. Anything deleted from the memorystick gets deleted from the backup folder, anything changed gets updated and anything unchanged is not transferred to save time.

You now have all the files from the root filesystem of the SD Card. If you break something on the filesystem you can mount the SD Card and synchronise the files back by reversing the command. However, this does not backup the files on the boot partition (if it has one) or the bootsector (if it has one).

To get the bootsector you can dd the blocks off or use some other tool like sfdisk as this is easier. The parition table can be re-created if damaged or lost. So if you would nuke an SD card, your script would have to do the following to get it back in order:

1. Recreate partitions and format correctly (assuming boot, swap and root for now)
2. Mount partitions and rsync the files back onto the SD Card
3. Restore the bootsector.


Kinda depends on a few things but theoretically you can get away with writing only the actual data instead of writing 15GiB of empty space every time.

I would post the partition layout first.
Is the SD Card bootable?
Which bootloader is it using?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54096
Location: 56N 3W

PostPosted: Sat Feb 01, 2014 9:23 pm    Post subject: Reply with quote

elomaniak,

pakjebakmeel has it spot on however different bootloaders use different amounts of the 'wasted' space before the first partition.
As this space is outside any filesystem, you need to use dd to save it.

e.g. Grub (legacy) uses about 12k here for its stage1.5
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sat Feb 01, 2014 9:27 pm    Post subject: Re: Backup Images too big Reply with quote

elomaniak wrote:

is there any way to make an image of the used card witht he actual size of data? and not the whole card?

Use pipes:

dd if=/dev/sdX | gzip > sdXimage.gz


To restore:

zcat sdXimage.gz | dd of=/dev/sdX


Restore should take max 30 minutes
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54096
Location: 56N 3W

PostPosted: Sat Feb 01, 2014 9:31 pm    Post subject: Reply with quote

Irre,

That still writes the entire empty space.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 434
Location: Stockholm

PostPosted: Sat Feb 01, 2014 9:52 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Irre,

That still writes the entire empty space.

I know, but this is a simple solution. Not good if you restore often.
Partimage is very useful for most partitions. (not all filesystems are supported). It does not backup or write empty space.

emerge partimage
Back to top
View user's profile Send private message
elomaniak
n00b
n00b


Joined: 27 Jan 2013
Posts: 47

PostPosted: Wed Feb 05, 2014 4:56 pm    Post subject: Reply with quote

Thanks for all your answers

i did the rsync method.
to have the bootloader on card i just dd'd the base image I am working from, then deleted the data and rsynced with my backup.

Thans for your advices :)

Cheer
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum