View previous topic :: View next topic |
Author |
Message |
elomaniak n00b
Joined: 27 Jan 2013 Posts: 47
|
Posted: Sat Feb 01, 2014 7:40 pm Post subject: Backup Images too big |
|
|
Hi Folks,
I am currently building my system for a small Wandboard Cluster.
So I do often backup all my done things, just to be able to revert without a problem.
I am using 16GB cards, and when I make a backup with dd i get a 16GB img. i compress them with gzip, and the size is about 800MB
but transfering them back on SD is a pain in the ass, even with class10 cards
is there any way to make an image of the used card witht he actual size of data? and not the whole card?
Thanks in advance |
|
Back to top |
|
|
pakjebakmeel n00b
Joined: 13 Feb 2012 Posts: 48
|
Posted: Sat Feb 01, 2014 7:59 pm Post subject: Re: Backup Images too big |
|
|
elomaniak wrote: | Hi Folks,
I am currently building my system for a small Wandboard Cluster.
So I do often backup all my done things, just to be able to revert without a problem.
I am using 16GB cards, and when I make a backup with dd i get a 16GB img. i compress them with gzip, and the size is about 800MB
but transfering them back on SD is a pain in the ass, even with class10 cards
is there any way to make an image of the used card witht he actual size of data? and not the whole card?
Thanks in advance |
dd is lowlevel so it will read/write every block, it has no knowledge of files. If you want to do that you need something that is file aware like rsync or similar. dd will just read all sectors beginning to end, gzip will compress the empty space nicely but when restoring the image it has to write the empty space too which is why it takes so long.
You could potentially write a bash script that:
1. Mounts partition(s) and rsync's all the data off (diff so it will be fast)
2. dd the bootsector into an img file
When using a bash script to restore you could make it:
1. Re-partition the card (Optionally when you really b0rked it)
2. Mount partition(s) and rsync data back
3. Restore bootsector (Optionally when you really b0rked it)
And when you're scripting skills are up to it you can automate it that when you call the script it can detect the state of the SD card and validate the partitions, repartition if it doesn't look allright etc and using parameters like --backup or --restore rather than making seperate scripts for the jobs..
idea |
|
Back to top |
|
|
elomaniak n00b
Joined: 27 Jan 2013 Posts: 47
|
Posted: Sat Feb 01, 2014 8:19 pm Post subject: |
|
|
currently i am not very familiar with bash scripting
could you help me out with the dd on the rsync, diff and dd on bootsector?
thanks in advance |
|
Back to top |
|
|
pakjebakmeel n00b
Joined: 13 Feb 2012 Posts: 48
|
Posted: Sat Feb 01, 2014 8:53 pm Post subject: |
|
|
Think you're mixing some things up here,
You us DD to read/write blocks from/to a device. (this is not aware of any file structure)
You use rsync to synchronise files.
Let's say your SD Card has a root partition you are concerned about, /dev/sdb3 for instance. You create a folder sdb3 under /mnt and mount the root partition of the SD Card there. You can then synchronize the files to a folder in your homefolder with something like this:
rsync -avz --delete /mnt/sdb ~/backup/
This will take all files from /mnt/sdb and copy them to the backup folder in your home folder. Anything deleted from the memorystick gets deleted from the backup folder, anything changed gets updated and anything unchanged is not transferred to save time.
You now have all the files from the root filesystem of the SD Card. If you break something on the filesystem you can mount the SD Card and synchronise the files back by reversing the command. However, this does not backup the files on the boot partition (if it has one) or the bootsector (if it has one).
To get the bootsector you can dd the blocks off or use some other tool like sfdisk as this is easier. The parition table can be re-created if damaged or lost. So if you would nuke an SD card, your script would have to do the following to get it back in order:
1. Recreate partitions and format correctly (assuming boot, swap and root for now)
2. Mount partitions and rsync the files back onto the SD Card
3. Restore the bootsector.
Kinda depends on a few things but theoretically you can get away with writing only the actual data instead of writing 15GiB of empty space every time.
I would post the partition layout first.
Is the SD Card bootable?
Which bootloader is it using? |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54237 Location: 56N 3W
|
Posted: Sat Feb 01, 2014 9:23 pm Post subject: |
|
|
elomaniak,
pakjebakmeel has it spot on however different bootloaders use different amounts of the 'wasted' space before the first partition.
As this space is outside any filesystem, you need to use dd to save it.
e.g. Grub (legacy) uses about 12k here for its stage1.5 _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
Irre Guru
Joined: 09 Nov 2013 Posts: 434 Location: Stockholm
|
Posted: Sat Feb 01, 2014 9:27 pm Post subject: Re: Backup Images too big |
|
|
elomaniak wrote: |
is there any way to make an image of the used card witht he actual size of data? and not the whole card?
|
Use pipes:
dd if=/dev/sdX | gzip > sdXimage.gz
To restore:
zcat sdXimage.gz | dd of=/dev/sdX
Restore should take max 30 minutes |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54237 Location: 56N 3W
|
Posted: Sat Feb 01, 2014 9:31 pm Post subject: |
|
|
Irre,
That still writes the entire empty space. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
Irre Guru
Joined: 09 Nov 2013 Posts: 434 Location: Stockholm
|
Posted: Sat Feb 01, 2014 9:52 pm Post subject: |
|
|
NeddySeagoon wrote: | Irre,
That still writes the entire empty space. |
I know, but this is a simple solution. Not good if you restore often.
Partimage is very useful for most partitions. (not all filesystems are supported). It does not backup or write empty space.
emerge partimage |
|
Back to top |
|
|
elomaniak n00b
Joined: 27 Jan 2013 Posts: 47
|
Posted: Wed Feb 05, 2014 4:56 pm Post subject: |
|
|
Thanks for all your answers
i did the rsync method.
to have the bootloader on card i just dd'd the base image I am working from, then deleted the data and rsynced with my backup.
Thans for your advices
Cheer |
|
Back to top |
|
|
|