| View previous topic :: View next topic |
| Author |
Message |
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 737
|
 Posted: Thu Apr 07, 2016 9:27 am Post subject: backup over network with ssh connection in a compressed file |
 |
|
i have two computers. pc1 and pc2
from pc2 i am connectet with ssh to pc1.
i will make a backup from sda in pc1 with dd.
i will create the backup file on pc2.
i will use a the best compression as possible to create the backup file.
can i do this on the fly?
how does it works?
i know there was anything that dd do not send in a file, dd must send in a compression programm?
And this compressions programm must write over ssh and create than a file... ?!
can anyone please help me? |
|
| Back to top |
|
 |
DawgG
l33t
Joined: 17 Sep 2003
Posts: 878
|
| Posted: Thu Apr 07, 2016 11:19 am Post subject: |
|
|
the simplest way is probably to install sys-fs/sshfs on the computer whose hard-disk you want to backup.
then you log into that computer and mount the target-dir (where you want to save the backup-file) of the computer that saves the backup-file some place; eg /tmp/backup-target with sshfs.
for compression with dd you pipe its output to a compression-prog; eg xz (app-arch/xz-utils is usually installed) and redirct that output to the compressed image-file.
to use the most extreme compression-level (9) of xz you could use:
| Code: | | dd if=/dev/sda bs=4K | xz -z9 > /tmp/backup-target/sda.img.xz |
(if you can't or do not want to use sshfs there are more advanced setups with netcat.)
GOOD LUCK!
_________________
DUMM KLICKT GUT. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Apr 07, 2016 11:50 am Post subject: |
|
|
SarahS93 ...
firstly, dd shouldn't really be considered a backup tool, you should really consider using rsync, or tar, for this (with the former you can do incremental backups, and can use ssh transparently).
As for passing data to ssh, you can use a pipe ... here we use 'tar over ssh' to backup /home
| Code: | | # tar Jcvf - /home | ssh root@192.168.x.x "cat > /backup/home-$(date +%F).tar.xz" |
Note that the compression is done by tar, so if you have 'Compression' set globally in ~/.ssh/config you may want to disable it for this host.
You could similarly use dd and pipe its output ...
| Code: | | # dd if=/dev/sda | ssh -C root@192.168.x.x "dd of=/backup/dd-sda-$(date +%F).img" |
... though probably sys-fs/ddrescue would be a better option as you could use --sparse on the output. As for compression a different sort of pipeline would be needed ... but again I don't think dd is the right tool for the job here. When you dd a disk you are backing up every block, including 'live' data such as /proc /sys /dev fifo's, etc ... rsync on the other had can be configured (via rsync-excludes) to omit these from the backup, so recovery, should it be needed, is less troublesome.
HTH & best ... khay
Last edited by khayyam on Thu Apr 07, 2016 11:58 am; edited 1 time in total |
|
| Back to top |
|
|
Syl20
l33t
Joined: 04 Aug 2005
Posts: 621
Location: France
|
| Posted: Thu Apr 07, 2016 11:54 am Post subject: |
|
|
| khayyam wrote: | | rsync on the other had can be configured (via rsync-excludes) to omit these from the backup, so recovery, should it be needed, is less troublesome. |
And you can use rsync "over ssh", without pipe, with the -e option. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Apr 07, 2016 12:00 pm Post subject: |
|
|
| Syl20 wrote: | | khayyam wrote: | | rsync on the other had can be configured (via rsync-excludes) to omit these from the backup, so recovery, should it be needed, is less troublesome. |
And you can use rsync "over ssh", without pipe, with the -e option. |
Syl20 ... I did say that ... "and can use ssh transparently".
best ... khay |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 737
|
| Posted: Thu Apr 07, 2016 12:12 pm Post subject: |
|
|
i have put the gentoo minimal boot cd on a flash drivee and boot it up on pc1, there is no sshfs installed.
i have mount from pc2 a samba-share-folder at pc1
and i am trying now the way with dd
dd if=/dev/sda bs=4K | xz -z9 > /mnt/pc2/sda.img.xz
dont know if i need it, but how can i do this backward?
how can i play the sda.img.xz file back on sda at pc1?
is there any way to see how many % are down in the dd job?
dont know if this the best way ( dd if=/dev/sda bs=4K | xz -z9 > /mnt/pc2/sda.img.xz )
pc1 is not the fastest, pc2 is very fast
is there any way that pc2 do the compress job? |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 737
|
| Posted: Thu Apr 07, 2016 12:26 pm Post subject: |
|
|
how can i start at pc2 a command thats connects via ssh without any compression to pc1, read there the complete /dev/sda file, send it without compression to pc2 and there comes the stream in xz that make the compression and write the file on a local drive?
(both computers are connectet with 1gbit lan and pc2 is a very fast 8 core machine.) |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Apr 07, 2016 12:54 pm Post subject: |
|
|
| SarahS93 wrote: | | dont know if i need it, but how can i do this backward? how can i play the sda.img.xz file back on sda at pc1? |
SarahS93 ... you reverse the pipeline:
| Code: | | # xzcat /mnt/pc2/sda.img.xz | dd of=/dev/sda |
| SarahS93 wrote: | | is there any way to see how many % are down in the dd job? |
People generally put sys-apps/pv (pipe viewer) in the pipeline for this purpose, but you could send a -USR1 signal and dd will report its status.
| Code: | | # kill -USR1 $(pgrep "^dd") |
or ... use 'watch' (the '60' is seconds):
| Code: | | # watch -n60 'kill -USR1 $(pgrep "^dd")' |
| SarahS93 wrote: | | is there any way that pc2 do the compress job? |
Yes, just make the pipeline use ssh ...
| Code: | | # dd if=/dev/sda bs=4K | ssh root@192.168.x.x "xz -z9 > /path/to/sda.img.xz" |
| SarahS93 wrote: | | how can i start at pc2 a command thats connects via ssh without any compression to pc1, read there the complete /dev/sda file, send it without compression to pc2 and there comes the stream in xz that make the compression and write the file on a local drive? |
*If* I understand ... you could send the 'ssh pipeline' as a command to pc1 via ssh
| Code: | | # ssh root@pc1 'dd if=/dev/sda bs=4K | ssh root@pc2 "xz -z9 > /path/to/sda.img.xz"' |
... that is untested, so I'm not sure how the quotes will be handled, single quotes should protect double quotes though.
best ... khay |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 737
|
| Posted: Thu Apr 07, 2016 1:02 pm Post subject: |
|
|
hey khay
there is a little problem. i can not connect from pc1 to pc2 with ssh, from pc2 to pc1 with ssh i possible. |
|
| Back to top |
|
|
SarahS93
l33t
Joined: 21 Nov 2013
Posts: 737
|
| Posted: Thu Apr 07, 2016 1:08 pm Post subject: |
|
|
at pc2 i can run this command.
| Code: | | ssh user@pc1 "cat /dev/sda" | xz -z4e -T 8 -c > /path/file/tosave |
dont know if this the right and best way for me
pc1: ~5% cpu use
pc2: ~75% cpu use
network: 10-20 mb/s
looks like it works with dd too
| Code: | | ssh user@pc1 "dd if=/dev/sda" | xz -z4e -T 8 -c > /path/file/tosave |
with the
| Code: | | watch -n10 'kill -USR1 $(pgrep "^dd")' |
i can see now how much are read |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Apr 07, 2016 1:37 pm Post subject: |
|
|
| SarahS93 wrote: | | there is a little problem. i can not connect from pc1 to pc2 with ssh, from pc2 to pc1 with ssh i possible. |
SarahS93 ... if using the minimal install image 'PermitRootLogin' is disabled, so you would need to enable it and restart ssh. I suspect that is the issue, if not then please provide more info.
edit: actually I think 'PermitRootLogin' is disabled by default on all ssh installs currently, so probably you need to enable it (see /etc/ssh/sshd_config).
best ... khay |
|
| Back to top |
|
|
Syl20
l33t
Joined: 04 Aug 2005
Posts: 621
Location: France
|
| Posted: Thu Apr 07, 2016 2:07 pm Post subject: |
|
|
| khayyam wrote: | | Syl20 ... I did say that ... "and can use ssh transparently". |
Oups. I missed it.  |
|
| Back to top |
|
|
Cyker
Veteran
Joined: 15 Jun 2006
Posts: 1746
|
| Posted: Thu Apr 07, 2016 7:35 pm Post subject: |
|
|
Do you have to use ssh?
Last time I did this, I just used netcat - It's MUCH faster, and you don't have to do stupid dangerous things like PermitRootLogins.
IIRC the commands were something like
| Destination: | | netcat -l 13370 | dd bs=1M of=/whatever-you-want |
| Source: | | dd bs=1M if=/dev/sda | netcat <address of Destination> 13370 |
|
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3606
|
| Posted: Thu Apr 07, 2016 7:59 pm Post subject: |
|
|
Hint: do not event try doing image backups from actively used filesystem. If you want to have image backup, you _MUST_ unmount the device first or backup a snapshot instead.
And I do mean it. Simple dd will silently corrupt your backup rendering it useless. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Apr 07, 2016 8:43 pm Post subject: |
|
|
| Cyker wrote: | | Do you have to use ssh? Last time I did this, I just used netcat - It's MUCH faster, and you don't have to do stupid dangerous things like PermitRootLogins. |
Cyker ... the OP is booting from a minimal install CD, and so no netcat is available, also the reason ssh would be slower than netcat is that the ssh transfer is encrypted, and so if "stupid dangerous" is of concern then cleartext would probably also check that box, besides, 'PermitRootLogin prohibit-password' is perfectly fine.
best ... khay |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Apr 07, 2016 8:57 pm Post subject: |
|
|
| szatox wrote: | | Hint: do not event try doing image backups from actively used filesystem. If you want to have image backup, you _MUST_ unmount the device first or backup a snapshot instead. And I do mean it. Simple dd will silently corrupt your backup rendering it useless. |
szatox ... hehe, I'd go one further ... don't use dd for backups (except perhaps for backing up the MBR). With rsync you get the flexibility of increments, get to control what exactly gets backed up (ie, exclude portage, distfiles, etc, as these are easily replaced), get to roll back to a specific backup, and don't need to backup everything again the next time you make a backup. Even tar would be preferable I think, so, unless you're making some sort of install image then really dd isn't the sort of thing I'd recommend (you probably know that, that's said purely for the purpose of the discussion).
best ... khay |
|
| Back to top |
|
|
toralf
Developer
Joined: 01 Feb 2004
Posts: 3943
Location: Hamburg
|
| Posted: Thu Apr 07, 2016 9:50 pm Post subject: |
|
|
if you are at the prompt at pc2, then do something like : | Code: | | ssh user@pc1 "dd if=/dev/sda" | xz > backup.xz |
|
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Thu Apr 07, 2016 10:14 pm Post subject: |
|
|
| toralf wrote: | | Code: | | ssh user@pc1 "dd if=/dev/sda" | xz > backup.xz |
|
toralf ... indeed, much better than the solution provided by me.
best ... khay |
|
| Back to top |
|
|
|
Multimedia
