Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Clone a reference PC with any OS to multiple targets quickly
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Off the Wall
View previous topic :: View next topic  
Author Message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1330
Location: Fayetteville, NC, USA

PostPosted: Tue Jul 17, 2018 4:34 am    Post subject: Clone a reference PC with any OS to multiple targets quickly Reply with quote

So I had to clone a single reference PC which has Windows Vista to eleven identical PCs. This was the fastest way to do it sector by sector. Hope it helps somebody!

Cloning PCs quickly

Note: The commands for what I did are in the video description.
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
Mardok45
n00b
n00b


Joined: 21 Jun 2008
Posts: 68
Location: Right behind you

PostPosted: Tue Jul 17, 2018 9:20 pm    Post subject: Reply with quote

Nice.

The memories. Had to wipe and refurbish at least 500 of those 760's at my college.

I would have split it up into two jobs:

* dd the MBR and boot partition like how you did it in the video.
* For cloning the OS, use partclone.ntfs using the same ssh/tee command. Unless you need a forensic copy, dd is not very suitable for massive cloning.

I used [Fog](https://fogproject.org/) for cloning multiple devices. It uses IP broadcasts instead of sending packets individually to each PC.
_________________
Bones McCracker wrote:
Somebody needs to sig that.
Back to top
View user's profile Send private message
patrix_neo
Guru
Guru


Joined: 08 Jan 2004
Posts: 468
Location: The Maldives

PostPosted: Wed Jul 18, 2018 3:16 pm    Post subject: Reply with quote

@Mardok - How would you have done the MBR on multiple devices at the same time? I am working on a bash script....
Back to top
View user's profile Send private message
Mardok45
n00b
n00b


Joined: 21 Jun 2008
Posts: 68
Location: Right behind you

PostPosted: Thu Jul 19, 2018 12:10 am    Post subject: Reply with quote

Using an example file:

Code:

$ dd if=/dev/zero of=test.img bs=512 count=10000
$ dd if=/dev/zero of=test2.img bs=512 count=10000
$ fdisk test.img
...
$ fdisk test.img -l
Disk test.img: 4.9 MiB, 5120000 bytes, 10000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x508f0998

Device     Boot Start   End Sectors  Size Id Type
test.img1        2048  4096    2049    1M 83 Linux
test.img2        6144  9999    3856  1.9M 83 Linux


The "disk" in this case has a 512 block size and the boot partition ends at block 4096. So:

Code:
dd if=test.img bs=512 count=4096 of=test2.img


If your disk doesn't have a boot partition, just dd MBR/GPT portion of the disk, which is everything before the start of the first partition. So if we wanted to just dd the MBR in this case, it would be:
Code:
dd if=test.img bs=512 count=2048 of=test2.img


Now use partclone to clone the OS partition.

All of this is predicated on all the disks being the same. If you have a disk that is a different size, you'll have to take a different approach. I've done it by using sfdisk to create partitions to their needed size, and before creating a clone, shrink the disk partition to the smallest size possible so it'll fit.
_________________
Bones McCracker wrote:
Somebody needs to sig that.
Back to top
View user's profile Send private message
johngalt
Apprentice
Apprentice


Joined: 09 Sep 2004
Posts: 257
Location: 3rd Rock

PostPosted: Thu Jul 19, 2018 12:29 am    Post subject: Reply with quote

The_Great_Sephiroth wrote:
So I had to clone a single reference PC which has Windows Vista to eleven identical PCs. This was the fastest way to do it sector by sector. Hope it helps somebody!

Cloning PCs quickly

Note: The commands for what I did are in the video description.


Nice video - I appreciate the script in the summary as well.

Mardok45 wrote:
Nice.

The memories. Had to wipe and refurbish at least 500 of those 760's at my college.

I would have split it up into two jobs:

* dd the MBR and boot partition like how you did it in the video.
* For cloning the OS, use partclone.ntfs using the same ssh/tee command. Unless you need a forensic copy, dd is not very suitable for massive cloning.

I used [Fog](https://fogproject.org/) for cloning multiple devices. It uses IP broadcasts instead of sending packets individually to each PC.


I used (and still do) CloneZIlla myself, with the DBRL over iPXE. I was doing a bunch of laptops at a start-up a few years back, and found that imaging a single drive using a 3-Wire SATA to USB3 dongle to image took me around 30-35 minutes, but setting up a laptop via my HP USB 3 128 GB Flash Drive cut it down to about 4.5 minutes.

I then poked around playing with DBRL / CZ live, and did the PXE method - it took around 15 minutes, including image verification. But it took 15 minutes whether I did 1 or I did 50. I used a dedicated closed network so no one could accidentally trip into the the VLAN and grab one of my reserved IPs for cloning, and DBRL also does multi-cast broadcast - makes life a $#!7-ton easier. All these laptops were EFI -based, but we had to set them up to use MBR/non-EFI boot, instead of GPT / (u)EFI (mainly because the discs where I built the host server was a virtual drive that was prepartitioned MBR, and CZ didn't (and still doesn't) like mixing partition tables for the purpose of making backups or restoring them).

I looked into Fog for a while, but never really went past that.

I'm now having to do the same for my 2 laptops and my dekstop, making images via CZ to portable HD (SSD as soon as I build a new machine, as one of my older SSDs will get pulled out). Or, I may opt to (finally) build up a NAS for things like backups.

Incidentally - the reason I know that CZ still doesn't like crossing MBR and GPT - my Windows 10 install on my desktop is on an old BIOS machine, and Windows refuses to boot from GPT is you are not using (u)EFI, so it is MBR / BIOS. And CZ would not back that up to my portable HD partitioned as GPT....

(fixed typos)
_________________
desultory wrote:
If you want to retain credibility as a functional adult; when you are told that you are acting boorishly, the correct response is to consider that possibility and act accordingly to correct that behavior.


Amen.


Last edited by johngalt on Thu Jul 19, 2018 1:41 am; edited 2 times in total
Back to top
View user's profile Send private message
BonezTheGoon
Bodhisattva
Bodhisattva


Joined: 14 Jun 2002
Posts: 1375
Location: Albuquerque, NM -- birthplace of Microsoft and Gentoo

PostPosted: Thu Jul 19, 2018 12:42 am    Post subject: Reply with quote

For cloning I use a method much closer to Mardok45.

In most cases I:

For capture -- make a tarball of all partition that are not /boot. Then I delete all partitions that are not /boot. Then I dd the boot partition.

For restore -- use the dd output to reconstruct the boot sector, partition table, and contents of /boot. Recreate the remaining partitions as desired and mount them appropriately. Then extract the remaining OS files.

One note -- if the system is running SELinux in enforcing mode before attempting to boot the system switch to permissive or disabled so that a relabel can be done on all the files that were in the tarball.

Another note -- save the partition table values exactly if you do not want to have to restore manually on the source machine. If you recreate the partitions exactly as they were prior to the process all data will be present still and usable on the source after capture.
_________________
mcgruff wrote:
I can't promise to be civil.


pjp wrote:
The greater evil is voting for the "lesser evil."
Back to top
View user's profile Send private message
Mardok45
n00b
n00b


Joined: 21 Jun 2008
Posts: 68
Location: Right behind you

PostPosted: Thu Jul 19, 2018 1:24 am    Post subject: Reply with quote

The problem with using tarballs is it doesn't preserve filesystem permissions for NTFS correctly. I never messed around with BTRFS, but I imagine you'll lose a bunch of metadata there as well. Should be fine for ext*/xfs filesystems, but if you want to back up more advanced filesystems or NTFS, you're shit out of luck
_________________
Bones McCracker wrote:
Somebody needs to sig that.
Back to top
View user's profile Send private message
BonezTheGoon
Bodhisattva
Bodhisattva


Joined: 14 Jun 2002
Posts: 1375
Location: Albuquerque, NM -- birthplace of Microsoft and Gentoo

PostPosted: Thu Jul 19, 2018 5:36 am    Post subject: Reply with quote

Mardok45 wrote:
The problem with using tarballs is it doesn't preserve filesystem permissions for NTFS correctly. I never messed around with BTRFS, but I imagine you'll lose a bunch of metadata there as well. Should be fine for ext*/xfs filesystems, but if you want to back up more advanced filesystems or NTFS, you're shit out of luck


Yes you are right on all points. Thanks for adding that note I failed to mention!
_________________
mcgruff wrote:
I can't promise to be civil.


pjp wrote:
The greater evil is voting for the "lesser evil."
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17409

PostPosted: Thu Jul 19, 2018 12:36 pm    Post subject: Reply with quote

Whenever I use tar for that, I have to find notes or other references on the correct way to capture everything correctly (including exclusions). I do it so rarely that I just don't retain those details.
_________________
Slowly I turned. Step by step.
Back to top
View user's profile Send private message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1330
Location: Fayetteville, NC, USA

PostPosted: Thu Jul 19, 2018 10:00 pm    Post subject: Reply with quote

I used to use CloneZilla and keep it as a backup but I noticed no real speed/time difference in using dd or CZ in cases like this. Maybe if they had larger drives it would make a big difference, but these are older so no biggy. I could have also added compression with GZip prior to sending it out over SSH, but did not see the point on a 512b sector size.

Also, if I did CZ to setup a server on one and started cloning on more than two or three clients it slowed down. My guess is that it is due to read head movement on the host. If target A started ten seconds before target B, the read had does a lot of moving. The method I showed in the video does all systems simultaneously, resulting in no extra read-head movement. The down-side is that it clones only as fast as the slowest machine in the group.
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
Mardok45
n00b
n00b


Joined: 21 Jun 2008
Posts: 68
Location: Right behind you

PostPosted: Fri Jul 20, 2018 2:07 am    Post subject: Reply with quote

I'd find it surprising if dd performed just as fast as partclone. Even if it was just as fast, you're restricted to disks or partitions that are equal or greater in size than the source. The advantage with partclone is you can shrink the filesystem to its minimal size and be able to clone to disks that have that's smaller (as long as it has enough room for the data on the partition).

Also, use lzop for compression. gzip is too damned slow and doesn't save you that much bandwidth compared to the speed and compression speed compared to lzop.
_________________
Bones McCracker wrote:
Somebody needs to sig that.
Back to top
View user's profile Send private message
Muso
l33t
l33t


Joined: 22 Oct 2002
Posts: 721
Location: The Holy city of Honolulu

PostPosted: Fri Jul 20, 2018 5:46 am    Post subject: Reply with quote

@ OP

Nice work ++
_________________
People Of Love

Kindness Evokes Kindness

Peace Emits Positive Energy
Back to top
View user's profile Send private message
johngalt
Apprentice
Apprentice


Joined: 09 Sep 2004
Posts: 257
Location: 3rd Rock

PostPosted: Fri Jul 20, 2018 6:24 pm    Post subject: Reply with quote

The_Great_Sephiroth wrote:
I used to use CloneZilla and keep it as a backup but I noticed no real speed/time difference in using dd or CZ in cases like this. Maybe if they had larger drives it would make a big difference, but these are older so no biggy. I could have also added compression with GZip prior to sending it out over SSH, but did not see the point on a 512b sector size.

Also, if I did CZ to setup a server on one and started cloning on more than two or three clients it slowed down. My guess is that it is due to read head movement on the host. If target A started ten seconds before target B, the read had does a lot of moving. The method I showed in the video does all systems simultaneously, resulting in no extra read-head movement. The down-side is that it clones only as fast as the slowest machine in the group.


That's odd.

When I cloned 43 laptops from a single image using the DRBL server via PXE, the imaging process would not start until all clients had authenticated to the server, and the server then multicast to all at once. IIRC, there was a way to set up the DRBL server so that it would wait for X amount of clients (alternately, you could set a time limit, and maybe even both, just don't member anymore), which I initially (when testing multicast) set to a number larger than my target count, and there was an option that allowed you to force start even if the target count was not met. Once I got it all working and fine tuned, I started using the exact target count, which allowed me to easily see if one of my targets was not authenticating. I set an average timeout before multicast initialization to something like 10 times the average time it took for a single target to authenticate, as I would have plenty of time to work my way down even 100 clients and get them booted up in time to start the duplication. The laptops themselves easily booted and started PXE via EFI in a short amount of time, and usually within 10 seconds any given laptop from power on was authenticated to the DRBL server. I still 'pretended' that it took 30 seconds, so my 10X factor was a timeout of 300 seconds, or 5 minutes - more than enough time ot walk around checking machines to make sure everyone was in a waiting state, or force a reboot to try to authenticate, or else remove a machine from the line in the rare case that a machine would not authenticate correctly for whatever reason.

In my case, though, all the laptops were identical, in terms of physical build and hardware, with the only exception that the HD manufacturers might be different, but the specs were as near to identical as possible. The HDs were 320 GB 5400 rpm platter drives, but I did the testing on a closed gigabit network, using a spare 48 port gigabit switch so there was no chance of outside interference. After all my testing, we moved it over to a live server, set up a VLAN for cloning, set up the server in a VM on the open source [url="https://xenserver.org/"]XenServer[/url].

And the previous times I mentioned, 15 minutes - that was from start to finish - remoting in to server to set parameters, starting all laptops in PXE mode, starting multicast, and imaging, until power down and reboot of all laptops. Of course the 15 minutes did not include actual staging of hardware for the imaging - but that is obviously subjective, based upon your hardware and target count.

The most I ever did was the 43, via the switch, but afterwards, it was a piece of cake to image as many or as few laptops as we wanted to at any given time.

Mardok45 wrote:
I'd find it surprising if dd performed just as fast as partclone. Even if it was just as fast, you're restricted to disks or partitions that are equal or greater in size than the source. The advantage with partclone is you can shrink the filesystem to its minimal size and be able to clone to disks that have that's smaller (as long as it has enough room for the data on the partition).

Also, use lzop for compression. gzip is too damned slow and doesn't save you that much bandwidth compared to the speed and compression speed compared to lzop.


Me too. And yeah, LZOp FTW.
_________________
desultory wrote:
If you want to retain credibility as a functional adult; when you are told that you are acting boorishly, the correct response is to consider that possibility and act accordingly to correct that behavior.


Amen.
Back to top
View user's profile Send private message
patrix_neo
Guru
Guru


Joined: 08 Jan 2004
Posts: 468
Location: The Maldives

PostPosted: Fri Jul 20, 2018 8:09 pm    Post subject: Reply with quote

Mardok45 wrote:
Using an example file:

Code:

$ dd if=/dev/zero of=test.img bs=512 count=10000
$ dd if=/dev/zero of=test2.img bs=512 count=10000
$ fdisk test.img
...
$ fdisk test.img -l
Disk test.img: 4.9 MiB, 5120000 bytes, 10000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x508f0998

Device     Boot Start   End Sectors  Size Id Type
test.img1        2048  4096    2049    1M 83 Linux
test.img2        6144  9999    3856  1.9M 83 Linux


The "disk" in this case has a 512 block size and the boot partition ends at block 4096. So:

Code:
dd if=test.img bs=512 count=4096 of=test2.img


If your disk doesn't have a boot partition, just dd MBR/GPT portion of the disk, which is everything before the start of the first partition. So if we wanted to just dd the MBR in this case, it would be:
Code:
dd if=test.img bs=512 count=2048 of=test2.img


Now use partclone to clone the OS partition.

All of this is predicated on all the disks being the same. If you have a disk that is a different size, you'll have to take a different approach. I've done it by using sfdisk to create partitions to their needed size, and before creating a clone, shrink the disk partition to the smallest size possible so it'll fit.


Just a thing that confuse me ( obviously some have to ). test.img gets bootable MBR here? If true, that is pure magic!
I have a flamsible know how of that the boot sector is at the first instant of the disc. 4096 bytes in? Why is that make it bootable with /dev/zero? If I have read this correctly...
EDIT : Sorry, you use fdisk to make it bootable. So good!
The last dd command, I am glad you gave me that. Gentoo on all computers at work. I now can have all the tools ready! :)
I hate to admit it, you have good qualities to be a good systemadministrator. Thank you for the lecture. Gave me a lot.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Off the Wall All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum