Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Hard disk duplication
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
tecknojunky
Veteran
Veteran


Joined: 19 Oct 2002
Posts: 1937
Location: Montréal

PostPosted: Wed Aug 06, 2003 5:44 pm    Post subject: Hard disk duplication Reply with quote

Tight space? Need to transfer your system onto a new bigger hard drive?

These 5 easy and safe steps will transfer the data on the current hard drive onto a new bigger one. The last partition will hold the newly aquired free space.

I drawn this from my own setup which is based upon the x86 installation howto from Gentoo. My hard drive was partition this way:

- /dev/hda1 : ext2 boot partition
- /dev/hda2 : swap partition
- /dev/hda3 : reiserfs root partition.

Also, I have a dvd-rom on /dev/hdb and the new disk was the primary drive on the second controler: /dev/hdc. Where is located the new disk is irrelevant. Just be sure you know that you are accessing the new drive and not another disk that hold crucial data!!!

These steps insure that your hda is untouched and will serve as a backup until you are sure the new drive is fonctionnal.

Prerequisites: make sure you have the tools for each of the file systems on the drive. Also, a partition program is essential (fdisk, cfdisk, ...).

1- Tranfer the data.

Code:
# dd if=/dev/hda of=/dev/hdc


This copies integraly the whole drive onto the new one, including the MBR and partition table. It can take a very long time depending on bus speed and drive size.

2- Fix the partition table.

Code:
# cfdisk /dev/hdc


You can use any disk partition program you wish.

You will see that on the new drive, you'll have the same partition table with additionnal free space trailing it.

Make the boot partitition bootable if it is not so.

Delete the last partition and recreate it using the whole free space. Write the partition table. This only change the partition table, not the data in the partition.

3- Resize the last partition.

Depending on the type of file system you had, use the proper tool to resize it. I had reiserfs:

Code:
# resize_reiserfs /dev/hdc3


4- Check file system integrity

Code:
# reiserfsck --fix-fixable /dev/hdc3
# e2fsck /dev/hdc1


5- Swap the drive and reboot.

Working? Congrat, you succefully duplicated your disk.

This is based on my setup. I'm guessing other setups may fail with this. If so, post your unsuccess here (with preferably a fix).
_________________
(7 of 9) Installing star-trek/species-8.4.7.2::talax.
Back to top
View user's profile Send private message
nsahoo
l33t
l33t


Joined: 17 Jul 2003
Posts: 618

PostPosted: Sat Aug 09, 2003 4:15 am    Post subject: Reply with quote

how did you know that I needed this tutorial ? ;)

thanks.
Back to top
View user's profile Send private message
nsahoo
l33t
l33t


Joined: 17 Jul 2003
Posts: 618

PostPosted: Sat Aug 09, 2003 5:41 am    Post subject: Re: Hard disk duplication Reply with quote

tecknojunky wrote:


5- Swap the drive and reboot.



do you mean i'll have to change the jumper settings (primary/secondary/cable) of the hard disks?
Back to top
View user's profile Send private message
nsahoo
l33t
l33t


Joined: 17 Jul 2003
Posts: 618

PostPosted: Sat Aug 09, 2003 5:52 am    Post subject: Re: Hard disk duplication Reply with quote

tecknojunky wrote:


1- Tranfer the data.

Code:
# dd if=/dev/hda of=/dev/hdc


This copies integraly the whole drive onto the new one, including the MBR and partition table. It can take a very long time depending on bus speed and drive size.


Does it copy the free space in /dev/hda too ?
Back to top
View user's profile Send private message
hertog
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jun 2002
Posts: 138
Location: Enschede/The Netherlands/Europe

PostPosted: Sat Aug 09, 2003 10:43 am    Post subject: Reply with quote

You seem to have missed this excelent scipt:

https://forums.gentoo.org/viewtopic.php?t=28123&highlight=backup
Back to top
View user's profile Send private message
nsahoo
l33t
l33t


Joined: 17 Jul 2003
Posts: 618

PostPosted: Sat Aug 09, 2003 3:54 pm    Post subject: Reply with quote

i finally did it .. woohooo .. :D
Back to top
View user's profile Send private message
hawson
n00b
n00b


Joined: 27 Sep 2002
Posts: 16

PostPosted: Sun Aug 10, 2003 1:25 am    Post subject: I wouldn't do it this way.... Reply with quote

I have to take issue with a few thing in this method of duplicating disks. Having done numerous disk replications and migrations, I have to say that following the instructions above is a Bad Idea.

    First of all, using 'dd' to copy data is fine ONLY when the hard disks are absolutely identical. Minor disk revisions don't cut it; they must be exactly the same.

  1. Your suggestion of using
    Quote:
    Code:
    dd if=/dev/hda of=/dev/hdc
    is incorrect, except in the case of identical hardware. As mentioned in an earlier post, this will also copy the partition table, so in this caseyou don't need the extra step of partitioning the new disk (see below for more details...). The biggest problem with this method is that it will not properly configure the new disk if it has a different geometry. For example, given that hda is 40GB and hdc is 80GB, running
    Code:
    dd if=/dev/hda of=/dev/hdc
    will copy only the 40GB of data on the source disk, and leave the remaining 40GB untouched, unformated, and unusable. Furthermore, since the disk geometry is necessarily different, I would be wary of the filesystem on the news disk. Essentially, using this method gives you an under-utilized, potentially unstable new hard disk.

  2. On a performance note, the default settings for 'dd' are to use 512byte blocks for data transfer. This is going to take a loooong time on any file larger than a few MBs. Adding the 'bs=' option will help. Something like:
    Code:
    dd if=/dev/hda of=/dev/hdc bs=2048k
    will speed things up quite a bit. There is some arcane magic that goes into finding the optimal block size, but some large number usually suffices (not really, but you can look it up on Google yourself :D)

  3. When I have to clone disks, I usually do something like this (assuming source is hda, dest is hdc):
    1. Ideally, boot off of a floppy disk or CDROM. You want the source disk to completely quiescent while you do this. Tomsrtbt works quite well for this, as would Knoppix, or a Gentoo CD. You can also do this from single user mode, after making absolutely sure you have shutdown everything you possibly can (all you need running, really, is a shell).

    2. Take a look at the source partition table for reference:
      Code:
      fdisk -l /dev/hda


    3. Parititon the new disk to taste. If you are shrinking any partitions on the new disk, make sure that you don't inadvertently make any of them too small to hold the source data.
      Code:
      fdisk /dev/hdc


    4. Mount the new paritions so they are in the desired arrangement. Given something like this for the sake of example: hda1 == /boot, hda2 == /, hda3 == /var, hda5 == /home, hda5 == swap, and the partitions on hdc are in the same arrangement and mount points, I would do the following:

      Code:
      mkdir /mnt/newdisk
      mount /dev/hdc2 /mnt/newdisk
      mkdir /mnt/newdisk/boot
      mkdir /mnt/newdisk/var
      mkdir /mnt/newdisk/home
      mount /dev/hdc1 /mnt/newdisk/boot
      mount /dev/hdc3 /mnt/newdisk/var
      mount /dev/hdc5 /mnt/newdisk/home


    5. You should now be ready to actually copy the data. I use tar, but cpio or would work just as well (if not better). The one snag here is that you need to copy only stuff on the source disk, you don't want to recursively copy the data in the /mnt/newdisk directory, nor things like /proc that don't really exist ;-). These commands usually work:
      Code:
      cd /
      tar cpf - bin boot dev etc home lib opt root sbin tmp usr var | ( cd /mnt/newdisk;   tar xpvf -)

      This will take a long time, and every file copied should scroll by. Be sure that you use the 'p' flag to tar, lest you loose all of your file ownership information. Needless to say, this must be done as root.

      If you use devfs, then you can omit the /dev/ directory from the tar invocation. You should also add any other top-level directories that are needed, and remove any that may not be approriate (such as /tmp and /var/tmp for example).

      A nice side effect of using tar/cpio instead of dd is that it should defragment the disk somewhat. 8)

    6. At this point, you should create any directories that you omitted in the last step, as well as any other initialization steps that need to be done--such as running 'mkswap' on your swap partitions.
      Code:
      cd /mnt/newdisk
      mkdir /proc
      mkswap /dev/hdc5


    7. I'm paranoid, so I reinstall the boot loader at this point. When you do so, make sure that you reinstall to hdc, and not hda. Check the LILO/GRUB docs for specifics on this.

    8. Finally, unmount the new disk, poweroff, swap the disks, and start everything up. With some luck, it will work the first time. If you are feeling adventurous, you can play clever tricks with kernel root= options, /etc/fstab to boot off of /dev/hdc before you swap disks. However, if you are willing to try that, you don't need my help. :-)
    .

  4. There are a *LOT* of scripts out there that take care of this for you, and I'd recommend looking at one of them if you don't feel comfortable with doing it manually. If you have to do a lot of hard disks, say for a computer lab, check out Dolly. It creates a "ring" (of sorts) between all of the hosts sending data around. SourceBox --> HostA --> HostB -->HostC --> .... --> SourceBox. It's quite nice once it gets going. You can even use it to clone non-Linux systems.


Good luck!
Back to top
View user's profile Send private message
nsahoo
l33t
l33t


Joined: 17 Jul 2003
Posts: 618

PostPosted: Sun Aug 10, 2003 2:58 am    Post subject: Re: I wouldn't do it this way.... Reply with quote

hawson wrote:

The biggest problem with this method is that it will not properly configure the new disk if it has a different geometry. For example, given that hda is 40GB and hdc is 80GB, running
Code:
dd if=/dev/hda of=/dev/hdc
will copy only the 40GB of data on the source disk, and leave the remaining 40GB untouched, unformated, and unusable. Furthermore, since the disk geometry is necessarily different, I would be wary of the filesystem on the news disk. Essentially, using this method gives you an under-utilized, potentially unstable new hard disk.


We are deleting the last partition and making a new partition in its place including the free space.. or else we can just create a new partition using the free space. whats wrong with that ?

Can you please clarify why the disk is going to be unstable ?
Back to top
View user's profile Send private message
hawson
n00b
n00b


Joined: 27 Sep 2002
Posts: 16

PostPosted: Sun Aug 10, 2003 6:24 am    Post subject: Reply with quote

Quote:
We are deleting the last partition and making a new partition in its place including the free space.. or else we can just create a new partition using the free space. whats wrong with that ?


My apologies, I missed that part. However, I'm not sure it matters...see below.
Quote:
Can you please clarify why the disk is going to be unstable ?

Well, based on doing the following on different sized drives:
Code:
dd if=/dev/hda of=/dev/hdc


That can potentially screw things up if the hardware doesn't have the same geometry.

When you have a larger destination drive than source disk, you will only copy as much is on the source drive. Let's say you have a 20GB source disk, and 60GB destination disk. The 'dd' command will only copy 20GB to the destination disk, leaving some 40GB uninitalized and unused. If you do manage to get a valid partition table out of this, you will have to manually create new partitions to make use of the remaining space (which, to be fair, is what you are planning on doing). I think you can get overlapping partitions if you do this as well--again, not something you want to do unless you really know what you are doing.

If the source disk is larger than the destination disk (60GB copied to 20GB for example), you have more insidious problems. The 'dd' command will likely complete without error, although if you check the 'XX records in / XX records out' message, you might notice a discrepancy (I know I sure wouldn't). In this case, the first 20GB will be copied, but the remaining 40GB of raw disk space will be silently dropped. It is quite possible to use a a drive built like this...for a short while. Sooner or later you are going to try and access something that exists beyond that that 20GB limit, and there will lots of interesting kernel errors in your logs. :-) Bear in mind that most filesystems spread their data out across the available space for various reasons (performance, fragmentation avoidance, etc). Thus, even though you may only have 5GB of data stored on that 60GB partition, it will likely be spread out over all of the availabe 60GB of space.

(FWIW, I've tested both of these, and neither has worked. The smaller --> larger copy gave me a screwed up partition table, and the larger --> smaller copy first silently lost data, then threw a ton of errors. However, I didn't expect either to actually work, nor should they. :-))

Furthermore, since the drive geometries are going to be different, the partition boundries are going to be completely out of whack, and may not even make sense for the new drive. I will admit to not knowing the hard-core details for partition tables, but I believe that partitions are defined as offsets in terms of either cylinders or sectors, relative to the geometry of a particular disk. So, 100 cylinders on one disk usually is not the same as 100 cylinders on a second disk. Thus, when you copy the partition table using 'dd', it's not going to work. So, even if you do nuke and remake the last partition on the disk (hdc3), it may work, but the rest likely will not, again, because the geometries won't match the partition table.

It's also a bad idea to fdisk partitions with data on them (no, I don't trust partition resizers very much).

Anyhow, there are better ways to clone disks than 'dd' for different hardware. One final advantage to using tar or cpio is that you will only copy what you care about, and not gigabytes of empty space. 5GB used of a 20GB partition is going to waste 3/4 of the time shoving around bits you don't care about in the first place.

So, all of that said, I have used
Code:
dd if=/dev/hda | nc 192.168.1.1 1234
and
Code:
nc -l -p 1234 | dd of=/dev/hda
to shove disk images over the network though. :-)
Back to top
View user's profile Send private message
Garbz
Apprentice
Apprentice


Joined: 02 Jul 2003
Posts: 260
Location: Brisbane, Australia

PostPosted: Mon Aug 11, 2003 10:44 pm    Post subject: Reply with quote

Hey can i get some input on how i would do this to a windows NTFS / FAT32 drive.

Yesterday i was put into a partiton where we are going to upgrade dad's 10gb drive to a shiny new 40gb (well not new but shiny all the same :lol: )

Anyway the layout is:
/dev/hda1 NTFS 4gb
/dev/hda2 FAT32 6gb

/dev/hdc Just been lowleveled.

The problem is i don't want to keep the partition sizes. Is it even possible to end up with:

/dev/hdc1 NTFS 7gb
/dev/hdc2 FAT32 13gb without re-formatting ? I hate installing windows, and there sure as well isn't any easy way to back the bastard up.
_________________
Every begining is another begining's end.
Back to top
View user's profile Send private message
really
Guru
Guru


Joined: 27 Aug 2002
Posts: 430
Location: nowhere

PostPosted: Mon Jul 05, 2004 7:49 pm    Post subject: Reply with quote

I agree that is is bad to just dd a whole disk onto another one. Youll be copying the unnecesary 0's too and wont have that little defragment usual cp'ing whould cause.

I did it like this...
Old drive, 80gig. New drive 160gig.

Becouse i am going to use the 80 drive for the os and my home, and the 160 gig one only for mp3s movies and such things.
I creaed four primary partitions with cfdisk on it, sized, 20, 23, 40 and 80 gigs.
On my old drive i had music in the home directory under /home/listen which was about 60gigs and also contained lots of other files, well my user dir which is about 4gigs, the home partition also contained /home/ftp which was took about 20gigs itself...

making a filesystem on the hdb1-4 partitions (the 160gig one)
and creating /warez/music /warez/motion /warez/motion/cartoons /warez/other
mounting hda1 to music hdb2 to motion and so on, as you like.

Then just simply enough, cp -Rpi /home/listen/* /warez/music/ and so on...
I also cp'ed my user dir to teh otehr one (for defragment purposes and testing speed, then cp back over.
Did the same thing for /usr and others...

Works perfectly.
_________________
NoManNoProblem

Get lost before you get shot.
Back to top
View user's profile Send private message
really
Guru
Guru


Joined: 27 Aug 2002
Posts: 430
Location: nowhere

PostPosted: Mon Jul 05, 2004 7:49 pm    Post subject: Reply with quote

I agree that is is bad to just dd a whole disk onto another one. Youll be copying the unnecesary 0's too and wont have that little defragment usual cp'ing whould cause.

I did it like this...
Old drive, 80gig. New drive 160gig.

Becouse i am going to use the 80 drive for the os and my home, and the 160 gig one only for mp3s movies and such things.
I creaed four primary partitions with cfdisk on it, sized, 20, 23, 40 and 80 gigs.
On my old drive i had music in the home directory under /home/listen which was about 60gigs and also contained lots of other files, well my user dir which is about 4gigs, the home partition also contained /home/ftp which was took about 20gigs itself...

making a filesystem on the hdb1-4 partitions (the 160gig one)
and creating /warez/music /warez/motion /warez/motion/cartoons /warez/other
mounting hda1 to music hdb2 to motion and so on, as you like.

Then just simply enough, cp -Rpi /home/listen/* /warez/music/ and so on...
I also cp'ed my user dir to teh otehr one (for defragment purposes and testing speed, then cp back over.
Did the same thing for /usr and others...

Works perfectly.
_________________
NoManNoProblem

Get lost before you get shot.
Back to top
View user's profile Send private message
karnesky
Apprentice
Apprentice


Joined: 18 Mar 2004
Posts: 218

PostPosted: Tue Jul 06, 2004 12:01 am    Post subject: Reply with quote

Garbz wrote:
Hey can i get some input on how i would do this to a windows NTFS / FAT32 drive.
Easiest way is to use partimage, which is in portage, on knoppix, and on the sysrescuecd. Partimage is a Norton Ghost-like program that works with all of the major file systems. It isn't always the best method of disk cloning, but it is usually better than dd & is almost always easy.
Back to top
View user's profile Send private message
veezi
Apprentice
Apprentice


Joined: 10 Nov 2003
Posts: 226

PostPosted: Tue Jul 06, 2004 1:09 am    Post subject: Reply with quote

Garbz wrote:
Hey can i get some input on how i would do this to a windows NTFS / FAT32 drive.

Yesterday i was put into a partiton where we are going to upgrade dad's 10gb drive to a shiny new 40gb (well not new but shiny all the same :lol: )

Anyway the layout is:
/dev/hda1 NTFS 4gb
/dev/hda2 FAT32 6gb

/dev/hdc Just been lowleveled.

The problem is i don't want to keep the partition sizes. Is it even possible to end up with:

/dev/hdc1 NTFS 7gb
/dev/hdc2 FAT32 13gb without re-formatting ? I hate installing windows, and there sure as well isn't any easy way to back the bastard up.

The ntfsprogs package might be useful to you. Particularly the 'ntfsclone' and 'ntfsresize' tools.

NOTE: Where did you get that image under your name from? Look cool :)

Cheers,
Back to top
View user's profile Send private message
Garbz
Apprentice
Apprentice


Joined: 02 Jul 2003
Posts: 260
Location: Brisbane, Australia

PostPosted: Tue Jul 06, 2004 1:49 pm    Post subject: Reply with quote

hate to be a spoil sport but after 11 months i've actually installed my drives and got the computer up and running :roll:
_________________
Every begining is another begining's end.
Back to top
View user's profile Send private message
Garbz
Apprentice
Apprentice


Joined: 02 Jul 2003
Posts: 260
Location: Brisbane, Australia

PostPosted: Tue Jul 06, 2004 1:50 pm    Post subject: Reply with quote

hate to be a spoil sport but after 11 months i've actually installed my drives and got the computer up and running :roll:
_________________
Every begining is another begining's end.
Back to top
View user's profile Send private message
veezi
Apprentice
Apprentice


Joined: 10 Nov 2003
Posts: 226

PostPosted: Tue Jul 06, 2004 6:27 pm    Post subject: Reply with quote

Oooooops :oops: I did it again

Hey, since you're still alive after 11 months :D, What about your forum image? I like it, where did you get it from?

Now, you aren't going to wait 11 months before answering that just to teeze me right?! :D

Cheers,
Back to top
View user's profile Send private message
Garbz
Apprentice
Apprentice


Joined: 02 Jul 2003
Posts: 260
Location: Brisbane, Australia

PostPosted: Wed Jul 07, 2004 2:32 am    Post subject: Reply with quote

lol
all my forum images in various forums are from Azumanga Daioh anime characters.

I've used Chio and Osaka since my sister rekons they are just like me. Chio is smart, and osaka has her head in the clouds.

I've basically cropped and resized wallpapers. Google them to find some there's some really super ones out there.
_________________
Every begining is another begining's end.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum