migrating to a new HDD

Message

Uncle_Psychosis · Post by **Uncle_Psychosis** » Tue Dec 28, 2010 12:12 am

Hi guys

The time has come to upgrade to a bigger, hopefully faster hard drive.

I have had my current gentoo install for a good five years so I'm a little bit nervous about what I need to do to move to a new hard drive. Do any of you have any good tips for doing so? Should I just follow the install guide again or is there an easy way to copy all my stuff to a new (and not necessarily the same) file system layout?

Also, what file systems would you recommend for / and /home? (I put them on separate partitions so they don't necessarily have to be the same)

Cheers

Sam

Post by **NeddySeagoon** » Tue Dec 28, 2010 12:31 am

Uncle_Psychosis,

Make a mount point for your new drive, say /mnt/new
partition it to your liking and make filesystems. I suggest ext4 but rieser and ext3 work too.

Assemble the partitions into a filesystem tree. under /mnt/new
Copy everything over. This is best done with a liveCD as you must not copy /dev /proc, /sys and there is no point in copying /tmp

You can use rsync, cp -a, tar etc for the copy.

After the copy install grub onto the MBR. What you do next depends what you will do with the old drive.
You may need to change /etc/fstab and grub.conf to get the new drive to boot.

Oh, be sure your kernel supports any new filesystems you have elected to use.

d2_racing · Post by **d2_racing** » Tue Dec 28, 2010 4:41 am

Hi, I use rsync all the time to backup or migrate my stuff.

I use this command :

Code: Select all

# time rsync -aHA --del --force --stats --progress /mnt/backup/gentoo/ /mnt/gentoo

Where /mnt/backup/gentoo is my old hdd and /mnt/gentoo is my new hdd.

Clean and simple.

Uncle_Psychosis · Post by **Uncle_Psychosis** » Tue Dec 28, 2010 8:50 am

Ahha...now that seems like the kind of thing I need!

The only problem I can forsee is that I also need to put windows onto the new drive. My plan is to:

1. remove exisiting drives
2. plug in new one
3. partition new drive
4. install windows
5. plug in existing drives again
6. boot from live cd
7. copy my gentoo root folder to the new drive using rsync (thanks for the command)
8. install grub on the new disk and modify fstab as necessary
9. boot into new system
10. sort out the horrible mess that is my existing disks...

Also, I have a question about swap space. How much do you actually need for a modern PC? I've got 8Gb of RAM---do I really need to waste a partition on swap?

XQYZ · Post by **XQYZ** » Tue Dec 28, 2010 12:56 pm

Uncle_Psychosis wrote:Also, I have a question about swap space. How much do you actually need for a modern PC? I've got 8Gb of RAM---do I really need to waste a partition on swap?

Hibernation would be an argument. also: can you really not spare the 8gb on your hard drive for a swap partition?

Post by **NeddySeagoon** » Tue Dec 28, 2010 1:06 pm

Uncle_Psychosis,

If you need more than 512Mb for swapping your really don't have enough RAM.
However, swap a little larger than your RAM is needed for hibernation, if thats important to you.

You should have a swap, as not having swap space does not stop the kernel from swapping, you just disable one of the mechanisims for swapping that is available to the kernel.

Without swap, there is no home on disc for dynamically allocated RAM. The kernel will still flush/reload code and data that belongs to a disk file when RAM is short. Thats a form of swapping.
Note that code/data that have a real home in a disk file are never written to swap.

Fitzcarraldo · Post by **Fitzcarraldo** » Tue Dec 28, 2010 4:36 pm

NeddySeagoon wrote:However, swap a little larger than your RAM is needed for hibernation, if thats important to you.

I've read in the past that swap can be slightly smaller, not larger, than your RAM if you want Linux to be able to hibernate. The reason for this was stated as being because the contents of RAM are compressed when saved to disk.

Does anyone have a definitive reference for the amount of swap required for hibernation (suspend-to-disk)?

XQYZ · Post by **XQYZ** » Tue Dec 28, 2010 5:01 pm

AFAIK at least the default configuration of tuxonice with hibernate-scripts doesn't compress the ram when it's written to disk. I'm not sure compression would make that much sense anyway as it takes quite a bit to do it and most of the time you wouldn't even get a good ratio out of it. Also I don't think you really can give an accurate number for this. I mean the safest bet is make it at least as big as your ram is, so even if it's not compressible at all, it still works. I mean personally I'd hate it if it wouldn't fit at some point just because I'm out of luck.

http://tuxonice.net/ could have some info on it, but apparently you need to register yourself first for wiki access.

Fitzcarraldo · Post by **Fitzcarraldo** » Tue Dec 28, 2010 8:20 pm

Had a hunt for the post I remembered reading, and found it in KernelTrap: http://kerneltrap.org/node/16317#comment-303704

Anonymous (not verified) on June 23, 2008 - 5:10pm
Isn't swap necessary for Suspend To Disk/Hibernate? And if so, how much?

Anonymous (not verified) on June 24, 2008 - 5:45am
I think for suspend to disk you need a storage space an large as your ram.

Anonymous (not verified) on June 24, 2008 - 12:57pm
suspend-to-disk uses compressed images, so you can do with less swap than RAM (if it compresses well). It also only stores actual program images, so caches are likely to get evicted, the kernel is mostly unloaded, etc.

Is that actually the case?

EDIT (21:03 GMT): Just found this post from TuxOnIce developer Nigel Cunningham (http://lists.tuxonice.net/pipermail/tux ... 05423.html):

Nigel Cunningham wrote:> * How the installer of a Linux distribution should decide on the size of
> the hibernation file/partition? Worst case seems to be the installed
> memory size + swap partition but given that LZO compresses the image
> well enough is it possible to assume a size for the image file based on
> some proportions or something else?

LZO compression almost always achieves greater than 50% compression, so
you should be able to fit all images in swapspace equal to half the
amount of memory you have. Of course you might want to add a margin for
normal swapping and variations in the compression that's achieved.

So it seems that making swap the same size as RAM is more than enough.

Fitzcarraldo · Post by **Fitzcarraldo** » Wed Dec 29, 2010 4:40 pm

I did a little experiment to see what TuxOnIce does.

Here are my RAM and swap sizes in bytes:

Code: Select all

# free -bt
             total       used       free     shared    buffers     cached
Mem:    4142669824 2399014912 1743654912          0  175312896 1153126400
-/+ buffers/cache: 1070575616 3072094208
Swap:   4293558272          0 4293558272
Total:  8436228096 2399014912 6037213184

My (2.6.36) kernel config was as follows w.r.t. LZO compression:

Code: Select all

$ cat /usr/src/linux/.config | grep LZO
CONFIG_HAVE_KERNEL_LZO=y
# CONFIG_KERNEL_LZO is not set  <---- Kernel Compression Mode was selected to be LZMA rather than LZO
CONFIG_RD_LZO=y
CONFIG_JFFS2_LZO=y
# CONFIG_JFFS2_CMODE_FAVOURLZO is not set
CONFIG_UBIFS_FS_LZO=y
CONFIG_SQUASHFS_LZO=y
CONFIG_CRYPTO_LZO=m
CONFIG_LZO_COMPRESS=m
CONFIG_LZO_DECOMPRESS=y
CONFIG_DECOMPRESS_LZO=y

and TuxOnIce did not work when I tried to hibernate my laptop:

Code: Select all

Dec 29 12:01:06 meshedgedx kernel: [  272.829994] TuxOnIce: Failed to initialise the lzo compression transform.
Dec 29 12:01:06 meshedgedx kernel: [  272.829995] Failed to initialise compression algorithm.
Dec 29 12:01:06 meshedgedx kernel: [  272.829996] Continuing without compressing the image.
.
.
.
Dec 29 12:01:06 meshedgedx kernel: [  316.908327] TuxOnIce debugging info:
Dec 29 12:01:06 meshedgedx kernel: [  316.908330] - TuxOnIce core  : 3.2-rc2
Dec 29 12:01:06 meshedgedx kernel: [  316.908331] - Kernel Version : 2.6.36-sabayon
Dec 29 12:01:06 meshedgedx kernel: [  316.908333] - Compiler vers. : 4.4
Dec 29 12:01:06 meshedgedx kernel: [  316.908335] - Attempt number : 1
Dec 29 12:01:06 meshedgedx kernel: [  316.908336] - Parameters     : 32777 667648 0 0 0 5
Dec 29 12:01:06 meshedgedx kernel: [  316.908338] - Overall expected compression percentage: 0.
Dec 29 12:01:06 meshedgedx kernel: [  316.908340] - Checksum method is 'md4'.
Dec 29 12:01:06 meshedgedx kernel: [  316.908344]   0 pages resaved in atomic copy.
Dec 29 12:01:06 meshedgedx kernel: [  316.908345] - Block I/O active.
Dec 29 12:01:06 meshedgedx kernel: [  316.908346]   Used 0 pages from swap on /dev/sda5.
Dec 29 12:01:06 meshedgedx kernel: [  316.908347] - Max outstanding reads 1052. Max writes 20353.
Dec 29 12:01:06 meshedgedx kernel: [  316.908348]   Memory_needed: 1024 x (4096 + 344 + 112) = 4661248 bytes.
Dec 29 12:01:06 meshedgedx kernel: [  316.908349]   Free mem throttle point reached 0.
Dec 29 12:01:06 meshedgedx kernel: [  316.908350] - Swap Allocator enabled.
Dec 29 12:01:06 meshedgedx kernel: [  316.908351]   Swap available for image: 1048232 pages.
Dec 29 12:01:06 meshedgedx kernel: [  316.908352] - File Allocator active.
Dec 29 12:01:06 meshedgedx kernel: [  316.908352]   Storage available for image: 0 pages.
Dec 29 12:01:06 meshedgedx kernel: [  316.908353] - I/O speed: Write 54180 KB/s.
Dec 29 12:01:06 meshedgedx kernel: [  316.908354] - Extra pages    : 5179 used/500.
Dec 29 12:01:06 meshedgedx kernel: [  316.908355] - Result         : Hibernation was aborted.
Dec 29 12:01:06 meshedgedx kernel: [  316.908356]                  : Insufficient storage was available.
Dec 29 12:01:06 meshedgedx kernel: [  316.908357]                  : I/O errors were encountered.

As you can see above, with LZO compression configured to be a module instead of in-kernel, TuxOnIce did not compress RAM, and the 4293558272 bytes of swap was insufficient for hibernation when I have 4142669824 bytes RAM (of which 2399014912 was in use). So I used the kernel configuration options specified in the Gentoo Wiki article on TuxOnIce, and rebuilt my kernel to perform in-kernel LZO compression:

Code: Select all

$ cat /usr/src/linux/.config | grep LZO
CONFIG_HAVE_KERNEL_LZO=y
# CONFIG_KERNEL_LZO is not set  <---- I left Kernel Compression Mode as LZMA rather than LZO
CONFIG_RD_LZO=y
CONFIG_JFFS2_LZO=y
# CONFIG_JFFS2_CMODE_FAVOURLZO is not set
CONFIG_UBIFS_FS_LZO=y
CONFIG_SQUASHFS_LZO=y
CONFIG_CRYPTO_LZO=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_DECOMPRESS_LZO=y

and now the laptop hibernates successfully:

Code: Select all

Dec 29 15:22:45 meshedgedx kernel: [ 1205.711101] TuxOnIce debugging info:
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711102] - TuxOnIce core  : 3.2-rc2
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711103] - Kernel Version : 2.6.36-sabayon
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711103] - Compiler vers. : 4.4
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711104] - Attempt number : 1
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711104] - Parameters     : 0 667648 0 0 0 5
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711105] - Overall expected compression percentage: 0.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711105] - Checksum method is 'md4'.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711106]   0 pages resaved in atomic copy.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711107] - Compressor is 'lzo'.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711107]   Compressed 2180071424 bytes into 914681193 (58 percent compression).
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711108] - Block I/O active.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711108]   Used 225238 pages from swap on /dev/sda5.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711109] - Max outstanding reads 1338. Max writes 10377.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711110]   Memory_needed: 1024 x (4096 + 344 + 112) = 4661248 bytes.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711110]   Free mem throttle point reached 0.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711111] - Swap Allocator enabled.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711112]   Swap available for image: 1048232 pages.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711112] - File Allocator active.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711113]   Storage available for image: 0 pages.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711113] - I/O speed: Write 130 MB/s, Read 153 MB/s.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711114] - Extra pages    : 5214 used/500.
Dec 29 15:22:45 meshedgedx kernel: [ 1205.711115] - Result         : Succeeded.

If I understand the above TuxOnIce output correctly, with in-kernel LZO compression enabled my hibernating laptop with 3950 KB RAM occupied 225238/1048232 pages of swap = 22 per cent of my laptop's 4094 KB swap. The amount of swap used for hibernation was 225238x4096/4142669824 = 23 per cent of my laptop's RAM.

So, making the swap size the same as the RAM size is more than enough. There is certainly no need to make the swap size larger than the RAM size, and one could get away with making the swap size smaller than the RAM size.

Uncle_Psychosis · Post by **Uncle_Psychosis** » Thu Jan 27, 2011 5:47 pm

d2_racing wrote:Hi, I use rsync all the time to backup or migrate my stuff.

I use this command :
Code: Select all
# time rsync -aHA --del --force --stats --progress /mnt/backup/gentoo/ /mnt/gentoo 
Where /mnt/backup/gentoo is my old hdd and /mnt/gentoo is my new hdd.

Clean and simple.

Right, I'm finally getting round to doing this. Will your command copy everything thats under / ?

Thanks

Sam

d2_racing · Post by **d2_racing** » Sun Jan 30, 2011 4:53 am

Uncle_Psychosis · Post by **Uncle_Psychosis** » Sun Mar 13, 2011 3:40 pm

Right, I've finally found the time to do this.

d2_racing, I'm going to use your command to copy my / partition to the new drive.

However, I currently have /home on its own partition and I dont want to back that up too---I want to back that up separately (plan is to partition the new drive the same as the old one, only each partition will be bigger)

Thanks

Sam

d2_racing · Post by **d2_racing** » Sun Mar 13, 2011 4:04 pm

Use the switch exclude then :

--exclude /home'

davidm · Post by **davidm** » Sun Mar 13, 2011 4:06 pm

Fitzcarraldo wrote:Had a hunt for the post I remembered reading, and found it in KernelTrap: http://kerneltrap.org/node/16317#comment-303704
So it seems that making swap the same size as RAM is more than enough.

Also see:

http://www.mjmwired.net/kernel/Document ... erface.txt

50 /sys/power/image_size controls the size of the image created by
51 the suspend-to-disk mechanism. It can be written a string
52 representing a non-negative integer that will be used as an upper
53 limit of the image size, in bytes. The suspend-to-disk mechanism will
54 do its best to ensure the image size will not exceed that number. However,
55 if this turns out to be impossible, it will try to suspend anyway using the
56 smallest image possible. In particular, if "0" is written to this file, the
57 suspend image will be as small as possible.
58
59 Reading from this file will display the current image size limit, which
60 is set to 500 MB by default.
61

Note: On mine it actually seems to be set to "847683584" by default so I think the default might have changed or there is some other process at work which I am not aware of that sets it. My RAM is 2 GB and I have also created a 2 GB swap.

Uncle_Psychosis · Post by **Uncle_Psychosis** » Tue Mar 15, 2011 9:40 pm

Just out of interest... can I do this with a windows partition? Or does windows actually need to be installed "properly"?

Post by Hu » Wed Mar 16, 2011 2:00 am

Copying an NTFS volume using rsync may not produce the desired results. If you need to copy the entire filesystem, use ntfsclone for a filesystem-aware copy or dd for a bit-for-bit copy. NTFS volumes can be resized using ntfsresize if needed.

d2_racing · Post by **d2_racing** » Wed Mar 16, 2011 2:21 am

I think that partimage can do the trick too.

cach0rr0 · Post by **cach0rr0** » Wed Mar 16, 2011 4:49 am

hey neat, ntfsclone even handles junctions (think symlinks) correctly.
trick ive had to use in the past for clients sending me their db backup with the drive hard-coded (MSSQL)

migrating to a new HDD

migrating to a new HDD

Size of swap if you want the PC to be able to hibernate

Size of swap for hibernation

Size of swap required for hibernation

Re: Size of swap for hibernation