best way to move gentoo . hdd -> ssd (solved)

[hedmo]

hi

i have read a lot about cloning , coping and making a stage X but what is the fastest and best way to switch my gentoo
installlation from hdd to ssd.

regards

[Buffoon]

Boot from external media, mount both HDD and SDD in temporary directories, use 'cp -a', chroot into your SDD and reinstall bootloader, fix everything where PARTUUID or filesystem UUID was used.

[Tony0945]

I had a thread on this recently. My problem was that I could not boot any rescue medium and I was also changing drive position. I solved it by putting both drives in an empty PC (the target actually) that could boot CD's.
For your case, if you have a rescue medium (like the Gentoo liveusb) on CD, DVD, or USBstick that you can boot, put both drives in that box (easiest if it is the target box), boot the rescue medium and mount both drives as say /mnt/src and /mnt/target. Any names will do. Create you partitions and filesystems as you would for a new install on the target drive if you haven't done so already. Then use "cp -ar" as root to copy the real filesystems (i.e. not /proc or /sys or /dev ...) but not symlinks! I found out that if you copy symlinks like /lib recursively as well as copying the target, then you will do a double copy.
If doing it as a remote, boot both systems to rescue media, start an rsync daemon on both, and use rsync. It's quicker and easier to monitor.

I wrote a shell script for the latter scenario but wound up doing the steps by hand. I'm bad at writing shell. Maybe you could fix it.
/usr/local/bin/cloneme

When switching to an SSD add a line to the root crontab:

Code: Select all

15 2 * * tue /sbin/fstrim -va |logger

Tuesday morning is convenient for me. Use whatever time is convenient for you.
fstrim is supplied by util-linux which is a dependency of portage (and many other packages) so you should have it.

EDIT:
cloning windows is easier because typically it's all on one drive partition and Microsoft provides a tool. Of course, you have no idea what kind of data mining the tol is doing nor what Windows itself is doing. I have a Win10 VM for TurboTax only. It uses a totally different LAN address so Turbotax can talk to the internet but nothing in the VM can see the other machines. When i need a file, I burn a CD and let the VM see the CDROM burner and also the USB printer (not the wireless printer).

I did an fstrim right away on the new SSD. Don't know if that is required or even advisable but I figured one extra trim can't hurt.

You might have to adjust /etc/fstab if you are booting by UUID or label and possibly by devname. Doesn't hurt to review it.
Since I was cloning to a machine with the same architecture, I also had to edit /etc/conf.d/hostname (OpenRC). I use fixed ipaddresses but I let the dhcp server assign them by MAC address so they are effectively fixed, but I do all my machines at one spot.

I was cloning to a machine with the same CPU family and you are cloning to the same machine. If one wanted to clone to a machine that wasn't binary compatible. It would be better to do a stage3 install, then copy /root, /home /etc/portage and the portage world file over and emerge -e @world

[hedmo]

Tony0945 and Buffoon

thank you for replying . i am going to start my box with gentoo livedvd 2008 and mount my hdd and sdd at mnt . gentoo-old and gentoo-new . what i have read is the problems with symlink and stuff about --excluding . this has got me confused and like to know the exact
command to not deal with problems .

[Tony0945]

In my script I excluded /home and /root because I was using it for two purposes - making a quick and dirty copy for a new machine, and for moving to a new drive like you. Also the symlinks are for profile 17.0
Not sure what needs to be changed for 17.1

[halcon]

Hi Tony0945,

I wrote a shell script for the latter scenario but wound up doing the steps by hand. I'm bad at writing shell. Maybe you could fix it.
/usr/local/bin/cloneme

I am not an expert in shell, but I have some comments.

Code: Select all

TARGET="999.999.999"

if [ "$#" -ne 1 ] ;
then
 echo "ERROR: use target ip address as a parameter"  && exit 1;
fi
TARGET=$1;

I would rewrite it as

Code: Select all

IP_ADDRESS="${1}"

if [[ -z "${IP_ADDRESS}" ]]; then
	echo "${0}: IP_ADDRESS argument not passed. Exiting 1." >&2
	exit 1
fi

Code: Select all

       scp -p -B -r /"$d" root@"$TARGET":/mnt/gentoo/
#        rsync -av    "$d" root@:"$TARGET":/mnt/gentoo/

1. Why do you prefer rsync to scp? The latter is vulnerable, are you aware about it?
2. I would never recommend to login remotely as root. A user login + rsync in sudoers would be much more safe.

[Hu]

Code: Select all

     2	

I suggest always starting with set -eu unless you have a good reason not to do so. These make the shell more strict by enabling the shell options errexit and nounset.

Code: Select all

     3	TARGET="999.999.999"

This is a dead assignment. If the user invoked the script properly, then TARGET is changed without being read. If the user invoked the script improperly, then the script exits and never reads TARGET. I would delete this line.

Code: Select all

     7	 echo "ERROR: use target ip address as a parameter"  && exit 1;

I agree with halcon. Redirect the error message to stderr (echo text >&2) and use ;, not && to chain to the exit. In the admittedly very unlikely case that the echo fails, your script as currently written would not exit. Consider:

Code: Select all

$ /bin/sh -c 'echo && exit 5'; echo $?

5
$ /bin/sh -c 'echo && exit 5' >/dev/full; echo $?
/bin/sh: line 0: echo: write error: No space left on device
1
$ /bin/sh -c 'echo && exit 5; exit 0' >/dev/full; echo $?
/bin/sh: line 0: echo: write error: No space left on device
0

Code: Select all

    10	       	set +x

set +x is the default if not set differently by prior statements (and this script has no such statements), or by the user invoking the shell (who, if he set -x, probably wants that, and turning it off is undesirable). I would delete this line.

Code: Select all

    15		echo "Copying $TRAGET"

This variable is misspelled, and so will expand to nothing. Under set -u, that failed expansion will become a fatal error.

As a style point, echoing the destination and not the directory name seems less useful to me than it could be. Did you mean to also echo $d in that line?

Code: Select all

    16	       scp -p -B -r /"$d" root@"$TARGET":/mnt/gentoo/
    17	#        rsync -av    "$d" root@:"$TARGET":/mnt/gentoo/

I would generally prefer rsync over scp. If I were using rsync, I would use more options than you have shown here, since -a does not enable all the options you likely want when the goal is a perfect copy.

Code: Select all

    20	ln -s @"$TARGET":/mnt/gentoo/lib64 @"$TARGET":/mnt/gentoo/lib

This looks very strange. What are you trying to do here?

Code: Select all

    23	for P in $(  dev proc sys tmp mnt root media )

This is wrong. dev is not a command. You probably want to remove the $( and ) here.

Code: Select all

    25	  	mkdir -p root@"$TARGET":/mnt/gentoo/"$P"
    27	chmod 0755 root@"$TARGET"/mnt/gentoo/home

As with the ln above, this does not make any sense. Is it just a reminder to the user to go run commands elsewhere?

I would rewrite it as

Code: Select all

IP_ADDRESS="${1}"

For completeness, I will note that while what you propose would work, it has two failure modes:

• It will not warn the user for attempting to pass too many arguments.
• It is not compatible with the set -u I recommend earlier in my post. Attempting to read $1 fails under set -u when there are no arguments:

  Code: Select all

  $ /bin/sh -c 'echo $#;echo 1=$1'
  0
  1=
  $ /bin/sh -u -c 'echo $#;echo 1=$1'
  0
  /bin/sh: $1: unbound variable

1. Why do you prefer rsync to scp? The latter is vulnerable, are you aware about it?

Are you referring to the scp glob vulnerability? If so, I think that is not applicable here since he is copying to the peer, not from it. Additionally, that vulnerability requires a server with unusual filesystem contents. For the case in question here, where the operator fully controls both systems, I think it is reasonable to assume the server will not try to trigger the glob expansion vulnerability.

2. I would never recommend to login remotely as root. A user login + rsync in sudoers would be much more safe.

I generally discourage remote root login, but for the case described here, it is much simpler to configure. If the login is restricted to key files (not passwords), and only well trusted systems have the private key, I think remote root is acceptable.

[hedmo]

i am following the handbook with :

Designing a partition scheme

i am going for this:

Code: Select all

mount /dev/sda1 /mnt/old
mount /dev/sdc1 /mnt/new
mkdir /mnt/new/boot
mount /dev/sdc2 /mnt/new/boot
cp -a /mnt/old/. /mnt/new

after that the handbook with :

Chrooting and Selecting a boot loader

[halcon]

I suggest always starting with set -eu unless you have a good reason not to do so. These make the shell more strict by enabling the shell options errexit and nounset.

For completeness, I will note that while what you propose would work, it has two failure modes:

• It will not warn the user for attempting to pass too many arguments.
• It is not compatible with the set -u I recommend earlier in my post. Attempting to read $1 fails under set -u when there are no arguments:

  Code: Select all

  $ /bin/sh -c 'echo $#;echo 1=$1'
  0
  1=
  $ /bin/sh -u -c 'echo $#;echo 1=$1'
  0
  /bin/sh: $1: unbound variable

Thanks for the reminder! I've been thinking about set -eu for a long time but never used it because worried many things would be broken. May be, indeed, it's time to just try - what will be broken and what won't
Also, maybe, set -e will help me to understand better how shell exiting works... (where to put || exit $?, and where not)

Are you referring to the scp glob vulnerability? If so, I think that is not applicable here since he is copying to the peer, not from it. Additionally, that vulnerability requires a server with unusual filesystem contents. For the case in question here, where the operator fully controls both systems, I think it is reasonable to assume the server will not try to trigger the glob expansion vulnerability.

Yes, I was referring to that. Okay then...

I generally discourage remote root login, but for the case described here, it is much simpler to configure. If the login is restricted to key files (not passwords), and only well trusted systems have the private key, I think remote root is acceptable.

Well... That's right

[figueroa]

Should always have a current backup. In a perfect world, restore from backup like backup created with the following script:

Code: Select all

#!/bin/sh
#stage4.scr
# Also note: https://wiki.archlinux.org/index.php/rsync#Full_system_backup
mount /mnt/backup0
cd /
date > /mnt/backup0/stage4date.txt
tar cpf /mnt/backup0/stage4.tar.zst -I "zstd -19 -T0" --acls --xattrs --numeric-owner --no-wildcards-match-slash -X /home/USERNAME/bin/exclude.master /
date >> /mnt/backup0/stage4date.txt
cd

The date lines are optional. ~/bin is my personal scripts directory.

Here is my exclude file:

Code: Select all

$ cat bin/exclude.master
dev/*
home/USERNAME/*
lost+found
media/*
mnt/*/*
proc/*
root/.cache/*
run/*
sys/*
tmp/*
var/tmp/*
var/cache/distfiles/*

When restoring the archive with tar, use the following switches: --acls --xattrs-include='*.*' --numeric-owner

My zstd stage 4 archive is 3.7 G.

This is tested and found to work, even with mounted partitions.

It does not backup or restore any personal files. It doesn't install a boot mechanism handle the EFI partition or MBR. Install your own boot stuff.

Adjust to match your USERNAME and layout. Disclaimers etc.

[Logicien]

Code: Select all

rsync -av --delete-after --progress --stats /source/directory/ /destination/directory/

[Goverp]

Having done this recently by taking a tar backup and restoring it to the new device, I found one issue - the dbus helper /usr/libexec/dbus-daemon-launch-helper program had lost its "suid" bit, which may be a result of me getting the tar command wrong. That meant a plethora of error messages on booting.

[tholin]

I just want to point out that there are some metadata that 'cp -a' won't preserve.

It won't keep shared extents of reflinked files. That means if files share extents those will be unshared and the copy will take more space. That can be a problem If you have reflinked copies of large disk images to save space.

Cp will not preserve file attributes (which should not be confused with extended attributes (xattr) which are preserved). File attributes are anything that can be set with 'chattr' including btrfs nocow, immutable flag, etc.

Some metadata can't be preserved by any tool because it can't be set. That include inode number and the ctime & crtime timestamps. That shouldn't matter but some backup programs might use that metadata to detect unmodified files and will therefore re-read all data after the migration.

[hedmo]

thank you all for the help.

[Tony0945]

Post how it all worked, And ask for help if there is a glitch.

Hu,
Thanks for that detailed critique.!I'm going to start another topic on that, quoting your response in full. It deserves comment and exploration of questions. You obviously spent considerable time and it should be explored fully. Perhaps in "tips and tricks"? It's Monday, my daughter's internet is out and there is lots of non-gentoo stuff to do. But I do earnestly want to get back to this.

[hedmo]

Post how it all worked, And ask for help if there is a glitch.

at the moment all works super and swished to ssd made it super fast . i will reopen it if i found something.

regards

[halcon]

I've been thinking about set -eu for a long time but never used it because worried many things would be broken. May be, indeed, it's time to just try - what will be broken and what won't

I've tried. set -u is great, I like it!
But I haven't yet understood what set -e does... It does not exit the program if the error was checked? Evidently, I don't know how to handle exits correctly with set -x, and paid attention to that only now.
Two examples:

Code: Select all

LC_ALL=C /bin/bash -c 'set -e; source /usr/local/bin/mclass_utilities.sh; var=$(get_newest_file_or_dir "file") || exit $?; echo "var = ${var}"'
Wrong number of arguments: 1 instead of 5.

Exiting 1.

Code: Select all

LC_ALL=C /bin/bash -c 'set -e; set -x; source /usr/local/bin/mclass_utilities.sh; var=$(get_newest_file_or_dir "file") || exit $?; echo "var = ${var}"'
+ source /usr/local/bin/mclass_utilities.sh
++ get_newest_file_or_dir file
++ [[ 1 -ne 5 ]]
++ exit_err_1 'Wrong number of arguments: 1 instead of 5'
++ local '__arg_error=Wrong number of arguments: 1 instead of 5'
++ echo 'Wrong number of arguments: 1 instead of 5.

Exiting 1.'
Wrong number of arguments: 1 instead of 5.

Exiting 1.
++ exit 1
+ var=
+ exit 1

As it can be seen, without set -x, the script exits correctly, before echoing. But with set -x, echoing works.

The function get_newest_file_or_dir is in the file mclass_utilities.sh and it contains that code:

Code: Select all

	if [[ $# -ne 5 ]]; then
		exit_err_1 'Wrong number of arguments: '"$#"' instead of 5'
	fi

And exit_err_1 is:

Code: Select all

function exit_err_1 {
	local __arg_error="${1}"
	echo "${__arg_error}"'.

Exiting 1.' >&2
	exit 1
}

How to exit in case of errors with set -x?

[Hu]

You are misreading the output. set -x shows you that the variable was assigned an empty value, then your explicit exit happens. Your echo is not hit. You can see this more clearly if you rewrite your echo to look less like the output of set -x.

Generally, set -e is roughly equivalent to appending || exit $? in many places. There are select places where it is inactive, though. For example, it is not active while evaluating a statement to determine the truth of an if:

Code: Select all

set -e
if false; then
  echo Nothing
fi
echo but we made it here

[halcon]

You are misreading the output. set -x shows you that the variable was assigned an empty value, then your explicit exit happens. Your echo is not hit. You can see this more clearly if you rewrite your echo to look less like the output of set -x.

Indeed. I see now that set -x has nothing to do with this...
Here are another examples that I do not understand:

CODE 1

Code: Select all

#!/bin/bash

set -eu

function step_clean_boot {
	local __newest=$(get_newest_file_or_dir 'file' 'file') || exit $?
	echo
	ls -a /boot/ || exit $?
}

step_clean_boot

exit 0

OUTPUT 1

Code: Select all

LC_ALL=C kerneler_test.sh
/usr/local/bin/kerneler_test.sh: line 6: get_newest_file_or_dir: command not found

.  ..  .keep  grub

CODE 2

Code: Select all

#!/bin/bash

set -eu

source /usr/local/bin/mclass_utilities.sh

function step_clean_boot {
	local __newest=$(get_newest_file_or_dir 'file' 'file') || exit $?
	echo
	ls -a /boot/ || exit $?
}

step_clean_boot

exit 0

OUTPUT 2

Code: Select all

LC_ALL=C kerneler_test.sh
Wrong number of arguments: 2 instead of 5.

Exiting 1.

.  ..  .keep  grub

Why is ls being executed?

[Hu]

local succeeds, so there is no error to trigger the exit. Per info bash:

Code: Select all

The return status
is zero unless 'local' is used outside a function, an invalid NAME
is supplied, or NAME is a readonly variable.

Split the use of local and the assignment into separate statements:

Code: Select all

f() {
  local a
  a=$(expression_that_may_fail)
}

[halcon]

local succeeds, so there is no error to trigger the exit.

Ah! I fell for it again... (knew but forgot)
Thanks a lot!

[halcon]

I suggest always starting with set -eu unless you have a good reason not to do so.

Found one of the good reasons for not using set -u:

if the script reads ENVIRONMENT VARIABLES.

...BTW, I've added not only set -eu to most my shell scripts, but set -euo pipefail (the latter required adding || true to some calls, it was interesting, made me think which calls to prefer etc)

[Hu]

A script can read environment variables under set -u, but you must be prepared for them to be absent and have a default value arranged. This is probably safer than running without set -u, although it may cause some frustration until you have adjusted all the variable references. If you want to have a one-shot default value, use :- or -, depending on exactly when you want the default. If you want to set a value for an unset variable, so that you can later reference it without a default, use :=.

Code: Select all

echo "${NO_SUCH:-default}"
: "${USUALLY_UNDEFINED:=abc}"
echo "$USUALLY_UNDEFINED"

[halcon]

Interesting! I have to play with this...

[ALF__]

i am following the handbook with :

Designing a partition scheme

i am going for this:

Code: Select all

mount /dev/sda1 /mnt/old
mount /dev/sdc1 /mnt/new
mkdir /mnt/new/boot
mount /dev/sdc2 /mnt/new/boot
cp -a /mnt/old/. /mnt/new

after that the handbook with :

Chrooting and Selecting a boot loader

Thank you so much for this.
This made the move to a new HDD very easy and straight forward!

Br,