| View previous topic :: View next topic |
| Author |
Message |
hedmo
Veteran
Joined: 29 Aug 2009
Posts: 1307
Location: sweden
|
 Posted: Sun Feb 07, 2021 2:06 pm Post subject: best way to move gentoo . hdd -> ssd (solved) |
 |
|
hi
i have read a lot about cloning , coping and making a stage X but what is the fastest and best way to switch my gentoo
installlation from hdd to ssd.
regards
Last edited by hedmo on Mon Feb 08, 2021 4:48 pm; edited 1 time in total |
|
| Back to top |
|
 |
Buffoon
Veteran
Joined: 17 Jun 2015
Posts: 1369
Location: EU or US
|
| Posted: Sun Feb 07, 2021 3:26 pm Post subject: |
|
|
Boot from external media, mount both HDD and SDD in temporary directories, use 'cp -a', chroot into your SDD and reinstall bootloader, fix everything where PARTUUID or filesystem UUID was used.
_________________
Life is a tragedy for those who feel and a comedy for those who think. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Sun Feb 07, 2021 3:40 pm Post subject: |
|
|
I had a thread on this recently. My problem was that I could not boot any rescue medium and I was also changing drive position. I solved it by putting both drives in an empty PC (the target actually) that could boot CD's.
For your case, if you have a rescue medium (like the Gentoo liveusb) on CD, DVD, or USBstick that you can boot, put both drives in that box (easiest if it is the target box), boot the rescue medium and mount both drives as say /mnt/src and /mnt/target. Any names will do. Create you partitions and filesystems as you would for a new install on the target drive if you haven't done so already. Then use "cp -ar" as root to copy the real filesystems (i.e. not /proc or /sys or /dev ...) but not symlinks! I found out that if you copy symlinks like /lib recursively as well as copying the target, then you will do a double copy.
If doing it as a remote, boot both systems to rescue media, start an rsync daemon on both, and use rsync. It's quicker and easier to monitor.
I wrote a shell script for the latter scenario but wound up doing the steps by hand. I'm bad at writing shell. Maybe you could fix it.
/usr/local/bin/cloneme
When switching to an SSD add a line to the root crontab: | Code: | 15 2 * * tue /sbin/fstrim -va |logger
|
Tuesday morning is convenient for me. Use whatever time is convenient for you.
fstrim is supplied by util-linux which is a dependency of portage (and many other packages) so you should have it.
EDIT:
cloning windows is easier because typically it's all on one drive partition and Microsoft provides a tool. Of course, you have no idea what kind of data mining the tol is doing nor what Windows itself is doing. I have a Win10 VM for TurboTax only. It uses a totally different LAN address so Turbotax can talk to the internet but nothing in the VM can see the other machines. When i need a file, I burn a CD and let the VM see the CDROM burner and also the USB printer (not the wireless printer).
I did an fstrim right away on the new SSD. Don't know if that is required or even advisable but I figured one extra trim can't hurt.
You might have to adjust /etc/fstab if you are booting by UUID or label and possibly by devname. Doesn't hurt to review it.
Since I was cloning to a machine with the same architecture, I also had to edit /etc/conf.d/hostname (OpenRC). I use fixed ipaddresses but I let the dhcp server assign them by MAC address so they are effectively fixed, but I do all my machines at one spot.
I was cloning to a machine with the same CPU family and you are cloning to the same machine. If one wanted to clone to a machine that wasn't binary compatible. It would be better to do a stage3 install, then copy /root, /home /etc/portage and the portage world file over and emerge -e @world |
|
| Back to top |
|
|
hedmo
Veteran
Joined: 29 Aug 2009
Posts: 1307
Location: sweden
|
| Posted: Sun Feb 07, 2021 4:09 pm Post subject: |
|
|
Tony0945 and Buffoon
thank you for replying . i am going to start my box with gentoo livedvd 2008 and mount my hdd and sdd at mnt . gentoo-old and gentoo-new . what i have read is the problems with symlink and stuff about --excluding . this has got me confused and like to know the exact
command to not deal with problems . |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Sun Feb 07, 2021 4:30 pm Post subject: |
|
|
In my script I excluded /home and /root because I was using it for two purposes - making a quick and dirty copy for a new machine, and for moving to a new drive like you. Also the symlinks are for profile 17.0
Not sure what needs to be changed for 17.1 |
|
| Back to top |
|
|
halcon
l33t
Joined: 15 Dec 2019
Posts: 629
|
| Posted: Sun Feb 07, 2021 4:35 pm Post subject: |
|
|
Hi Tony0945,
| Tony0945 wrote: | I wrote a shell script for the latter scenario but wound up doing the steps by hand. I'm bad at writing shell. Maybe you could fix it.
/usr/local/bin/cloneme |
I am not an expert in shell, but I have some comments.
| Quote: | | Code: | TARGET="999.999.999"
if [ "$#" -ne 1 ] ;
then
echo "ERROR: use target ip address as a parameter" && exit 1;
fi
TARGET=$1; |
|
I would rewrite it as
| Code: | IP_ADDRESS="${1}"
if [[ -z "${IP_ADDRESS}" ]]; then
echo "${0}: IP_ADDRESS argument not passed. Exiting 1." >&2
exit 1
fi |
| Quote: | | Code: | scp -p -B -r /"$d" root@"$TARGET":/mnt/gentoo/
# rsync -av "$d" root@:"$TARGET":/mnt/gentoo/ |
|
1. Why do you prefer rsync to scp? The latter is vulnerable, are you aware about it?
2. I would never recommend to login remotely as root. A user login + rsync in sudoers would be much more safe.
_________________
A wife asks her husband, a programmer:
- Could you please go shopping for me and buy one carton of milk, and if they have eggs, get 6?
He comes back with 6 cartons of milk.
- Why did you buy 6 cartons of milk?
- They had eggs. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 21918
|
| Posted: Sun Feb 07, 2021 5:39 pm Post subject: |
|
|
I suggest always starting with set -eu unless you have a good reason not to do so. These make the shell more strict by enabling the shell options errexit and nounset.This is a dead assignment. If the user invoked the script properly, then TARGET is changed without being read. If the user invoked the script improperly, then the script exits and never reads TARGET. I would delete this line.I agree with halcon. Redirect the error message to stderr (echo text >&2) and use ;, not && to chain to the exit. In the admittedly very unlikely case that the echo fails, your script as currently written would not exit. Consider: | Code: | $ /bin/sh -c 'echo && exit 5'; echo $?
5
$ /bin/sh -c 'echo && exit 5' >/dev/full; echo $?
/bin/sh: line 0: echo: write error: No space left on device
1
$ /bin/sh -c 'echo && exit 5; exit 0' >/dev/full; echo $?
/bin/sh: line 0: echo: write error: No space left on device
0 |
set +x is the default if not set differently by prior statements (and this script has no such statements), or by the user invoking the shell (who, if he set -x, probably wants that, and turning it off is undesirable). I would delete this line.This variable is misspelled, and so will expand to nothing. Under set -u, that failed expansion will become a fatal error.
As a style point, echoing the destination and not the directory name seems less useful to me than it could be. Did you mean to also echo $d in that line?I would generally prefer rsync over scp. If I were using rsync, I would use more options than you have shown here, since -a does not enable all the options you likely want when the goal is a perfect copy.This looks very strange. What are you trying to do here?This is wrong. dev is not a command. You probably want to remove the $( and ) here.As with the ln above, this does not make any sense. Is it just a reminder to the user to go run commands elsewhere? | halcon wrote: | | I would rewrite it as |
For completeness, I will note that while what you propose would work, it has two failure modes:- It will not warn the user for attempting to pass too many arguments.
- It is not compatible with the set -u I recommend earlier in my post. Attempting to read $1 fails under set -u when there are no arguments:
| Code: | $ /bin/sh -c 'echo $#;echo 1=$1'
0
1=
$ /bin/sh -u -c 'echo $#;echo 1=$1'
0
/bin/sh: $1: unbound variable |
| halcon wrote: | | 1. Why do you prefer rsync to scp? The latter is vulnerable, are you aware about it? |
Are you referring to the scp glob vulnerability? If so, I think that is not applicable here since he is copying to the peer, not from it. Additionally, that vulnerability requires a server with unusual filesystem contents. For the case in question here, where the operator fully controls both systems, I think it is reasonable to assume the server will not try to trigger the glob expansion vulnerability. | halcon wrote: | | 2. I would never recommend to login remotely as root. A user login + rsync in sudoers would be much more safe. |
I generally discourage remote root login, but for the case described here, it is much simpler to configure. If the login is restricted to key files (not passwords), and only well trusted systems have the private key, I think remote root is acceptable. |
|
| Back to top |
|
|
hedmo
Veteran
Joined: 29 Aug 2009
Posts: 1307
Location: sweden
|
| Posted: Sun Feb 07, 2021 5:41 pm Post subject: |
|
|
i am following the handbook with :
| Quote: |
Designing a partition scheme
|
i am going for this:
| Code: |
mount /dev/sda1 /mnt/old
mount /dev/sdc1 /mnt/new
mkdir /mnt/new/boot
mount /dev/sdc2 /mnt/new/boot
cp -a /mnt/old/. /mnt/new
|
after that the handbook with :
| Quote: |
Chrooting and Selecting a boot loader
|
|
|
| Back to top |
|
|
halcon
l33t
Joined: 15 Dec 2019
Posts: 629
|
| Posted: Sun Feb 07, 2021 6:43 pm Post subject: |
|
|
| Hu wrote: | | I suggest always starting with set -eu unless you have a good reason not to do so. These make the shell more strict by enabling the shell options errexit and nounset. |
| Hu wrote: | For completeness, I will note that while what you propose would work, it has two failure modes:- It will not warn the user for attempting to pass too many arguments.
- It is not compatible with the set -u I recommend earlier in my post. Attempting to read $1 fails under set -u when there are no arguments:
| Code: | $ /bin/sh -c 'echo $#;echo 1=$1'
0
1=
$ /bin/sh -u -c 'echo $#;echo 1=$1'
0
/bin/sh: $1: unbound variable |
|
Thanks for the reminder! I've been thinking about set -eu for a long time but never used it because worried many things would be broken. May be, indeed, it's time to just try - what will be broken and what won't 
Also, maybe, set -e will help me to understand better how shell exiting works... (where to put || exit $?, and where not)
| Hu wrote: | | Are you referring to the scp glob vulnerability? If so, I think that is not applicable here since he is copying to the peer, not from it. Additionally, that vulnerability requires a server with unusual filesystem contents. For the case in question here, where the operator fully controls both systems, I think it is reasonable to assume the server will not try to trigger the glob expansion vulnerability. |
Yes, I was referring to that. Okay then...
| Hu wrote: | | I generally discourage remote root login, but for the case described here, it is much simpler to configure. If the login is restricted to key files (not passwords), and only well trusted systems have the private key, I think remote root is acceptable. |
Well... That's right
_________________
A wife asks her husband, a programmer:
- Could you please go shopping for me and buy one carton of milk, and if they have eggs, get 6?
He comes back with 6 cartons of milk.
- Why did you buy 6 cartons of milk?
- They had eggs. |
|
| Back to top |
|
|
figueroa
Advocate
Joined: 14 Aug 2005
Posts: 2978
Location: Edge of marsh USA
|
| Posted: Mon Feb 08, 2021 4:54 am Post subject: |
|
|
Should always have a current backup. In a perfect world, restore from backup like backup created with the following script:
| Code: | #!/bin/sh
#stage4.scr
# Also note: https://wiki.archlinux.org/index.php/rsync#Full_system_backup
mount /mnt/backup0
cd /
date > /mnt/backup0/stage4date.txt
tar cpf /mnt/backup0/stage4.tar.zst -I "zstd -19 -T0" --acls --xattrs --numeric-owner --no-wildcards-match-slash -X /home/USERNAME/bin/exclude.master /
date >> /mnt/backup0/stage4date.txt
cd |
The date lines are optional. ~/bin is my personal scripts directory.
Here is my exclude file:
| Code: |
$ cat bin/exclude.master
dev/*
home/USERNAME/*
lost+found
media/*
mnt/*/*
proc/*
root/.cache/*
run/*
sys/*
tmp/*
var/tmp/*
var/cache/distfiles/* |
When restoring the archive with tar, use the following switches: --acls --xattrs-include='*.*' --numeric-owner
My zstd stage 4 archive is 3.7 G.
This is tested and found to work, even with mounted partitions.
It does not backup or restore any personal files. It doesn't install a boot mechanism handle the EFI partition or MBR. Install your own boot stuff.
Adjust to match your USERNAME and layout. Disclaimers etc.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi
Last edited by figueroa on Mon Feb 08, 2021 5:54 pm; edited 1 time in total |
|
| Back to top |
|
|
Logicien
Veteran
Joined: 16 Sep 2005
Posts: 1555
Location: Montréal
|
| Posted: Mon Feb 08, 2021 1:26 pm Post subject: |
|
|
| Code: | | rsync -av --delete-after --progress --stats /source/directory/ /destination/directory/ |
_________________
Paul |
|
| Back to top |
|
|
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2027
|
| Posted: Mon Feb 08, 2021 1:32 pm Post subject: |
|
|
Having done this recently by taking a tar backup and restoring it to the new device, I found one issue - the dbus helper /usr/libexec/dbus-daemon-launch-helper program had lost its "suid" bit, which may be a result of me getting the tar command wrong. That meant a plethora of error messages on booting.
_________________
Greybeard
Last edited by Goverp on Mon Feb 08, 2021 4:30 pm; edited 1 time in total |
|
| Back to top |
|
|
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 204
|
| Posted: Mon Feb 08, 2021 3:44 pm Post subject: |
|
|
I just want to point out that there are some metadata that 'cp -a' won't preserve.
It won't keep shared extents of reflinked files. That means if files share extents those will be unshared and the copy will take more space. That can be a problem If you have reflinked copies of large disk images to save space.
Cp will not preserve file attributes (which should not be confused with extended attributes (xattr) which are preserved). File attributes are anything that can be set with 'chattr' including btrfs nocow, immutable flag, etc.
Some metadata can't be preserved by any tool because it can't be set. That include inode number and the ctime & crtime timestamps. That shouldn't matter but some backup programs might use that metadata to detect unmodified files and will therefore re-read all data after the migration. |
|
| Back to top |
|
|
hedmo
Veteran
Joined: 29 Aug 2009
Posts: 1307
Location: sweden
|
| Posted: Mon Feb 08, 2021 4:47 pm Post subject: |
|
|
| thank you all for the help. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Mon Feb 08, 2021 4:53 pm Post subject: |
|
|
Post how it all worked, And ask for help if there is a glitch.
Hu,
Thanks for that detailed critique.!I'm going to start another topic on that, quoting your response in full. It deserves comment and exploration of questions. You obviously spent considerable time and it should be explored fully. Perhaps in "tips and tricks"? It's Monday, my daughter's internet is out and there is lots of non-gentoo stuff to do. But I do earnestly want to get back to this. |
|
| Back to top |
|
|
hedmo
Veteran
Joined: 29 Aug 2009
Posts: 1307
Location: sweden
|
| Posted: Mon Feb 08, 2021 5:03 pm Post subject: |
|
|
| Tony0945 wrote: | | Post how it all worked, And ask for help if there is a glitch. |
at the moment all works super and swished to ssd made it super fast . i will reopen it if i found something.
regards |
|
| Back to top |
|
|
halcon
l33t
Joined: 15 Dec 2019
Posts: 629
|
| Posted: Mon Feb 08, 2021 6:47 pm Post subject: |
|
|
| halcon wrote: | | I've been thinking about set -eu for a long time but never used it because worried many things would be broken. May be, indeed, it's time to just try - what will be broken and what won't |
I've tried. set -u is great, I like it!
But I haven't yet understood what set -e does... It does not exit the program if the error was checked? Evidently, I don't know how to handle exits correctly with set -x, and paid attention to that only now.
Two examples:
| Code: | LC_ALL=C /bin/bash -c 'set -e; source /usr/local/bin/mclass_utilities.sh; var=$(get_newest_file_or_dir "file") || exit $?; echo "var = ${var}"'
Wrong number of arguments: 1 instead of 5.
Exiting 1. |
| Code: | LC_ALL=C /bin/bash -c 'set -e; set -x; source /usr/local/bin/mclass_utilities.sh; var=$(get_newest_file_or_dir "file") || exit $?; echo "var = ${var}"'
+ source /usr/local/bin/mclass_utilities.sh
++ get_newest_file_or_dir file
++ [[ 1 -ne 5 ]]
++ exit_err_1 'Wrong number of arguments: 1 instead of 5'
++ local '__arg_error=Wrong number of arguments: 1 instead of 5'
++ echo 'Wrong number of arguments: 1 instead of 5.
Exiting 1.'
Wrong number of arguments: 1 instead of 5.
Exiting 1.
++ exit 1
+ var=
+ exit 1 |
As it can be seen, without set -x, the script exits correctly, before echoing. But with set -x, echoing works.
The function get_newest_file_or_dir is in the file mclass_utilities.sh and it contains that code: | Code: | if [[ $# -ne 5 ]]; then
exit_err_1 'Wrong number of arguments: '"$#"' instead of 5'
fi |
And exit_err_1 is: | Code: | function exit_err_1 {
local __arg_error="${1}"
echo "${__arg_error}"'.
Exiting 1.' >&2
exit 1
} |
How to exit in case of errors with set -x?
_________________
A wife asks her husband, a programmer:
- Could you please go shopping for me and buy one carton of milk, and if they have eggs, get 6?
He comes back with 6 cartons of milk.
- Why did you buy 6 cartons of milk?
- They had eggs. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 21918
|
| Posted: Mon Feb 08, 2021 7:36 pm Post subject: |
|
|
You are misreading the output. set -x shows you that the variable was assigned an empty value, then your explicit exit happens. Your echo is not hit. You can see this more clearly if you rewrite your echo to look less like the output of set -x.
Generally, set -e is roughly equivalent to appending || exit $? in many places. There are select places where it is inactive, though. For example, it is not active while evaluating a statement to determine the truth of an if: | Code: | set -e
if false; then
echo Nothing
fi
echo but we made it here
|
|
|
| Back to top |
|
|
halcon
l33t
Joined: 15 Dec 2019
Posts: 629
|
| Posted: Mon Feb 08, 2021 8:12 pm Post subject: |
|
|
| Hu wrote: | | You are misreading the output. set -x shows you that the variable was assigned an empty value, then your explicit exit happens. Your echo is not hit. You can see this more clearly if you rewrite your echo to look less like the output of set -x. |
Indeed. I see now that set -x has nothing to do with this...
Here are another examples that I do not understand:
CODE 1
| Code: | #!/bin/bash
set -eu
function step_clean_boot {
local __newest=$(get_newest_file_or_dir 'file' 'file') || exit $?
echo
ls -a /boot/ || exit $?
}
step_clean_boot
exit 0 |
OUTPUT 1
| Code: | LC_ALL=C kerneler_test.sh
/usr/local/bin/kerneler_test.sh: line 6: get_newest_file_or_dir: command not found
. .. .keep grub |
CODE 2
| Code: | #!/bin/bash
set -eu
source /usr/local/bin/mclass_utilities.sh
function step_clean_boot {
local __newest=$(get_newest_file_or_dir 'file' 'file') || exit $?
echo
ls -a /boot/ || exit $?
}
step_clean_boot
exit 0 |
OUTPUT 2
| Code: | LC_ALL=C kerneler_test.sh
Wrong number of arguments: 2 instead of 5.
Exiting 1.
. .. .keep grub |
Why is ls being executed?
_________________
A wife asks her husband, a programmer:
- Could you please go shopping for me and buy one carton of milk, and if they have eggs, get 6?
He comes back with 6 cartons of milk.
- Why did you buy 6 cartons of milk?
- They had eggs. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 21918
|
| Posted: Mon Feb 08, 2021 8:56 pm Post subject: |
|
|
local succeeds, so there is no error to trigger the exit. Per info bash: | Code: | The return status
is zero unless 'local' is used outside a function, an invalid NAME
is supplied, or NAME is a readonly variable. |
Split the use of local and the assignment into separate statements: | Code: | f() {
local a
a=$(expression_that_may_fail)
} |
|
|
| Back to top |
|
|
halcon
l33t
Joined: 15 Dec 2019
Posts: 629
|
| Posted: Mon Feb 08, 2021 9:00 pm Post subject: |
|
|
| Hu wrote: | | local succeeds, so there is no error to trigger the exit. |
Ah! I fell for it again... (knew but forgot) 
Thanks a lot!
_________________
A wife asks her husband, a programmer:
- Could you please go shopping for me and buy one carton of milk, and if they have eggs, get 6?
He comes back with 6 cartons of milk.
- Why did you buy 6 cartons of milk?
- They had eggs. |
|
| Back to top |
|
|
halcon
l33t
Joined: 15 Dec 2019
Posts: 629
|
| Posted: Thu Mar 11, 2021 10:18 pm Post subject: |
|
|
| Hu wrote: | | I suggest always starting with set -eu unless you have a good reason not to do so. |
Found one of the good reasons for not using set -u:
if the script reads ENVIRONMENT VARIABLES.
...BTW, I've added not only set -eu to most my shell scripts, but set -euo pipefail (the latter required adding || true to some calls, it was interesting, made me think which calls to prefer etc)
_________________
A wife asks her husband, a programmer:
- Could you please go shopping for me and buy one carton of milk, and if they have eggs, get 6?
He comes back with 6 cartons of milk.
- Why did you buy 6 cartons of milk?
- They had eggs. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 21918
|
| Posted: Fri Mar 12, 2021 2:40 am Post subject: |
|
|
A script can read environment variables under set -u, but you must be prepared for them to be absent and have a default value arranged. This is probably safer than running without set -u, although it may cause some frustration until you have adjusted all the variable references. If you want to have a one-shot default value, use :- or -, depending on exactly when you want the default. If you want to set a value for an unset variable, so that you can later reference it without a default, use :=. | Code: | echo "${NO_SUCH:-default}"
: "${USUALLY_UNDEFINED:=abc}"
echo "$USUALLY_UNDEFINED" |
|
|
| Back to top |
|
|
halcon
l33t
Joined: 15 Dec 2019
Posts: 629
|
| Posted: Fri Mar 12, 2021 3:32 am Post subject: |
|
|
Interesting! I have to play with this...
_________________
A wife asks her husband, a programmer:
- Could you please go shopping for me and buy one carton of milk, and if they have eggs, get 6?
He comes back with 6 cartons of milk.
- Why did you buy 6 cartons of milk?
- They had eggs. |
|
| Back to top |
|
|
ALF__
Apprentice
Joined: 30 Nov 2003
Posts: 246
|
| Posted: Sun Jul 17, 2022 8:39 pm Post subject: |
|
|
| hedmo wrote: | i am following the handbook with :
| Quote: |
Designing a partition scheme
|
i am going for this:
| Code: |
mount /dev/sda1 /mnt/old
mount /dev/sdc1 /mnt/new
mkdir /mnt/new/boot
mount /dev/sdc2 /mnt/new/boot
cp -a /mnt/old/. /mnt/new
|
after that the handbook with :
| Quote: |
Chrooting and Selecting a boot loader
|
|
Thank you so much for this.
This made the move to a new HDD very easy and straight forward!
Br, |
|
| Back to top |
|
|
|
Installing Gentoo
