View previous topic :: View next topic |
Author |
Message |
crocket Guru
Joined: 29 Apr 2017 Posts: 558
|
Posted: Wed Nov 14, 2018 8:22 am Post subject: How do I best build packages on a separate system? |
|
|
I want to minimize downtime of tmux and a few other packages during package upgrades. So, I want to build them on a separate system and install them all at once.
I heard there were a few ways.
- A chroot that mirrors my own gentoo installation
- Build on another system, and run a BINHOST to serve binaries
- ...
I have only one gentoo system, but the second method can be achieved in a linux container or a virtual machine. How would you best go about building packages on a separate system? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Wed Nov 14, 2018 8:52 am Post subject: |
|
|
Ultimately to get another system to make bin packages for your main machine, that other system needs to basically have a mirror copy of your main machine, else it too will have trouble building (think dependencies.) An easy way to think about it is just to create a VM duplicate of your main machine, and just emerge world without using tmux and just let it go in the VM "console", then build packages off of that with /usr/bin/quickpkg (part of portage).
You could also set up a secondary hard drive or partition that you mirror to, and do the upgrade on that (either by VM or even ROOT=xxx emerge) - then quickpkg them to install on the main disk. You still need that to fully complete so you have all the dependencies though since that's not a main machine, things can be temporarily broken without issues.
However, I don't do binhost and just play the dice with self-hosted updates because keeping the USE flags as well as other config synchronized between the two machines is a challenge. You'll forget at some point and then build a binpkg that has the wrong USE flags... ouch!
I think tmux/gnu-screen is an easy problem to get around, are there other issues you're worried about? Portage has gotten most of the other stuff right, be glad we no longer have to deal with broken libraries after an upgrade anymore, the mandatory revdep-rebuild of yesteryear made it very possible to kill your box during an upgrade. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
crocket Guru
Joined: 29 Apr 2017 Posts: 558
|
Posted: Thu Nov 15, 2018 6:30 am Post subject: |
|
|
eccerr0r wrote: | Ultimately to get another system to make bin packages for your main machine, that other system needs to basically have a mirror copy of your main machine, else it too will have trouble building (think dependencies.) An easy way to think about it is just to create a VM duplicate of your main machine, and just emerge world without using tmux and just let it go in the VM "console", then build packages off of that with /usr/bin/quickpkg (part of portage).
You could also set up a secondary hard drive or partition that you mirror to, and do the upgrade on that (either by VM or even ROOT=xxx emerge) - then quickpkg them to install on the main disk. You still need that to fully complete so you have all the dependencies though since that's not a main machine, things can be temporarily broken without issues.
However, I don't do binhost and just play the dice with self-hosted updates because keeping the USE flags as well as other config synchronized between the two machines is a challenge. You'll forget at some point and then build a binpkg that has the wrong USE flags... ouch!
I think tmux/gnu-screen is an easy problem to get around, are there other issues you're worried about? Portage has gotten most of the other stuff right, be glad we no longer have to deal with broken libraries after an upgrade anymore, the mandatory revdep-rebuild of yesteryear made it very possible to kill your box during an upgrade. |
How do I best make a mirror in a separate filesystem? Perhaps, shall I make a ZFS snapshot and mount the snapshot on a directory with read-write permission?
The only way to get around tmux breakage that I found is to simply keep the tmux session running throughout the update. Once I detach, I won't be able to reattach until upgrade is done.
Do you think revdep-rebuild is not required anymore? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu Nov 15, 2018 8:54 am Post subject: |
|
|
If you can make COW ZFS snapshots then that sounds like a reasonable possibility.
However I think you don't need to go to such extents unless you're concerned after an upgrade it leaves your machine in an unusable state. At least if you have a VM you can test. But if it's just a detached session that's the problem, as said in the other thread, I'm sure it's possible to preserve tmux sessions. At least I am readily able to preserve gnu-screen such that I never lose detached sessions after a long upgrade session. I've constantly lost gnu-screen sessions in the past prior to finding the 'workaround' solutions that haven't failed me yet.
revdep-rebuild is not needed as much as it had been in the past where portage left broken packages around. Portage now no longer breaks packages when a package gets upgraded and a package that depends on it does not. You should still upgrade the dependent packages, of course. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
crocket Guru
Joined: 29 Apr 2017 Posts: 558
|
Posted: Thu Nov 15, 2018 9:08 am Post subject: |
|
|
eccerr0r wrote: | If you can make COW ZFS snapshots then that sounds like a reasonable possibility.
However I think you don't need to go to such extents unless you're concerned after an upgrade it leaves your machine in an unusable state. At least if you have a VM you can test. But if it's just a detached session that's the problem, as said in the other thread, I'm sure it's possible to preserve tmux sessions. At least I am readily able to preserve gnu-screen such that I never lose detached sessions after a long upgrade session. I've constantly lost gnu-screen sessions in the past prior to finding the 'workaround' solutions that haven't failed me yet.
revdep-rebuild is not needed as much as it had been in the past where portage left broken packages around. Portage now no longer breaks packages when a package gets upgraded and a package that depends on it does not. You should still upgrade the dependent packages, of course. |
An upgrade can fail in the middle. Packages from haskell overlay sometimes fail to build during upgrades.
If a system upgrade that includes tmux failed in the middle, it would be tricky to debug the issue without the error messages on the defunct tmux session. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu Nov 15, 2018 9:35 am Post subject: |
|
|
I still do not understand your concern with tmux...
Ever since I prevented gnu-screen from upgrading, I had no issues reattaching to the session to check its status - if it failed or it completed. It all depends on a little diligence to remember to upgrade gnu-screen/tmux separately after everything else is completed. Because portage now prevents breaking of existing packages (and thus requiring a mandatory revdep-rebuild) an unupgraded package will continue to work.
In any case even if you can't reattach, you can still check out /var/log/emerge.log and then go to the appropriate $PORTAGE_TMPDIR to see where it died - which is what I did when gnu-screen failed reattach in the past. Though I don't need to do it as often anymore because gnu-screen doesn't fail reattaching anymore, I still refer to these if I accidentally clear the screen or otherwise clear the history in gnu-screen.... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
crocket Guru
Joined: 29 Apr 2017 Posts: 558
|
Posted: Thu Nov 15, 2018 10:32 am Post subject: |
|
|
It's not just tmux that breaks if a system upgrade fails in the middle. Sometimes, other softwares required for work can remain broken until I finish system upgrade.
Anki, firefox, and so on all broke at least once for hours. A successful upgrade to qt 5.11 broke anki. I had to downgrade qt to 5.9. So, a successful upgrade is not a guarantee of a properly working system.
Sometimes, it takes hours to fix a broken system. For hours, I couldn't do work.
eccerr0r wrote: | Because portage now prevents breaking of existing packages (and thus requiring a mandatory revdep-rebuild) an unupgraded package will continue to work. |
How does portage prevent breaking of existing packages? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu Nov 15, 2018 3:34 pm Post subject: |
|
|
Okay, now you're making it more clear that you're concerned that the upgrade is not atomic, as in it's never all updated or all not updated at the same time versus a tmux issue. Then yes you will require a second installation of some sort as the package update times are long due to compilation time. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
Cuong Nguyen Apprentice
Joined: 18 Jan 2018 Posts: 152
|
Posted: Thu Nov 15, 2018 5:05 pm Post subject: |
|
|
I am using byobu - very nice frontend to tmux or screen.
I do compile on bigger box - actually a server of multi-thread Xeon and 32 GB RAM to do compile in a chroot environment then I use rsync or binhost emerge from the server. The only minus it is impossible to compile with exact CPU -march of my laptop, but only x86-64 generic. |
|
Back to top |
|
|
crocket Guru
Joined: 29 Apr 2017 Posts: 558
|
Posted: Fri Nov 16, 2018 3:01 am Post subject: |
|
|
eccerr0r wrote: | Okay, now you're making it more clear that you're concerned that the upgrade is not atomic, as in it's never all updated or all not updated at the same time versus a tmux issue. Then yes you will require a second installation of some sort as the package update times are long due to compilation time. |
Yes, I'd like atomic upgrades. |
|
Back to top |
|
|
|