Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How do I best build packages on a separate system?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Wed Nov 14, 2018 8:22 am    Post subject: How do I best build packages on a separate system? Reply with quote

I want to minimize downtime of tmux and a few other packages during package upgrades. So, I want to build them on a separate system and install them all at once.
I heard there were a few ways.

  • A chroot that mirrors my own gentoo installation
  • Build on another system, and run a BINHOST to serve binaries
  • ...

I have only one gentoo system, but the second method can be achieved in a linux container or a virtual machine. How would you best go about building packages on a separate system?
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA

PostPosted: Wed Nov 14, 2018 8:52 am    Post subject: Reply with quote

Ultimately to get another system to make bin packages for your main machine, that other system needs to basically have a mirror copy of your main machine, else it too will have trouble building (think dependencies.) An easy way to think about it is just to create a VM duplicate of your main machine, and just emerge world without using tmux and just let it go in the VM "console", then build packages off of that with /usr/bin/quickpkg (part of portage).

You could also set up a secondary hard drive or partition that you mirror to, and do the upgrade on that (either by VM or even ROOT=xxx emerge) - then quickpkg them to install on the main disk. You still need that to fully complete so you have all the dependencies though since that's not a main machine, things can be temporarily broken without issues.

However, I don't do binhost and just play the dice with self-hosted updates because keeping the USE flags as well as other config synchronized between the two machines is a challenge. You'll forget at some point and then build a binpkg that has the wrong USE flags... ouch!

I think tmux/gnu-screen is an easy problem to get around, are there other issues you're worried about? Portage has gotten most of the other stuff right, be glad we no longer have to deal with broken libraries after an upgrade anymore, the mandatory revdep-rebuild of yesteryear made it very possible to kill your box during an upgrade.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Thu Nov 15, 2018 6:30 am    Post subject: Reply with quote

eccerr0r wrote:
Ultimately to get another system to make bin packages for your main machine, that other system needs to basically have a mirror copy of your main machine, else it too will have trouble building (think dependencies.) An easy way to think about it is just to create a VM duplicate of your main machine, and just emerge world without using tmux and just let it go in the VM "console", then build packages off of that with /usr/bin/quickpkg (part of portage).

You could also set up a secondary hard drive or partition that you mirror to, and do the upgrade on that (either by VM or even ROOT=xxx emerge) - then quickpkg them to install on the main disk. You still need that to fully complete so you have all the dependencies though since that's not a main machine, things can be temporarily broken without issues.

However, I don't do binhost and just play the dice with self-hosted updates because keeping the USE flags as well as other config synchronized between the two machines is a challenge. You'll forget at some point and then build a binpkg that has the wrong USE flags... ouch!

I think tmux/gnu-screen is an easy problem to get around, are there other issues you're worried about? Portage has gotten most of the other stuff right, be glad we no longer have to deal with broken libraries after an upgrade anymore, the mandatory revdep-rebuild of yesteryear made it very possible to kill your box during an upgrade.


How do I best make a mirror in a separate filesystem? Perhaps, shall I make a ZFS snapshot and mount the snapshot on a directory with read-write permission?

The only way to get around tmux breakage that I found is to simply keep the tmux session running throughout the update. Once I detach, I won't be able to reattach until upgrade is done.

Do you think revdep-rebuild is not required anymore?
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA

PostPosted: Thu Nov 15, 2018 8:54 am    Post subject: Reply with quote

If you can make COW ZFS snapshots then that sounds like a reasonable possibility.

However I think you don't need to go to such extents unless you're concerned after an upgrade it leaves your machine in an unusable state. At least if you have a VM you can test. But if it's just a detached session that's the problem, as said in the other thread, I'm sure it's possible to preserve tmux sessions. At least I am readily able to preserve gnu-screen such that I never lose detached sessions after a long upgrade session. I've constantly lost gnu-screen sessions in the past prior to finding the 'workaround' solutions that haven't failed me yet.

revdep-rebuild is not needed as much as it had been in the past where portage left broken packages around. Portage now no longer breaks packages when a package gets upgraded and a package that depends on it does not. You should still upgrade the dependent packages, of course.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Thu Nov 15, 2018 9:08 am    Post subject: Reply with quote

eccerr0r wrote:
If you can make COW ZFS snapshots then that sounds like a reasonable possibility.

However I think you don't need to go to such extents unless you're concerned after an upgrade it leaves your machine in an unusable state. At least if you have a VM you can test. But if it's just a detached session that's the problem, as said in the other thread, I'm sure it's possible to preserve tmux sessions. At least I am readily able to preserve gnu-screen such that I never lose detached sessions after a long upgrade session. I've constantly lost gnu-screen sessions in the past prior to finding the 'workaround' solutions that haven't failed me yet.

revdep-rebuild is not needed as much as it had been in the past where portage left broken packages around. Portage now no longer breaks packages when a package gets upgraded and a package that depends on it does not. You should still upgrade the dependent packages, of course.


An upgrade can fail in the middle. Packages from haskell overlay sometimes fail to build during upgrades.

If a system upgrade that includes tmux failed in the middle, it would be tricky to debug the issue without the error messages on the defunct tmux session.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA

PostPosted: Thu Nov 15, 2018 9:35 am    Post subject: Reply with quote

I still do not understand your concern with tmux...

Ever since I prevented gnu-screen from upgrading, I had no issues reattaching to the session to check its status - if it failed or it completed. It all depends on a little diligence to remember to upgrade gnu-screen/tmux separately after everything else is completed. Because portage now prevents breaking of existing packages (and thus requiring a mandatory revdep-rebuild) an unupgraded package will continue to work.

In any case even if you can't reattach, you can still check out /var/log/emerge.log and then go to the appropriate $PORTAGE_TMPDIR to see where it died - which is what I did when gnu-screen failed reattach in the past. Though I don't need to do it as often anymore because gnu-screen doesn't fail reattaching anymore, I still refer to these if I accidentally clear the screen or otherwise clear the history in gnu-screen....
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Thu Nov 15, 2018 10:32 am    Post subject: Reply with quote

It's not just tmux that breaks if a system upgrade fails in the middle. Sometimes, other softwares required for work can remain broken until I finish system upgrade.

Anki, firefox, and so on all broke at least once for hours. A successful upgrade to qt 5.11 broke anki. I had to downgrade qt to 5.9. So, a successful upgrade is not a guarantee of a properly working system.

Sometimes, it takes hours to fix a broken system. For hours, I couldn't do work.

eccerr0r wrote:
Because portage now prevents breaking of existing packages (and thus requiring a mandatory revdep-rebuild) an unupgraded package will continue to work.


How does portage prevent breaking of existing packages?
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9679
Location: almost Mile High in the USA

PostPosted: Thu Nov 15, 2018 3:34 pm    Post subject: Reply with quote

Okay, now you're making it more clear that you're concerned that the upgrade is not atomic, as in it's never all updated or all not updated at the same time versus a tmux issue. Then yes you will require a second installation of some sort as the package update times are long due to compilation time.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Cuong Nguyen
Apprentice
Apprentice


Joined: 18 Jan 2018
Posts: 152

PostPosted: Thu Nov 15, 2018 5:05 pm    Post subject: Reply with quote

I am using byobu - very nice frontend to tmux or screen.

I do compile on bigger box - actually a server of multi-thread Xeon and 32 GB RAM to do compile in a chroot environment then I use rsync or binhost emerge from the server. The only minus it is impossible to compile with exact CPU -march of my laptop, but only x86-64 generic.
Back to top
View user's profile Send private message
crocket
Guru
Guru


Joined: 29 Apr 2017
Posts: 558

PostPosted: Fri Nov 16, 2018 3:01 am    Post subject: Reply with quote

eccerr0r wrote:
Okay, now you're making it more clear that you're concerned that the upgrade is not atomic, as in it's never all updated or all not updated at the same time versus a tmux issue. Then yes you will require a second installation of some sort as the package update times are long due to compilation time.


Yes, I'd like atomic upgrades.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum