Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Backups
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
lars_the_bear
n00b
n00b


Joined: 05 Jun 2024
Posts: 63

PostPosted: Mon Jun 10, 2024 7:18 am    Post subject: Backups Reply with quote

Hi folks

I've just kind-of finished my first install of Gentoo on real hardware. It's a laptop from 2012, and it's working fine.

But this installation took about four days, because a huge amount was built from source. I suspect that if hadn't "USE'd away" many so components, to get a smaller system, more of the installation could have been from binaries. But, in the end, the installation required building Firefox, Thunderbird, Libreoffice, GTK, the X server, most of the Xfce4 desktop, and a bunch of other stuff.

I imagine that my part in this process might get less time-consuming, if I learn more about Gentoo. But I'm not sure the 48+ hours of compiler time can be reduced -- not on this laptop, anyway.

Incidentally, this laptop has a 200W PSU and, after two days of constant running at full capacity, the hot air exhausted from its cooling system has softened the varnish on my desk.That's not a problem that Linux has given me in the past. :)

I presume that other people are in a similar situation -- at least, there are plenty of folks on this forum bragging about how long it takes Firefox to build on their Pentium II's ;) So how do we protect ourselves against a catastrophic administrator error?

This isn't a problem with Fedora, because the worst that can happen is that I have to re-install from scratch, which usually takes less than an hour. But rebuilding from scratch with Gentoo isn't really an option, if it's on a computer that you use every day.

Perhaps if you're very familiar with Gentoo, there isn't a problem you can't fix by booting from an external drive and tinkering with things. I'm a long way from being at that point.

What steps do other people take, to make it possible to recover from a total bork without reinstalling?

BR, Lars.
Back to top
View user's profile Send private message
kgdrenefort
Apprentice
Apprentice


Joined: 19 Sep 2023
Posts: 263
Location: Somewhere in the 77

PostPosted: Mon Jun 10, 2024 7:28 am    Post subject: Reply with quote

Hello,

Indeed backups are very important.

As the binhosting new system is great to decrease time on such hardware.

For firefox, you could use jumbo-build use flag to reduce by ~2 the time of compilation, but the amount of RAM is more important, could lead to a crash of the system while compiling if you did not paid attention to your -jX value.

If you use a swapfile or swap partition, be aware that per default, the swapiness value is at 60, I prefer to set it to 1. RAM is of course way more quicker than disk, specially if you use a HDD.

Another way could be to add zram & zswap I guess.

About backups, every few days I run two RSync script, one keeping the old file and one removing what's not on the source any more, a «live» backup.

Would say, also, the best way to avoid breaking your Gentoo would be to avoid goofy stuff with your system, follow the recommandation of devs (which are mostly given in the news system).

You could use firefox-bin or libreoffice-bin for the packages that are too huge for your CPU.

I hope my answer helps you a bit.

Regards,
GASPARD DE RENEFORT Kévin
_________________
Traduire le wiki en français: relire mon travail ou participer dans ce sujet du forum.
Back to top
View user's profile Send private message
lars_the_bear
n00b
n00b


Joined: 05 Jun 2024
Posts: 63

PostPosted: Mon Jun 10, 2024 8:48 am    Post subject: Reply with quote

Hi

kgdrenefort wrote:

You could use firefox-bin or libreoffice-bin for the packages that are too huge for your CPU.


Thanks. The problem is that, so far as I know, I can't use these binaries. The binaries in the repository are built with support for things I'm not installing, like pulseaudio, evolution, gstreamer...

I can understand why the binary repositories only host one version of a specific package, but it seems to me that it's not a minimal version. Again, I can understand why.

But if you're building a minimal system for an under-resourced (by modern standards) machine, I think you have to resign yourself to a whole lot of compilation. And it's exactly that kind of machine that's going to struggle with the compilation.

I don't mind doing that once; what I"m trying to avoid is doing it repeatedly.

BR, Lars.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54455
Location: 56N 3W

PostPosted: Mon Jun 10, 2024 10:21 am    Post subject: Reply with quote

lars_the_bear,

If you have more powerful system, a VM is fire, make your own binary packages. Build in a chroot
As you describe, flexibility has a price.

There is also distcc but that not as useful as it appears at first sight.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
kgdrenefort
Apprentice
Apprentice


Joined: 19 Sep 2023
Posts: 263
Location: Somewhere in the 77

PostPosted: Mon Jun 10, 2024 10:24 am    Post subject: Reply with quote

lars_the_bear wrote:
Hi

kgdrenefort wrote:

You could use firefox-bin or libreoffice-bin for the packages that are too huge for your CPU.


Thanks. The problem is that, so far as I know, I can't use these binaries. The binaries in the repository are built with support for things I'm not installing, like pulseaudio, evolution, gstreamer...

I can understand why the binary repositories only host one version of a specific package, but it seems to me that it's not a minimal version. Again, I can understand why.

But if you're building a minimal system for an under-resourced (by modern standards) machine, I think you have to resign yourself to a whole lot of compilation. And it's exactly that kind of machine that's going to struggle with the compilation.

I don't mind doing that once; what I"m trying to avoid is doing it repeatedly.

BR, Lars.


Surely it could lead to long compilation.

The most infamous software taking too much time to compile could be:

- Chromium and like it tools, using webkit, such as webkit-gtk, or qtwebengine. These are really painful.
- Firefox, taking way less than Chromium, still a long time to compile it, as thunderbird too.
- Libreoffice, it's almost as awful as Chromium.

On older system, Gentoo can be a bit painful. Specially when you know it's better to update your system max every week, but it could be done less often, but that could lead in the future to more complicated update process.

Regards,
GASPARD DE RENEFORT Kévin
_________________
Traduire le wiki en français: relire mon travail ou participer dans ce sujet du forum.
Back to top
View user's profile Send private message
Banana
Moderator
Moderator


Joined: 21 May 2004
Posts: 1484
Location: Germany

PostPosted: Mon Jun 10, 2024 12:35 pm    Post subject: Reply with quote

Do I understand that you want to avoid the long compile times at the time you need a backup, or what is your goal exactly?

There is a wikipage, some general advice and also a forums thread about this:
https://wiki.gentoo.org/wiki/Backup
https://askubuntu.com/questions/19901/how-to-make-a-disk-image-and-restore-from-it-later
https://forums.gentoo.org/viewtopic-p-8786121.html
_________________
My personal space
My delta-labs.org snippets do expire

PFL - Portage file list - find which package a file or command belongs to.
Back to top
View user's profile Send private message
lars_the_bear
n00b
n00b


Joined: 05 Jun 2024
Posts: 63

PostPosted: Mon Jun 10, 2024 1:31 pm    Post subject: Reply with quote

Banana wrote:
Do I understand that you want to avoid the long compile times at the time you need a backup, or what is your goal exactly?


My goal is to avoid a four-day downtime, if I carelessly misconfigure something, or delete something that is not easy to reconstruct, and I have to reinstall. My personal and working data is already secure -- it's the long time delay to install Gentoo that's the potential problem. I'm wondering if there's a way to back up key parts of the Gentoo system installation, in order to make it less likely that I will ever have to reinstall from scratch.

I don't worry about this on my Fedora/Ubuntu machines, because reinstalling from scratch only takes a short time, as nothing has to be compiled.

BR, Lars
Back to top
View user's profile Send private message
lars_the_bear
n00b
n00b


Joined: 05 Jun 2024
Posts: 63

PostPosted: Mon Jun 10, 2024 1:43 pm    Post subject: Reply with quote

kgdrenefort wrote:

On older system, Gentoo can be a bit painful. Specially when you know it's better to update your system max every week, but it could be done less often, but that could lead in the future to more complicated update process.


Sure, but here's the problem: it's the possibility to support under-resourced systems (particular, in due course, ARM boards) that draws me to Gentoo in the first place. And, to be fair, my 2012 laptop absolutely flies, with everything compiled for the CPU, and all the cruft removed. I couldn't really run Fedora on this thing. Well, not without going out for a meal while it boots to desktop. But it's a perfectly serviceable computer, for day-to-day work, with Gentoo.

I imagine that the same would be true for a Raspberry Pi, or something of that nature.

"Backing up" a Raspberry Pi system partition is trivially easy: I just take the SD card out, put it in a desktop computer, and 'dd' the entire partition to a file. It's a big file, but it compresses well. I suppose I could do something similar on a desktop system -- I could dd the whole Gentoo system partition onto a file on a removable storage device of some kind.

Seems a lot of hassle, though, which is why I was wondering what other folks have done to avoid the potential problem. Or, in fact, whether there actually is a problem, for other people. If you're in a position to use binaries for everything, there likely isn't even a problem.

BR, Lars.
Back to top
View user's profile Send private message
Banana
Moderator
Moderator


Joined: 21 May 2004
Posts: 1484
Location: Germany

PostPosted: Mon Jun 10, 2024 1:43 pm    Post subject: Reply with quote

thank you for the explanation. Then a full disk backup should be the "best" option.
_________________
My personal space
My delta-labs.org snippets do expire

PFL - Portage file list - find which package a file or command belongs to.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54455
Location: 56N 3W

PostPosted: Mon Jun 10, 2024 1:43 pm    Post subject: Reply with quote

lars_the_bear,

Turn on
Code:
FEATURES="buildpkg"
to save binary package copies of everything you build.
You install from the binaries when things go wrong with either -K or -k.

Its all in
Code:
man make.conf
now you know what to look for.

You will build a collection like my public Pi4 binhost.
The quickpkg tool can make binary packages form your install too but it cannot save the original unmodifed config files.
This has security implications. Do you really want to package your password hashes from /etc/shadow?
As long as you are aware of the risks, you can make an informed choice.

Full disk backup for solid state media is a bad thing at restore time. It backs up and restores all the unallocated space too.
The restore incurs a full media write.
Research a stage4. That's like a stage3 but it includes your install too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
lars_the_bear
n00b
n00b


Joined: 05 Jun 2024
Posts: 63

PostPosted: Mon Jun 10, 2024 1:50 pm    Post subject: Reply with quote

NeddySeagoon wrote:

Turn on
Code:
FEATURES="buildpkg"
to save binary package copies of everything you build.
You install from the binaries when things go wrong with either -K or -k.


Argh -- why did I not know this last week, before I built everthing?

OK, don't answer that. The answer is obvious -- RTFM ;)

BR, Lars.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54455
Location: 56N 3W

PostPosted: Mon Jun 10, 2024 2:15 pm    Post subject: Reply with quote

lars_the_bear,

The friendly manual is only useful if you know what want.
It's not a HOWTO. Its a this is what you can do.

Code:
man make.conf
will make your eyes glaze over if you try to read it from top to bottom.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 2601
Location: Here and Away Again

PostPosted: Mon Jun 10, 2024 7:36 pm    Post subject: Reply with quote

lars_the_bear wrote:
NeddySeagoon wrote:

Turn on
Code:
FEATURES="buildpkg"
to save binary package copies of everything you build.
You install from the binaries when things go wrong with either -K or -k.


Argh -- why did I not know this last week, before I built everthing?

OK, don't answer that. The answer is obvious -- RTFM ;)

BR, Lars.

The 'quickpkg' command can be used to create binary packages from already installed packages. :]
_________________
Kindest of regardses.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20115

PostPosted: Mon Jun 10, 2024 10:42 pm    Post subject: Reply with quote

As with any backup solution, remember to test the recovery process.

That's always good to remember, but in particular I bring it up because of references to quickpkg.

In 2019, quickpkg had some odd behavior (I haven't checked to see if any of the linked bugs have been resolved):
buildpkg, downgrade-backup & unmerge-backup conflict?
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
kgdrenefort
Apprentice
Apprentice


Joined: 19 Sep 2023
Posts: 263
Location: Somewhere in the 77

PostPosted: Tue Jun 11, 2024 11:29 am    Post subject: Reply with quote

pjp wrote:
As with any backup solution, remember to test the recovery process.

That's always good to remember, but in particular I bring it up because of references to quickpkg.

In 2019, quickpkg had some odd behavior (I haven't checked to see if any of the linked bugs have been resolved):
buildpkg, downgrade-backup & unmerge-backup conflict?


By curiosity, how would you recommand to test that ?

Remove some files in source that are not critical and apply back the backup ?

Regards,
GASPARD DE RENEFORT Kévin
_________________
Traduire le wiki en français: relire mon travail ou participer dans ce sujet du forum.
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 21922

PostPosted: Tue Jun 11, 2024 12:09 pm    Post subject: Reply with quote

I would unpack some or all of the backup to an initially empty temporary area, then check that the restored files have the expected permissions, names, and contents. Deleting the live files is risky, since by definition we do not know yet whether you can get them back from the backup, so if the backup is broken, removing the live files will remove your only copy.
Back to top
View user's profile Send private message
kgdrenefort
Apprentice
Apprentice


Joined: 19 Sep 2023
Posts: 263
Location: Somewhere in the 77

PostPosted: Tue Jun 11, 2024 3:06 pm    Post subject: Reply with quote

Hu wrote:
I would unpack some or all of the backup to an initially empty temporary area, then check that the restored files have the expected permissions, names, and contents. Deleting the live files is risky, since by definition we do not know yet whether you can get them back from the backup, so if the backup is broken, removing the live files will remove your only copy.


Very good point (about being sure to retrieve the file).

I think I'll test that inside a nspawn and duck up a little bit it, worst case I broke a fake-system.
_________________
Traduire le wiki en français: relire mon travail ou participer dans ce sujet du forum.
Back to top
View user's profile Send private message
lars_the_bear
n00b
n00b


Joined: 05 Jun 2024
Posts: 63

PostPosted: Tue Jun 11, 2024 4:47 pm    Post subject: Reply with quote

From what I see, running `quickpkg` generates files in /var/cache/bindings. How do I install these packages, if I need them again? Can I just do

emerge {filename}.tar

or do I have to set up an HTTP server with a specific structure? That's what the documentation alludes to; but I'm not sure whether it's assuming you actually want to distribute the files.

Incidentally, do I need to keep all the .tar files in /var/cache/bindings? They currently take up nearly 1Gb.

BR, Lars.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54455
Location: 56N 3W

PostPosted: Tue Jun 11, 2024 5:33 pm    Post subject: Reply with quote

lars_the_bear,

man emerge:
       --usepkg [ y | n ], -k
              Tells emerge to use binary packages (from $PKGDIR)  if  they  are
              available,  thus  possibly avoiding some time-consuming compiles.
              This  option  is  useful  for  CD  installs;   you   can   export
              PKGDIR=/mnt/cdrom/packages  and  then  use  this  option  to have
              emerge "pull" binary packages from the CD in order to satisfy de‐
              pendencies.  Note this option implies --with-bdeps=n.  To include
              build time dependencies, --with-bdeps=y must be specified explic‐
              itly.

       --usepkgonly [ y | n ], -K
              Tells emerge to only use binary packages (from $PKGDIR).  All the
              binary packages must be available at the time of dependency  cal‐
              culation  or  emerge  will  simply  abort.   Portage does not use
              ebuild repositories when calculating  dependency  information  so
              all  masking  information is ignored.  Like -k above, this option
              implies --with-bdeps=n.   To  include  build  time  dependencies,
              --with-bdeps=y must be specified explicitly.



You will need
Code:
emerge -K =<category/package>-<ver>
to force a specific slot or version.

Its -g or -G for remote binrepros,
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
lars_the_bear
n00b
n00b


Joined: 05 Jun 2024
Posts: 63

PostPosted: Tue Jun 11, 2024 7:57 pm    Post subject: Reply with quote

Hi

Thanks. I hope I don't screw up so badly that I have to rely on this ;)

BR, Lars.
Back to top
View user's profile Send private message
figueroa
Advocate
Advocate


Joined: 14 Aug 2005
Posts: 2978
Location: Edge of marsh USA

PostPosted: Fri Jun 14, 2024 7:23 pm    Post subject: Reply with quote

Follow the link (here in these forums) to where I share my well tested stage4 script. https://forums.gentoo.org/viewtopic-t-1132899-highlight-stage4.html Once there you can search the page for stage4 to easily locate the post.

This is part of my regular and regularly tested backup routine. My current stage4 is 6.0 G whereas it is usually closer to 5.0 G, but I currently have three gentoo-sources kernels installed, and I've also reduced zstd compression from 9 to 3.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi
Back to top
View user's profile Send private message
mortonP
Tux's lil' helper
Tux's lil' helper


Joined: 22 Dec 2015
Posts: 88

PostPosted: Sun Jun 16, 2024 7:28 am    Post subject: Reply with quote

NeddySeagoon wrote:
lars_the_bear,
There is also distcc but that not as useful as it appears at first sight.


Ran the full profile rebuild on a Intel 10th gen quadcore laptop -> ~1300 packages.

Chromium takes the longest with ~3h
Firefox ~30m
Nodejs ~6.5m

Distcc very much helps with an old box, Nodejs seems to parallelize the best, but the bottleneck is now the laptop distributing the work and not the worker cores available on the LAN :-)
Back to top
View user's profile Send private message
mortonP
Tux's lil' helper
Tux's lil' helper


Joined: 22 Dec 2015
Posts: 88

PostPosted: Sun Jun 16, 2024 7:36 am    Post subject: Re: Backups Reply with quote

lars_the_bear wrote:

What steps do other people take, to make it possible to recover from a total bork without reinstalling?


Periodic borg backup/snapshot of root partition - differential+compressed backups don't take up much space.
Any borg snapshot then can be mounted and files recovered if needed.
Saved me several times already when I accidentally mangled important system files and needed the original back.
Whole packages... hmm... fortunately never had that problem.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54455
Location: 56N 3W

PostPosted: Sun Jun 16, 2024 8:42 am    Post subject: Reply with quote

mortonP,

With whole binary packages, its possible to extract single files.

The HOWTO is left as an exercise for the reader :)
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3204

PostPosted: Sun Jun 16, 2024 10:31 am    Post subject: Reply with quote

Yeah, it is possible to pick a single file from a binary package, but why would you even want to do that?
The title of this thread is "backup", yet all you guys have been talking about is speeding up installation process.

Faster installation is not a backup. It doesn't cover user data, and also if you have a good backup, you don't need a separate solution for reinstalling your system.
Instead of figuring out how to build and extract binary packages and infrastructure for throwing more CPUs at the problem, just copy _everything_ with rsync, serialize with tar, or feed it to borg. Or pick any of a few dozens of other options. They all get the job done, though depending on your particular setup and use case (requirements and constraints AKA your metric of choice), one will always do it slightly better than the other.
_________________
Make Computing Fun Again
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum