What would be needed to make gentoo work like Guix/Nix ?

[alexander-n8hgeg5e]

I love Gentoo and I thought about whether it would be possible to
add this "functional package management" feature to Gentoo.

I mean basically if you make some system declaration
(I'd love to have it in python) which defines a diff to "/etc"
and then recreate a "/root" somewhere every time you want to update
or when you want a different system to be "checked out",
then wouldn't it be almost enough?

Probably emerge should run inside the new system tree and not with
the system tree as target because it could change the results,
which should preferably be bit identical.
The ebuilds already have their manifests and are hashed and can be used in the system declaration.

It could work like this:
Everytime you compile a package only the dependencies are checked out and the system
state of all the package versions gets recorded for the next time this package is needed.
The next time if some other package needs this package,
the recorded state is checked out from binpkgs and then emerge is called as usual.

The system declaration could live in /etc
and (if needed) only one single file determines which one is active.
The single file hopfully does not change the compilation results.

Emerge can check out binpkgs.
Only some "meta" program would need to read something like /etc/portage/system-declaration.d/default
and then use emerge to calculate a dependency tree,
then checkout some stage-x archive as base.
After updating the toolchain the whole toolchain can be checked out as binpkg.


Did I miss something ?
Is it possible ?

I dream of defining the system like this:

Code: Select all

gentoo = [ {  "name" : "default",
                    "repo-tree-version"   :  get_start_time_of_last_successful_update()
                    "profile" : "default/linux/amd64/17.1"
                   
                    "apply-etc-patches" : { "etc-patches-dir" : "/etc/portage/system-declaration/patches"
                                                      "mode" : "all of them"
                                                    }
                    "packages" :  [ "x11-terms/xterm","x11-wm/qtile" ]
                 }
                   { "name" : None ,  # gets filled in automatically, don't need this entry
                     "packages-0"  : ['=media-libs/libsixel-1.10.3', 'www-client/w3m']
                     "packages-1"  :  ['i-need-libsixel-and-w3m-but-DEPEND-is-missing-them'] # better fix DEPEND
                   }
               ]

Because gentoo is defined as a list, you can say the definition of a package
is the package entry and everything that comes prior to the package in the list.
If you refer to "gentoo[1]" it will checkout gentoo[0] as a base.
And using something like "packages-1" could
make sure that "packages-0" is checked out befor packages-1 is compiled.

It probably needs to check the dependency list for every package
whether all the prior packages are really needed.
(Not in the case of "gentoo[1]" over "gentoo[0]", I mean inside of such a block like "gentoo[0]" )
This because it should always use the minimal dependency list ,
so the package can be reused when needed for other packages.
Then it should add an entry to the packages list ( gentoo[0]['packages'].append("x11-terms/xterm"))
and save the binpkg state for each entry.

There would be a patch that creates make.conf
and other ones for use flags , etc.

The file that defines the active declaration would point to either,
"gentoo[0]" for a base system,
or to "gentoo[1]" if you want libsixel,
or is omitted and the "default" entry is used.

Code: Select all

{
  "active":  declarations.gentoo[0] "   # for a base system
  "repo_dates": { 'gentoo' : 'some date',
                          'other_repo' : 'some date',
                        },
  # "repo_hashes" : { 'gentoo' : 0xf5aeb3c }
 }

If you switch to "gentoo[1]" it would check out everything and auto create
a file that points to the current system declaration.

[pingtoo]

alexander-n8hgeg5e,

it is a little bit hard to follow since I don't have the original idea. But my guess is that you want to use declarative way to describe current system state. It sound that NixOS but I am not sure if that is the right one.

Let me ask, can you use a pseudo logic (but use a close to gentoo way as possible) to do what you want so can help I understand better? I have feeling Gentoo may already do some degree of what you want.

[szatox]

There are automation systems like ansible, salt, puppet and probably many more.
Looks like a decent starting point.

I used to work with ansible, one caveat was that it would not remove anything unless this thing was declared as absent, so adding a thing to a config and then removing it would result in that thing being installed on the target system, but not tracked. This approach worked fine for us; runnning a few thousands of servers we really didn't want to accidentally break with a single command, BUT since the machines had their own memories, builds weren't _really_ fully reproducible like some of those projects advertise.

You'd need an insane amount of patience to fully declare the default configuration though, so we just never did.

[alexander-n8hgeg5e]

my guess is that you want to use declarative way to describe current system state

Yes, your guess is right, I think that's how it is called properly. Somewhere I read the term "functional".
This declarative way like NixOS does it.
I was thinking about what is the the essential difference.
If I understand it correctly the essential thing is that the build does always start with a defined
state of installed packages.
So when you install Gentoo and you use a specific hashed stage tarball and then you
update the base system with the portage tree from a specific date,
and you record all your edits to /etc,
then you should also end up with a defined state.
Any package that you add with this state as base should be really reliable to recompile.
To the degree that every bit is the same.
In GUIX they have the command "guix-challenge" , it essentially rebuilds and checks whether the result is
the same as the binary you got from somewhere. (together with the declaration)
I read it here:
https://guix.gnu.org/de/manual/devel/en ... lenge.html

I don't know much more about Guix or NixOS than this.
All the other things are my idea about how it could be possible to have this with Gentoo.

[pingtoo]

alexander-n8hgeg5e,

So if you have follow the Configuration Management methodology, you will know some of terminologies. It is like property, state, converge and so on.

I don't think Gentoo have anything ready mode to do what you described. I also think it is not Gentoo's state of direction for making such system. I think Gentoo is doing what it is good at doing on how to custom a system to one's desire. However how to refine the customisation is should left with its user.

I think if you use CM tools like puppet, ansible or cfengine on top of Gentoo it may archive your desire.

I think manually if I want to do what you described, I would place all the configuration file (/etc for example) and /var/lib/portage/world as well add /etc/portage/sets/ into RCS (git for example) and tag/branch each base/stage as some sort of profile (branch) and use RCS check out as you described as way to switch between profile. Once a profile checked out, first emerge depclean than emerge world should be able do closely to your design.

[alexander-n8hgeg5e]

I think Gentoo is doing what it is good at doing on how to custom a system to one's desire

It shouldn't interfere with this or do big directional changes(I know this sounds a bit questionable but how do I know without evaluating it).
Maybe it's good to consider this feature.

[pingtoo]

I think Gentoo is doing what it is good at doing on how to custom a system to one's desire

It shouldn't interfere with this or do big directional changes(I know this sounds a bit questionable but how do I know without evaluating it).
Maybe it's good to consider this feature.

If we were to follow good old unix design philosophy Do one thing simple and do it good, use another tools to add on more features

I would prefer Gentoo continue to improve on how to make emerge more easier to control building packages, give out more hint on how to recover from build failures, easier to define profile for customisation.

I would rather leave the transition of system from state to state use another tools.

I do agree it would be nice to have such tool consist with gentoo's design.

And I also agree it is nice feature to have.

[spica]

I dream of defining the system like this:

Code: Select all

gentoo = [ {  "name" : "default",
                    "repo-tree-version"   :  get_start_time_of_last_successful_update()
                    "profile" : "default/linux/amd64/17.1"
                   
                    "apply-etc-patches" : { "etc-patches-dir" : "/etc/portage/system-declaration/patches"
                                                      "mode" : "all of them"
                                                    }
                    "packages" :  [ "x11-terms/xterm","x11-wm/qtile" ]
                 }
                   { "name" : None ,  # gets filled in automatically, don't need this entry
                     "packages-0"  : ['=media-libs/libsixel-1.10.3', 'www-client/w3m']
                     "packages-1"  :  ['i-need-libsixel-and-w3m-but-DEPEND-is-missing-them'] # better fix DEPEND
                   }
               ]

It looks to me, you can unlock your dream in just a few lines of scripting away.
Delve into the possibilities with the following pseudocode, tailored for seamlessly processing this JSON file:

Code: Select all

eselect profile set data['profile']

for patch in data['apply-etc-patches']
  extract patch // I guess patches are provided within this json file as well

for package in data[ $1 if $1 else 'packages']
  emerge -av1 package

[alexander-n8hgeg5e]

Code: Select all

for patch in data['apply-etc-patches']
    extract patch // I guess patches are provided within this json file as well

Yes, something simple like this is probably good to test things out.
The patches I thought would be in something like
"/etc/system-declaration.d/patches"
or
"/etc/sys-dec.d"
Let's say it would be an extra tool separate from portage so not in /etc/portage.
I think they would be to big to be inside the json file.
You would instruct the tool to apply the base system to a tree and then
"chroot" into it (or so) do your edits and then run a command to
generate the patches.
If you need to edit them again you would instruct the tool to apply them to the base tree.
for this it would be enough to have the following config:
I replace the "gentoo" with "system" to have it more like an extra tool.
Let's say try the minimal to get it working.
It would need to know the version of the repository to use
because usually gentoo doesn't requires a specific version of the tree.

Code: Select all

system = [
            {
            # the multiple tarballs are more for syntax illustration
            'tarball-0' : Tarball('tarball definition for a tarball that needs to be first)
            'tarball-1' : [Tarball('tarball definition for tarballs that need to be second)]
            'tarballs'  : [Tarball('tarball definition')]
            #'packages':[] # want no packages here
            # because I want to be able to checkout this stage and
            # apply patches on it to have a custom make.conf
            'patches' : [PatchDirPatches("/etc/sysdeclare/patches/tarball/")]
            },
            {
            # It probably should do a full update at the beginning of this stage
            # (by default in any "stage" that has packages)
            # At least the state after the update should be recorded
            # (the required binpkgs)
            # And then when the packages are compiled
            # it should record all individual packages 
            # as "installable atoms" that fullfill the previous stage ( here system[0])
            # and remember their binpkgs.
            # Probably everything that compiles should be recorded as 
            # "package atom" that fulfills all the requirements here in this system definition
            # at this point (system[0] done, system[1] in progress)
            # together with the identification of the binpkgs that would recreate this state.
            # (portage should know the installed packages at this point)
            # A question is whether it is necessary to completely reassemble a fresh
            # tree from binpkgs everytime you want to compile a package that
            # only depends on system[0] after you compiled a package that also 
            # only depends on system[0].
            # Or maybe it is enough to unmerge it ?
            # BTW system[0] does not depend on any successfull update
            # But all the packages here can only be recorded as valid if
            # the "full" update succeeds.
            # Probably it's better to say:
            # If you request system[1] then they should be valid for system[1] if system[1]
            # completes but not yet for system[2]
            # 'patches' : [] # need no patches here, USE flags can be defined with the tarballs
            'packages' : ["media-libs/libsixel", "www-client/w3m" ]
            },
            # because of packages a repo-version info is required 
            'repo-version' : RepoVersion('gentoo',VersionDefinition("last successfull build"))
            {
            # here I could configure some packages
            # but this could be done as usual in the systems /etc
            # The result is a situation like with "config protect"
            # essentially 2 systems that need a "merge"
            'patches' : [PatchDirPatches("/etc/sysdeclare/patches/config-my-packages/")]
            }
         ]

# Maybe this can be used to
# "save" the insertion order even after
# the dictionary has been created above
# so you don't have to check the postfix of something like "packages-0"
from collections import OrderedDict as OD
system[0]=OD(system[0])

With this config you could run a command like "sysdeclare checkout system[0]"
Then you could use git to make the patches and put them into their directory ("/etc/sysdeclare/patches/tarball")

After it is done compiling there are 2 systems and the /etc/files of the running system would need
a treatment like with "config-protect". And then basically the new system
and it's packages database should become the running system somehow.

Hm and now I notice something, in the situation when a package
needs to be build but there are other packages installed that
are not a dependency of this package,
maybe then it is enough to change the world file and let portage do a depclean
so you get an as-good-as-fresh tree, like Pingtoo suggested.

Hm,... probably portage can do all of the updating, because the result of this would be a set of binpkgs. Why should it be different? There is "protect-owned".
I already made script that checks all the files and also searches for orphan files.
For the "etc-patches" it could generate an ebuild and portage could handle the
etc-update.
In the end it turns out portage can do it already.

[sam_]

I don't have the time at the moment to get into a full discussion about the differences, but I'd like to note that declarative configuration doesn't imply isolation per-packages or becoming fully nix-like.

[alexander-n8hgeg5e]

, but I'd like to note that declarative configuration doesn't imply isolation per-packages or becoming fully nix-like.

I agree. In the following quote I mixed it up I think.

This declarative way like NixOS does it.
I was thinking about what is the the essential difference.
If I understand it correctly the essential thing is that the build does always start with a defined
state of installed packages.

It's more the "essential thing" that the build always starts
with a defined state.
(reproducible on different systems with different packages installed)
This means everything that could potentially change the result is the same.
A build could potentially read the time and produce a different result.
Because intercepting the call to the clock would be overkill, it makes sense
to omit this requirement on such a system.
Guix has this tool to check which packages do such things.
Actually I don't know whether NixOS pushes it this far and even intercepts time calls but
I guess no.
So I think it makes sense to remove everything from the list of requirements that usually does not
change the result of a build.
I should better say:
Everything that doesn't has the potential to change the build in case
the build is behaving "normal" or "somewhat normal".
Where this line of normality is drawn is a question.
It defines how much of the "features" usefulness is kept.
The most prominent "usefulness", I think is that it detects
all the missing dependencies in the packages.
For this to work my guess is that it would be needed to set a world file with
only the desired package and then do a depclean
and then emerge it.
Would this do the job ?

You could even omit a system that exchanges the root with some mechanism,
because portage already can assemble trees.
I don't really know how the "emerge --root" feature
would compare to chrooting into it.
It uses another compiler...
With some tricks this could even be made to work as fast.
without the need to rewrite everything on the disk.

Maybe if the binpks would experimentally stored decompressed.
Then the reads from these achieves are maybe cached with kernels "virtual memory" ?
If you then run the build in ramdisk it should be fast.
But the actual rootfs that will install the made binpkgs, shouldn't be on a ramdisk.
Then this part would be slow.

Maybe if you would use a copy-on-write filesystem and
you would map the binpkgs to the filesystem somehow.
Then when the contents are needed they could simply be
'copied' without actually doing much.

[pingtoo]

In general Portage is a system already did what you described. Except it is implemented in a reversed order of your description.

Imagine, in a declarative Portage design, you would put all the desired program/application into the "world" file and you will run "emerge world" which in tern will examine current state and start finding resolution to bring current state to the desired state. This is how Portage is doing now.

Except its current implementation seems more toward to a Programmer mind where you ran a command (emerge <something>) first and because the <something> was not explained as "state" therefor it is not intuitively think it is asking change "state".

Portage was never "promised" that every invocation will correctly take you to the intended result, In fact as we all know emerge usually will take several round before it will settle down to a execution plan. And what you described now is more or less an optimisation of the execution plan. As we understood "Optimisation" is a trade off, you trade time with space, you can either save time or save space. Using "bin host" and/or COW file system is just trading space with time, It dose not change the design principle of Portage.

If I have a use case where I would like to switch my workstation from state to state I would use Cfengine together with Git. I prefer Cfengine over other like Ansible or Puppet because I like Cfengine 's design idea of targeting single machine as the principle entity as opposite to Ansible/Puppet assume there always a Group of something.

I would utilize Git to store current state (The current state's "Property' will include system/application configuration file and the "world" file) in a branch and the branch name is the named state (or "Profile" as I call it) I will utilize Cfengine's command line argument '-D <define>' as way to indicate state changes. and Cfengine will in tern check out the "<define>"(branch) and compare with current state, use cfengine builtin dependency resolution strategy to come up with a execution place and carry it out.

[alexander-n8hgeg5e]

Portage was never "promised" that every invocation will correctly take you to the intended result, In fact as we all know emerge usually will take several round before it will settle down to a execution plan.

Yes, it's not portages fault, portage can handle a lot but because the possible states a system can be in, portage cannot guarantee to find a solution and also the solution will not be a defined state.
Some orphan files in a directory that is scanned by some program change the systems state in a relevant way.
It's almost impossible to get back to a defined state from some some states.

And what you described now is more or less an optimisation of the execution plan. As we understood "Optimisation" is a trade off, you trade time with space, you can either save time or save space.

In case you can't go back without setting up an new root it's more than a tradeoff. But if you plan to setup a new root it's a tradeoff because then you can reach the state and decide how to reach it.

Using "bin host" and/or COW file system is just trading space with time, It dose not change the design principle of Portage.

I want to figure something out that leaves portage untouched.
My idea is to ask portage for a simpler task (like with a fresh root) so the outcome can be determined.
If you do all the compiling in a fresh root and the result is a bunch of binpkgs, then I guess the task to install them and clean the system from everything else is much more guaranteed to be possible.
Because the compilation ran with a fresh root the binaries are determined.
Because portage can handle the install of the binpkgs it can be used to "switch" to the new system.
Maybe together with a script that scans for orphan files the outcome can be almost the same as with NixOS.

Cfengine will in tern check out the "<define>"(branch) and compare with current state, use cfengine builtin dependency resolution strategy to come up with a execution place and carry it out.

I looked a little bit into what cfengine does. Maybe I overlooked it but it seems that it works on a live system
and sets up all the files and the "state".
If I would feed cfengine with a fresh root to setup, I think It would produce the intended result,
but it seems cfengines features are more intended to check the state of a system and
figure out how to "live update" the state.
the design of a system setup "like with nix" is more to start fresh and simply expect the system to be the intended state. You would only check the resulting file hashes to make sure
everything worked as intended.

[pingtoo]

Cfengine will in tern check out the "<define>"(branch) and compare with current state, use cfengine builtin dependency resolution strategy to come up with a execution place and carry it out.

I looked a little bit into what cfengine does. Maybe I overlooked it but it seems that it works on a live system
and sets up all the files and the "state".
If I would feed cfengine with a fresh root to setup, I think It would produce the intended result,
but it seems cfengines features are more intended to check the state of a system and
figure out how to "live update" the state.
the design of a system setup "like with nix" is more to start fresh and simply expect the system to be the intended state. You would only check the resulting file hashes to make sure
everything worked as intended.

live-state changes is what I under impress from your post, if you just want to build two or more different state (profile) I suggest you have a look dev-util/catalyst. It can build a clean profile however you defined.