Unofficial Request for Comments: Gentoo Filesystem Hierarchy

[marcion]

New Elements:

/portage

/portage/distfiles - automatically downloaded files
/portage/fetch - manually downloaded files
/portage/tmp - portage working directory
/portage/tree - main portage tree

Deprecated Elements:

/usr/portage
/var/tmp/portage

Reasoning:

I was looking at Mac OS X and I noticed that instead of slavishly following the 15 year old 'Filesystem Hierarchy Standard' based on a 30 year old Unix, they have tweaked the layout to represent a 21st Century networked operating system.

I was also thinking about Gentoo, and I think we should be willing to consider having a filesystem layout that represents how Gentoo works rather than how System V once worked.

Here I am not actually arguing against the Filesystem Hierarchy Standard, only the Gentoo-specific extensions to it. The Gentoo default directories have been dumped onto the structure without enough due care and attention.

/usr is supposed to be for more static data such as program files. Therefore you can put the /usr directory on a partition or dedicated disk, safely away from frequently written parts of the disk such as /tmp or the swap partition.

So there is a contradiction within Gentoo's default filesystem as the portage tree, which changes quite a bit everytime that I sync, really wants to be away from my compiled programs.

Next problem is /var/tmp/portage. Ideally, at least on a production system, /tmp and /var/tmp should be isolated and restricted, such as using dedicated partitions with the noexec bit set. In normal use, it is very rare to envision a situation where you need 6GBs of space in /var/tmp. However, in Gentoo to some programs to compile you need a massive amount of space for /var/tmp (for the most extreme example, think OpenOffice).

In my opinion, the simpliest solution is to keep to the main portage specific directory within a new top-level directory called /portage

This simplification would allow partitioning to be more sensible, backups to be more focused and have a positive side effect of being easier for new users.

What do you think?

[kopp]

I think this thread could be moved to the Userreps Forum where it would fit well.

[runningwithscissors]

This has been debated to death. The filesystem hierarchy is the way it is for very good reasons.

I do however, think that the portage tree ought to be moved to /var/portage and distfiles to /var/portage/distfiles.
The scripts belong in /usr/lib. The tree kind of feels awkward in there.

Of course, I couldn't care less if it remains in /usr, I just think /var is a more logical choice.

[mark_alec]

Moved from Gentoo Chat to Gentoo User Representatives.

[antarus]

New Elements:

/portage

/portage/distfiles - automatically downloaded files
/portage/fetch - manually downloaded files
/portage/tmp - portage working directory
/portage/tree - main portage tree

Deprecated Elements:

/usr/portage
/var/tmp/portage

Reasoning:

I was looking at Mac OS X and I noticed that instead of slavishly following the 15 year old 'Filesystem Hierarchy Standard' based on a 30 year old Unix, they have tweaked the layout to represent a 21st Century networked operating system.

I was also thinking about Gentoo, and I think we should be willing to consider having a filesystem layout that represents how Gentoo works rather than how System V once worked.

Here I am not actually arguing against the Filesystem Hierarchy Standard, only the Gentoo-specific extensions to it. The Gentoo default directories have been dumped onto the structure without enough due care and attention.

/usr is supposed to be for more static data such as program files. Therefore you can put the /usr directory on a partition or dedicated disk, safely away from frequently written parts of the disk such as /tmp or the swap partition.

So there is a contradiction within Gentoo's default filesystem as the portage tree, which changes quite a bit everytime that I sync, really wants to be away from my compiled programs.

Next problem is /var/tmp/portage. Ideally, at least on a production system, /tmp and /var/tmp should be isolated and restricted, such as using dedicated partitions with the noexec bit set. In normal use, it is very rare to envision a situation where you need 6GBs of space in /var/tmp. However, in Gentoo to some programs to compile you need a massive amount of space for /var/tmp (for the most extreme example, think OpenOffice).

In my opinion, the simpliest solution is to keep to the main portage specific directory within a new top-level directory called /portage

This simplification would allow partitioning to be more sensible, backups to be more focused and have a positive side effect of being easier for new users.

What do you think?

I've brought this up a dozen times. No one wants to rock the boat with regards to backwards compat, upgrading to a portage that changes the defautls and then downgrading again; problems with upgrades, rewriting tools, fixing screw ups.

I get the same answer every time:
"Those directories are already configurable, if you hate having stuff in /usr or whereever, just move it via the respective ENVVAR."

And I tend to agree actually. It matters not (to me anyway) where the default lives as long as I can change it.

[timeBandit]

I get the same answer every time:
"Those directories are already configurable, if you hate having stuff in /usr or whereever, just move it via the respective ENVVAR."

And I tend to agree actually. It matters not (to me anyway) where the default lives as long as I can change it.

I was about to post the same observation so it's nice to see agreement from an authoritative (or at least, far more experienced) source. I vaguely recall that some of the variables (PORTDIR?) once carried moderately dire warnings in make.conf that it was unwise to change their values. Were the warnings due to bugs long since repaired, or merely a deterrent to meddling by careless/inexperienced hands?

I keep /usr mounted read-only in normal use, and frequently forget to remount it read/write before sync or merge operations. The latter is just me being dumb, but the former is annoying. The package tree really is run-time data for Portage, as another poster noted, and arguably does belong in /var...but I'm not going to waste energy on the argument--not least because eclasses and whatnot are code not data, arguing against a move to /var. Whatever.

If the tree really can be moved reliably I may have a go at it, one of these days. I already moved distfiles for space management, so I'm on the slippery slope anyway.

[marcion]

This has been debated to death.

That is a null point. Slavery was debated for a thousand years with the same conclusion, that slavery was ordained by God, before William Wilberforce came along.

The filesystem hierarchy is the way it is for very good reasons.

The Linux filesystem hierarchy is good, the Gentoo hack to it is not. I.e. the reasons for the Gentoo hacks are not 'good', but it is part of human psychology to justify failure. Windows has 90% market share for reasons (monopoly, illegal agreements with OEMs, first-mover advantage, etc) but not good ones. What are the reasons for putting the portage tree in /usr? Except for someone did it and we cannot be bothered to think about it now?

I do however, think that the portage tree ought to be moved to /var/portage and distfiles to /var/portage/distfiles.

You have contradicted yourself here, but I see what you mean. I am glad that you agree that the portage tree has no place in /usr.

I've brought this up a dozen times.

(Because you are smart )

No one wants to rock the boat with regards to backwards compat,

This is Gentoo here, not RHEL. Gentoo is not really backwards compatible anyway, if you emerge world but do not etc-update then your system will die eventually, and the old versions are removed ('cleaned') from the tree very quickly.

Obviously this would need to be implemented very cleverly and over many portage versions.

The least clever system is as follows, I'm sure someone smarter can think of something better. To start with, the current portage directory options in /etc/make.conf could be explicitly stated for everyone - etc-update could add that to existing systems as part of a portage upgrade.

Sometime after that, new versions would come with the new layout. Then the next version of portage would expect the new layout. People wanting to change their system would upgrade to the latest version of portage, move the directories, and then delete the lines from /etc/make.conf. Or do nothing and things will go on for them as normal.

Where Gentoo has re-engineered the system before (i.e. started things again) they have provided guides, like the Java upgrade guide and the Apache upgrade guide, but Gentoo has been willing to break updates in order to fix up some messy hacks. This is actually far less severe than this. In the previous example people could (/did?) end up with broken web servers, the worst that can go wrong here is that people have to add a line to /etc/make.conf.

[Insanity5902]

I agree, the portage directories need to be re-aligned, most of this is already in place to happen with the variables PORTDIR, DISTDIR, PORTAGE_TMPDIR.

What is not covered in this is the stuff in /var/lib/portage (should it stay here), and what about the different symlinks. They don't use the variables (the one, and only one, I know about is make.profile ... which is kind of important.

I don't see how upgrading would be hard at all, a simple script could copy the information over, and have it happen with the next stable version of portage (2.2 or 2.1.3, etc). Like it has been pointed out before, backwards compatibility is not something gentoo has concerned it self with too much before, when it needs to make a change, it is well thought out, documented, tested and then executed.

[1clue]

Even so, you could just draw a line in the sand. Any existing system has its portage directories in the current spots, and any new installation gets the new places.

This is not hard to implement given the existence of the portage-specific variables already mentioned. The next version of portage sets the variables explicitly to the old locations. Following that, the upgrades only set it if it has not already been set on this system. At that point, the directories on new systems go to the newer places, the old systems are not altered.

[marcion]

I don't see how upgrading would be hard at all, a simple script could copy the information over,

Good idea, there could be something like this but it would have to:

[0. Check for existing dotfile and hashes]
1. Make a hash of the existing data (tree and distfiles).
2. Copy the existing data.
3. Check that the new data is the same as the old.
4. Delete the old stuff.
[5. Delete the dotfile and hashes]

At each process it could record how far it has got in a temporary file (similar to .revdep*) and then delete that file when it is all complete.

Why would I bother with all this? Just using mv alone is messy because there might be a power failure in the X number of seconds required to move the data. Unlikely perhaps but it *could* happen. In this more complicated version, after power has been restored, the script would check for the dotfile, read the dotfile, recheck the hashes and carry on where it left off.

This is not hard to implement given the existence of the portage-specific variables already mentioned. The next version of portage sets the variables explicitly to the old locations. Following that, the upgrades only set it if it has not already been set on this system. At that point, the directories on new systems go to the newer places, the old systems are not altered.

I agree 100%, anyone that says this is too hard does not realise how complicated some other Gentoo upgrades are.

There are many advantages not yet mentioned. For example, if the tree and distfiles are always under /portage then making local mirrors would be even more of a breeze. Also, if you have a room of X Gentoo machines then they could just network mount one /portage disk or partition. You can do this already, making three different setups but just doing it once is far easier. Making it easy on people to save bandwidth is a *good* thing.

[richfish]

I actually couldn't care less whether the portage tree lives at /portage, /usr/portage, /var/portage, or /home/portage (my personal favorite, since portage is user after all!). As long as it can be changed to suite a particular users' tastes, I don't think it matters much. But specifically regarding your proposed layout:

Missing:
- location for binary packages.
- location for layman overlays.

I Like:
- that distfiles have moved out of the tree hierarchy. Much better for "grep -r" usage!!

Dislikes:
- Don't see a point for having separate dirs for "automatically downloaded" vs "manually downloaded". Regardless of how they got on the system, the content is the same.
- I predict lots of users would fail to create a separate filesystem for /portage/tmp, and would end up filling up their root filesystems during a openoffice build+merge. Better to leave this in /var/tmp IMO. Most people who don't make one big filesystem on their box know they should keep /var separate, and I'd much rather see them run out of space on /var than /.

Also for my case, I prefer having the distfiles and binary packages in the same heirarchy, so I can easily make a single filesystem for both. So rather than .../packages and .../distfiles, I'd like .../pkgs and .../pkgs/src/, with binary packages in pkgs/, and "distfiles" in pkgs/src/.

Of course, thanks to the power of symlinks, I already have something similar on my system

[Conan]

I guess what I don't understand is why you feel it is necessary for the defaults to change knowing that is going to cause a gigantic headache for supporting portage. All of the locations are already configurable, if you take offense to directories being where they are, then move them. Asking for the default to be changed would result in the irc support being overloaded for a few days, the forums being overloaded forever, and a lot of explaining why that doesn't really need to happen.

With reguard to the java/php upgrades, the comparisson isn't even close. Java/php are used by only a small amount of users, and the support for that stretched months. Portage is (obviously) used by all users and the support for that could stretch for a long time.

Hell, we still have people come into irc that havn't switched over to cascading profiles, and thats at least a few years old.

[Insanity5902]

Another idea for the portage dir, instead of putting the tree -in ./tree, do something like ./tree-base, and then for you local portdirs (you could have many, I know I have at least 2 - 3 right now), is to do a ./tree-local. This way a simple ./tree-* will give you then entire portage, instead of putting them in to completely different locations.

Putting the tree in /home is also a perfectly good idea, since Portage is a user.

Another thing about the source downloads, is that what /usr/src has been for? Technically there is a good reason to put several parts of portage in several different places, all of which have pretty good reasons.

@Conan, it isn't just taking offense to the location of the different portage dirs, it is creating a default schema that not only provides better organization, but that allows for easier administration of a machine or several machines. I don't want to have to remember to setup each server to allow for correct partitioning (setting noexec, read-only, etc) of my disk. Your are right though, for a home user there is no reason as to why this matters, as most follow the gentoo guide of 3 MAYBE 4 partition of /boot swap / AND MABYE /home, so they wouldn't be effected, but those of us that run a tighter ship on our desktops, or run a server to minimize any damage done by a rogue program/virus/intruder or just a dumb user, will want to find a way that makes more sense and makes life easier to administer.

Even if the answer is just to create a simple HOW-TO in the docs, and create an ebuild or script that takes care of everything would be easier than having to mainly re-setup portage to take of all of this.

While yes the initial and probably well after the initial change will create a flood of post and help request, the good thing is everyone who browsing all the sections will easily spot the answer and inform the user. And if this is the only reason for not doing it, then I really think it should be done, because this is a none issue to me (and yes, I do help a lot of the forums) and it brings plenty of benefits.

[Genone]

I've brought this up a dozen times. No one wants to rock the boat with regards to backwards compat, upgrading to a portage that changes the defautls and then downgrading again; problems with upgrades, rewriting tools, fixing screw ups.

Maybe once or twice, but not a dozen times. And backwards compat isn't the biggest issue here IMO (should be mostly limited to cache regen), but invalidating a lot of information out there (think about how often people say "I don't have a /etc/portage", changing the $PORTDIR default would cause confusion on a similar scale). Well, that and people arguing what the new default should be
Oh, and Infra might get quite upset (for causing massive mirror loads) if we'd just change the default without making sure that the old tree is copied over to the new location (which has its own set of issues).

I get the same answer every time:
"Those directories are already configurable, if you hate having stuff in /usr or whereever, just move it via the respective ENVVAR."

By whom? Just curious.

And I tend to agree actually. It matters not (to me anyway) where the default lives as long as I can change it.

Sometimes I think we should just remove the default completely and force everyone to set the locations themselves But many people probably won't like that kind of education

@marcion: A new top level dir wouldn't be an option anyway (violates FHS), if we'd change things we'd move it most likely to somewhere in /var (technically it belongs in /var/cache, but that doesn't really fit IMO).

[marcion]

@marcion: A new top level dir wouldn't be an option anyway (violates FHS)

What about /media? That was added later and the world did not end. Anyway, the whole point is that, by definition, portage has no place in the FHS, and that /usr/portage is certainly against the plan.

irc support being overloaded for a few days, the forums being overloaded forever...

...and 'the River Tiber foaming with much blood'?

Sounds like an over-reaction, and it is a non-technical argument again. For the reasons outlined above, technologically putting the tree in /usr is quite bad, other people here have suggested other good places (/home/portage is interesting, and a lot of people do this, but maybe a little confusing to new users as the metaphors start to wear down) . Saying we can't change it because we can't change it, this is not an argument. No one yet has attempted to explain why the portage tree should hide in /usr - 'Arrayed against us: the forces of conservatism, the cynics, the elites, the establishment. Those who will live with decline.'

Sorry, must stop making jokes that only British people will get...

So since the FHS is has been referred to, lets go back to the sources (http://www.pathname.com/fhs/pub/fhs-2.3.html):

/usr is shareable, read-only data. That means that /usr should be shareable between various FHS-compliant hosts and must not be written to. Any information that is host-specific or varies with time is stored elsewhere. Large software packages must not use a direct subdirectory under the /usr hierarchy.

So in not so many words, /usr is meant for binaries, e.g. X, libraries, games and so on and so on.

/usr/src ... Source code may be place placed in this subdirectory, only for reference purposes ... The only source code that should be placed in a specific location is the Linux kernel source code. It is located in /usr/src/linux

So /usr/src/linux taking on such as big role is a quirk, an exception, not a role model.

/var contains variable data files. This includes spool directories and files, administrative and logging data, and transient and temporary files.

So according to the FHS, /var would be the more obvious place., a vast improvement over /usr

My personal feeling is that since portage itself is so far outside what normal systems have, we should just keep it completely separate in /portage.

[antarus]

I've brought this up a dozen times. No one wants to rock the boat with regards to backwards compat, upgrading to a portage that changes the defautls and then downgrading again; problems with upgrades, rewriting tools, fixing screw ups.

Maybe once or twice, but not a dozen times. And backwards compat isn't the biggest issue here IMO (should be mostly limited to cache regen), but invalidating a lot of information out there (think about how often people say "I don't have a /etc/portage", changing the $PORTDIR default would cause confusion on a similar scale). Well, that and people arguing what the new default should be
Oh, and Infra might get quite upset (for causing massive mirror loads) if we'd just change the default without making sure that the old tree is copied over to the new location (which has its own set of issues).

I get the same answer every time:
"Those directories are already configurable, if you hate having stuff in /usr or whereever, just move it via the respective ENVVAR."

By whom? Just curious.

And I tend to agree actually. It matters not (to me anyway) where the default lives as long as I can change it.

Sometimes I think we should just remove the default completely and force everyone to set the locations themselves But many people probably won't like that kind of education

@marcion: A new top level dir wouldn't be an option anyway (violates FHS), if we'd change things we'd move it most likely to somewhere in /var (technically it belongs in /var/cache, but that doesn't really fit IMO).

Zac, Brian, SpanKY, and maybe solar? I don't have my IRC logs here.

Any yeah maybe a 'dozen times' was me being overexagerating a bit

[i92guboj]

I personally would like to see something like this by default, though, as someone else already declared: I couldn't care less what the defaults are as long as I can change them.

I would say "yes" to /var/portage (or even /home/portage, though I would continue using /var/portage, for a number of reasons that I will resume later). I would definitely say "no" to ${PORTDIR}/tmp, there is already a temporal location for ALL the temporal stuff, and there is NO REASON WHY a) portage should have its own, and b) [...and there is NO REASON WHY] portage needs to be always writable.

My reasons to vote for a /var/portage and not an /usr/portage are those mainly:

1.- The already mentioned FHS, I still can't understand what is the hard thing to understand in this:

Code: Select all

Chapter 4. The /usr Hierarchy
Purpose
/usr is the second major section of the filesystem. /usr is shareable, read-only data. That means that /usr should be shareable between various FHS-compliant hosts and must not be written to. Any information that is host-specific or varies with time is stored elsewhere.

Which portage does not fit at all. I would -however- perfectly fit under /var.

2.- Practical reasons derived from the previous one:

Like for example the fact that sometimes it is needed to have a r/o filesystem in /usr without having to mount it just to sync (yeah, I have portage in a separate partition, but that fact does not fix the real problem, /usr is conceptually something more than a dir to put stuff on it). And the fact that there is nothing that a normal user would need to use in that tree, unlike most stuff in /usr.

3.- Not a reason, but still I will say:

Look at any package manager out there, they use /var for volatile stuff like that. And yes, I know about the bsd ports, I think that that is the reason why portage is under /usr, but not all genetic heritages are good ones Yeah, yeah, as I said, that is not a valid argument, I know.

Anyway, I fail to see the problem with layman or any related thing. Just base all the rest of variables by default relative to ${PORTDIR}, then setting ${POTDIR} is almost all you need. The config for layman is nothing compared to the installation handbook, I don't think users are such morons that can't handle that (and the ones that can't, should not be messing with layman anyway). And, the amount of mainteinance for this, would be far lesser than the mainteinance for projects like the kde split ebuilds or the modular xorg. Far far less.

Of course, some documentation would need to be added/updated.

As I also said, this is just an opinion of which I consider correct, not a critic to the gentoo develoopers, that have demonstrated that they already do a good job, I don't mind if this ever happen or not. Gentoo is about choice, and I can change what I dont like and maintain it myself.

[marcion]

Which portage does not fit at all. I would -however- perfectly fit under /var.

I am glad that someone else can see that Gentoo's use of /usr is really odd.

I would say "yes" to /var/portage

Well I think that is a happy compromise and appears to be the general consensus.

[bmichaelsen]

another /me-too voting for the portage tree in /var from an long time gentoo user ...

[zerojay]

Yes, /var please.

[antarus]

I am glad that someone else can see that Gentoo's use of /usr is really odd.

I doubt you will find many people who say that /usr is a good place to put the tree.

I think you will find a lot (of developers) who just don't care enough.

It's a burden on us; we have to rewrite all our docs, any tools that are broken (due to relying on /usr/portage being hardcoded) and re-educate everyone on where the default path is; I don't think you will find many devs who think it's worth it.

If you hate it and want FHS conformance; change the location yourself.

[bmichaelsen]

I am glad that someone else can see that Gentoo's use of /usr is really odd.

I doubt you will find many people who say that /usr is a good place to put the tree.

I think you will find a lot (of developers) who just don't care enough.

It's a burden on us; we have to rewrite all our docs, any tools that are broken (due to relying on /usr/portage being hardcoded) and re-educate everyone on where the default path is; I don't think you will find many devs who think it's worth it.

If you hate it and want FHS conformance; change the location yourself.

Well, there is this wonderful invention of symlink on POSIX systems. So why not putting the tree on /var/portage and symlink to it from /usr in the next gentoo and portage release and remove to symlink in one or two years? That shouldnt hurt anybody ...

[steveL]

Well, there is this wonderful invention of symlink on POSIX systems. So why not putting the tree on /var/portage and symlink to it from /usr in the next gentoo and portage release and remove to symlink in one or two years? That shouldnt hurt anybody ...

Great idea.

[Lloeki]

I second /var/somewhere as a portage tree location.

[smart stuff]

yes indeed.

[madchaz]

Personally, if we want to go thought a significant upgrade, moving the tree wouldn't really be the way to go I'd like to see.

Right now, using a FS tree is becoming more and more of an issue. It creates a structure with over 100 000 ebuilds in it and makes for a lot of load when it comes to updating.

What I'd like to see is a move away from the FS tree entirely and on to something saner, like a database. This could make searching a LOT faster and reduce the load for updating portage.

When you think about it, there is little need to even have the actual ebuild on the machine, unless it gets installed. It would be easy to only keep the metadata and dependencies in a database and when someone tries to build, portage gets the proper ebuild.

In my view, this would weild a lot of benifits for everyone. The users would need less time to sync and use less bandwidth to do so. Infra would have to deal with less load and it would make searching much faster.

Edit: It would also be easy to keep the list of installed packages in the database. This would also make uninstalling a lot easier as it would be easy to cross reference dependencies to only remove those that are truly un-needed.