| View previous topic :: View next topic |
| Author |
Message |
phil_r
Apprentice
Joined: 14 Mar 2006
Posts: 259
Location: Lincoln, NE, USA
|
 Posted: Fri Dec 02, 2022 3:56 pm Post subject: Best practice for filesystems layout |
 |
|
Hi everyone. I'm asking this based off what I read in the latest (as of today!) news item in Gentoo about the split/merged-usr going away. At first I thought it was down to systemd bullying around the user base again, but as I researched it I realised it's something that's been going on for quite a while, probably over 10 years to try and make the move where (most) directories on / are actually symlinks to the /usr counterpart.
So what I'm wondering, is if / mainly contains symlinks to /usr/bin etc, is the "right" thing to have a small, say 500Mb, / filesystem and a larger (50Gb or whatever) /usr filesystem? Or just have a large / ?
For the hell of it I built a Fedora 37 VM with a 1Gb / and a 20Gb /usr and it installed just fine, doing ls -l / I can see the symlinks pointing to /usr.
My current layout on Gentoo is:
| Code: |
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 16G 219M 16G 2% /dev/shm
tmpfs 6.3G 3.7M 6.3G 1% /run
/dev/nvme0n1p7 60G 35G 26G 57% /
/dev/nvme0n1p6 974M 49M 858M 6% /boot
tmpfs 16G 43M 16G 1% /tmp
tmpfs 32G 0 32G 0% /var/tmp
/dev/nvme0n1p2 95M 45M 51M 48% /boot/efi
/dev/nvme1n1p1 916G 357G 517G 41% /home
|
What is the "correct" (as in best practice) way to layout the filesystems? When I built this system I just generally throw it all on / for simplicity but I'd like to do TheRightThing(c) where I can.
(I know there needs to be room for /etc and /var, or at least put /var on it's own filesystem, something I used to do years ago...)
_________________
Just when you think you know the answers, I change the questions. |
|
| Back to top |
|
 |
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Fri Dec 02, 2022 4:17 pm Post subject: |
|
|
systemd / udevd support for having /usr mounted separately from / has been broken and treated by upstream as not-a-bug for years, so if you make /usr a distinct filesystem, you need an initramfs to mount it early so that udevd can pretend it is not separate. For that reason, typical advice is to make /usr a directory on /. Prior to this latest stunt (which, as you say, has been grinding forward for a while), it was possible to have a partially functional system with /usr separate and unmounted, though some things (especially around sound and bluetooth, if I remember correctly) would be broken. Once the symlink stunt is completed, if /usr is separate and unmounted, / will be almost totally useless, since your shell and all your utilities would be on the not-yet-mounted /usr.
For that reason, I would say:- No separate filesystem for /usr. Leave it as a directory on /.
- No separate filesystem for /etc. For the root filesystem to be useful, you will almost certainly need /etc available, and having it as a directory instead of a separate filesystem makes that easy.
- Optionally, separate /var. This was the advice historically, and I am not aware of anything which is substantially broken when it is followed.
- Separate /home, so that users cannot fill up /.
- Separate /boot, if required by your boot process. Most people want to use UEFI boot, and most UEFI boot loaders are limited in their ability to interact with traditional Linux filesystems, so the boot process often needs a FAT format filesystem to let the firmware read it. Rely as little as possible on the FAT filesystem though, because FAT is not a very good filesystem, so you should use it only when necessary to interact with your firmware.
|
|
| Back to top |
|
|
phil_r
Apprentice
Joined: 14 Mar 2006
Posts: 259
Location: Lincoln, NE, USA
|
| Posted: Fri Dec 02, 2022 4:45 pm Post subject: |
|
|
| Hu wrote: | systemd / udevd support for having /usr mounted separately from / has been broken and treated by upstream as not-a-bug for years, so if you make /usr a distinct filesystem, you need an initramfs to mount it early so that udevd can pretend it is not separate. For that reason, typical advice is to make /usr a directory on /. Prior to this latest stunt (which, as you say, has been grinding forward for a while), it was possible to have a partially functional system with /usr separate and unmounted, though some things (especially around sound and bluetooth, if I remember correctly) would be broken. Once the symlink stunt is completed, if /usr is separate and unmounted, / will be almost totally useless, since your shell and all your utilities would be on the not-yet-mounted /usr.
For that reason, I would say:- No separate filesystem for /usr. Leave it as a directory on /.
- No separate filesystem for /etc. For the root filesystem to be useful, you will almost certainly need /etc available, and having it as a directory instead of a separate filesystem makes that easy.
- Optionally, separate /var. This was the advice historically, and I am not aware of anything which is substantially broken when it is followed.
- Separate /home, so that users cannot fill up /.
- Separate /boot, if required by your boot process. Most people want to use UEFI boot, and most UEFI boot loaders are limited in their ability to interact with traditional Linux filesystems, so the boot process often needs a FAT format filesystem to let the firmware read it. Rely as little as possible on the FAT filesystem though, because FAT is not a very good filesystem, so you should use it only when necessary to interact with your firmware.
|
- No separate filesystem for /usr. Leave it as a directory on /.
Looks like I'm already winning there 
- No separate filesystem for /etc.
Wouldn't dream of it. I'd imagine that would be a nightmare to get working at bootup.
- Optionally, separate /var.
I do do that on servers, in case /var/logs gets full. Never had an issue on desktop with that though.
- Separate /home, so that users cannot fill up /.
Yup got that.
- Separate /boot, if required by your boot process.
Yup have that also. My /boot is ext4 but /boot/efi which is it's own filesystem is fat32, as per the requirements. I have an initramfs just for the purposes of Plymouth, never understood before then why I needed one - and as it turns out, I guess I didn't as everything was on the one filesystem. Just seemed like an extra layer of complexity, but as I say now I need it for Plymouth.
So it seems like the only real change is going to be running merge-usr and letting that copy everything to the new locations, create the symlinks and switching the profile to a merged-usr one.
_________________
Just when you think you know the answers, I change the questions. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Fri Dec 02, 2022 5:17 pm Post subject: |
|
|
phil_r,
There was a poor (in my opinion) system design decision by udev a long time ago to require that /usr be mounted before udev is started.
That shuffled everything up a layer.
The root filesystem used to be enough to allow the root user to fix a broken system.
In effect, the initrd has become the root filesystem and /usr has taken the place of the root filesystem.
This breaks lots of things. Mostly things that are not used much any more, like a read only /usr shared over NFS.
The symlinks are only a stopgap. Once everything is happy with the new filesystem layout, they will be redundant.
There are ways to delay udev starting until after localmount, so the the udev problem goes away.
Then (much harder), you don't have to have udev at all.
My main system is partitioned
| Code: | $ df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/root ext4 2.0G 1.4G 492M 74% /
tmpfs tmpfs 13G 228K 13G 1% /run
shm tmpfs 63G 0 63G 0% /dev/shm
cgroup_root tmpfs 10M 0 10M 0% /sys/fs/cgroup
tmpfs tmpfs 63G 1.3M 63G 1% /tmp
/dev/mapper/nvmestatic-usr ext4 295G 70G 210G 25% /usr
/dev/mapper/nvmestatic-var ext4 5.9G 1.3G 4.3G 23% /var
/dev/mapper/storage-home ext4 2.0T 1.4T 543G 72% /home
/dev/mapper/nvmestatic-opt ext4 2.9G 364M 2.4G 14% /opt
/dev/mapper/nvmestatic-local ext4 926M 40K 859M 1% /usr/local
/dev/mapper/storage-distfiles ext4 492G 348G 119G 75% /var/cache/distfiles
/dev/mapper/storage-packages ext4 118G 24G 89G 21% /var/cache/binpkgs
/dev/mapper/nvmestatic-portage ext4 2.9G 646M 2.1G 24% /var/db/repos/gentoo
/dev/shm tmpfs 63G 0 63G 0% /var/tmp/portage
/dev/nvme0n1p2 ext4 117M 276K 108M 1% /boot
/dev/nvme0n1p1 vfat 61M 41M 20M 67% /boot/EFI |
/dev/root is /dev/mapper/nvmestatic-root. That's root in LMV so I can grow it if I need to.
/usr and /var are separate. I don't have udev so it all works with no initrd.
There is no need for /opt and /usr/local to be separate. It all depends on your paranoia level.
/var/db/repos/gentoo is full of small files. I used to use ext4 with no journal and a 1kB block size there. However, that's a verybadthing when the underlying media has a 4kB block size.
Its still no journal.
distfiles is so big as I have all my distfiles from about June 2006
/var/tmp/portage is in tmpfs as it saves writes to my nvme that will never be read. It will probably outlast me by why take that chance :)
/tmp is tmpfs is a standard thing now.
The other reason for lots of partitions, is security. Its possible to play with the mount options ro,nodev,nosuid,noexec on a per filesystem basis.
Try noexec on /home. Users will not be able to run programs from /home.
/usr can be ro until it needs to be updated.
My storage LV in 4x8T rotating rust HDD in raid 5. That gives me 24TB HDD space. I don't plan for any upgrades.
/home, distfiles packages and my media collection (12TB) are there. That's normally mounted read only.
/dev/sdi2 ext4 7.3T 3.1T 3.9T 45% /mnt/floppy
/dev/nvme0n1p2 ext4 117M 276K 108M 1% /boot
/dev/nvme0n1p1 vfat 61M 41M 20M 67% /boot/EFI
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 4152
Location: Bavaria
|
| Posted: Fri Dec 02, 2022 6:31 pm Post subject: |
|
|
phil_r,
I want confirm @Hu and add this: You dont need two partitions for booting; one for /boot and one for /boot/efi. Just take one partition (formatted withFAT32) as ESP (Efi System Partition) and mount it at /boot directly. This is also the way our AMD64 Handbook recommends it. (so you dont have /boot/efi/EFI/... only /boot/EFI/...)
For a desktop I recommend 4 partitions:
1. ESP Size 256 MB
2. SWAP
3. /
4. /home
For a server I recommend 4 or 5 partitions - depending if you have users allowed on this server (or more if you have RAID):
1. ESP Size 256 MB
2. SWAP
3. /
4. /var
... with users
5. /home
The system from @Neddy is ... a little bit ... special ... and for security reasons there are MANY other things to do (e.g. hardened sources; hardened kernel; firewall; SELinux or AppArmor, IMA) |
|
| Back to top |
|
|
phil_r
Apprentice
Joined: 14 Mar 2006
Posts: 259
Location: Lincoln, NE, USA
|
| Posted: Fri Dec 02, 2022 7:21 pm Post subject: |
|
|
| pietinger wrote: | phil_r,
I want confirm @Hu and add this: You dont need two partitions for booting; one for /boot and one for /boot/efi. Just take one partition (formatted withFAT32) as ESP (Efi System Partition) and mount it at /boot directly. This is also the way our AMD64 Handbook recommends it. (so you dont have /boot/efi/EFI/... only /boot/EFI/...)
For a desktop I recommend 4 partitions:
1. ESP Size 256 MB
2. SWAP
3. /
4. /home
For a server I recommend 4 or 5 partitions - depending if you have users allowed on this server (or more if you have RAID):
1. ESP Size 256 MB
2. SWAP
3. /
4. /var
... with users
5. /home
The system from @Neddy is ... a little bit ... special ... and for security reasons there are MANY other things to do (e.g. hardened sources; hardened kernel; firewall; SELinux or AppArmor, IMA) |
Thanks for the pointers, everyone. I understand @Neddy's setup, it's more than I what need, but with him being who he is around here I get it
Also thanks for the pointer about /boot and /boot/EFI - I wasn't aware of that. It's several years since I started going with UEFI and that was how I got it to work back then and I've always just stuck with that. I have /home on a dedicated nvme drive. The OS is on a separate nvme drive too, as @Neddy alluded to, I've taken a few steps with /var/tmp and friends to move those into memory to reduce wear on the nvme when compiling. Maybe it makes a difference, maybe not, but I'll go with it just in case
Thanks for the info everyone!
_________________
Just when you think you know the answers, I change the questions. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Fri Dec 02, 2022 7:46 pm Post subject: |
|
|
phil_r,
My /boot is actually not used, other than as a mount point for /boot/EFI.
The intent was to keep as little as possible on the VFAT partition, then I installed syslinux as my boot loader.
It can only see the partition it is executed from, so that didn't work out.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Leonardo.b
Apprentice
Joined: 10 Oct 2020
Posts: 297
|
| Posted: Fri Dec 02, 2022 8:55 pm Post subject: |
|
|
I abuse /home for many things: distfiles, sources, old logs.
In this way I don't use many partitions. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Installing Gentoo
