Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Why does Portage download masked / uninstallable packages?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
davidbryant
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jun 2020
Posts: 137
Location: Canyon Lake, Texas

PostPosted: Sun May 02, 2021 10:03 pm    Post subject: Why does Portage download masked / uninstallable packages? Reply with quote

I've been using Gentoo Linux for almost a year. I've been using openSUSE Linux since 2003, but didn't get into Gentoo until recently. One of the problems I see is with disk space. Gentoo tends to gobble up a lot of space with its cache of source code files. The first time I installed Gentoo on my hard drive I was running low on space within two weeks. So I doubled the size of my root partition (I put /home in its own separate partition) and started over. I'm doing OK on disk space now -- both partitions are about 60% free space.

I've been a little unhappy with "eclean". It seems to get stuck in an endless loop sometimes, and then I have to kill it. It also seems to have some trouble identifying all the redundant source code tarballs. That's not real surprising. There are a dizzying variety of naming conventions in use, apparently. Trying to decide which patches go with which version level is sometimes pretty hard to do.

Anyway, I have devised a procedure to keep /var/cache/distfiles pretty compact. Today I tested my procedure. I used a terminal command to make a text file list of all the files in /var/cache/distfiles, loaded the list into a spreadsheet, then used some simple function calls to locate the embedded hyphens in each name, and to compare the parts of each name preceding the rightmost hyphen to the previous item in the sorted list, and also to the next item, using a logical OR. So I ended up with a bunch of "TRUE" and "FALSE" flags, which whittled the list of possible duplicates from ~3,700 to 50 or 60.

Anyway, after I had combed through that residue I was left with two packages that should not be in my system, as far as I can tell.

Code:
gcc-multilib-bootstrap-20180511.tar.xz,TRUE
gcc-multilib-bootstrap-20201208.tar.xz,TRUE
thin-provisioning-tools-0.8.5.tar.gz,TRUE
thin-provisioning-tools-0.9.0.tar.gz,TRUE

I don't understand why the "multilib" source code belongs in my system. I remember that I chose "nomultilib" during the initial Gentoo installation last June. I went with the pure 64-bit tarball because I was confident that I would never need to run any old 32-bit programs. So why did Portage download this package? My installed compilers are gcc (Gentoo 10.2.0-r5 p6) 10.2.0 and clang version 11.1.0 (Target: x86_64-pc-linux-gnu). I'm not sure I know which package "multilib-bootstrap" belongs to. Maybe it's a dependency?

The other package ("thin-provisioning") is less of a mystery, but still seems odd. I do have a "thin-provisioning" package installed -- 0.7.0. The 0.8.5 and 0.9.0 packages have both been masked. So why did Portage download them into the cache?

Any insight will be appreciated. Thanks!
_________________
David Bryant
Canyon Lake, Texas
Back to top
View user's profile Send private message
asturm
Developer
Developer


Joined: 05 Apr 2007
Posts: 8936

PostPosted: Sun May 02, 2021 10:26 pm    Post subject: Reply with quote

Tarballs are not packages. Packages do not necessarily relate to a single tarball. Source tarballs and their naming conventions are derived from upstreams, not some repackaged binary download that you get from opensuse, so unless they collide with our QA rules this is what you get in your distfiles directory.

In case of the thin-provisioning-tools tarballs, if you have been using sys-kernel/genkernel, this package is pulling multiple source tarballs from different projects, that being one of them.

4.2.1-r1 is the latest stable version:
Code:
$ eshowkw sys-kernel/genkernel
Keywords for sys-kernel/genkernel:
           |                             |   u   | 
           | a   a     p s   a   r       |   n   | 
           | m   r h   p p   l i i m m s | e u s | r
           | d a m p p c a x p a s 6 i 3 | a s l | e
           | 6 r 6 p p 6 r 8 h 6 c 8 p 9 | p e o | p
           | 4 m 4 a c 4 c 6 a 4 v k s 0 | i d t | o
-----------+-----------------------------+-------+-------
3.5.3.3    | + + + ~ + + + + ~ ~ o o ~ ~ | 5 # 0 | gentoo
 4.0.10    | + + + ~ + + + + ~ ~ o o ~ ~ | 7 #   | gentoo
  4.1.2-r3 | + + + ~ + + + + ~ ~ o o ~ ~ | 7 #   | gentoo
  4.2.1-r1 | + + + ~ + + + + ~ ~ ~ o ~ ~ | 7 o   | gentoo
   9999    | o o o o o o o o o o o o o o | 7 o   | gentoo

It is pulling tarball of thin-provisioning-tools-0.9.0.

tl;dr: Portage *does not* download masked / uninstallable packages.
Back to top
View user's profile Send private message
Phoenix591
Guru
Guru


Joined: 17 Sep 2007
Posts: 484

PostPosted: Mon May 03, 2021 6:08 am    Post subject: Reply with quote

Glibc pulls down that gcc tarball if the multilib useflag is enabled
Back to top
View user's profile Send private message
davidbryant
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jun 2020
Posts: 137
Location: Canyon Lake, Texas

PostPosted: Mon May 03, 2021 3:49 pm    Post subject: More on keeping /var/cache/distfiles clean Reply with quote

asturm wrote:
Tarballs are not packages.

Thanks for the quick response, Andreas. I do understand that. But many times there is a one-to-one correspondence between a package and a tarball. For instance, a few months ago Portage upgraded app-editors/bluefish (a generic "package") from version 2.2.10 to 2.2.12. After that upgrade there were two tarballs in /var/cache/distfiles: bluefish-2.2.10.tar.bz2 and bluefish-2.2.12.tar.bz2. Clearly the 2.2.10 tarball contained much of the source code for the superseded "bluefish" package, and the 2.2.12 tarball replaced it when Portage recompiled / relinked "bluefish". So after the new version had been compiled, linked, and installed, the 2.2.10 tarball was superfluous, for most intents and purposes.

I just want to keep my hard disk from getting full. I understand that I could delete everything from /var/cache/distfiles, and just pull all the source tarballs via the internet whenever Portage wants to install / upgrade a package. That probably wouldn't hurt me a whole lot. I have a very fast internet connection. But it would also probably put more of a burden on the remote servers than necessary. So I also want to find a happy medium. I used to work in a multi-programming environment, so I know quite a lot about resource hogs. I don't really want to make a nuisance of myself.

asturm wrote:
Portage *does not* download masked / uninstallable packages.

Well, you can probably understand why I find that assertion a little bit confusing. Here.

Code:
localhost ~ # emerge --ask =sys-block/thin-provisioning-tools-0.9.0

These are the packages that would be merged, in order:

Calculating dependencies... done!

!!! All ebuilds that could satisfy "=sys-block/thin-provisioning-tools-0.9.0" have been masked.
!!! One of the following masked packages is required to complete your request:
- sys-block/thin-provisioning-tools-0.9.0::gentoo (masked by: ~amd64 keyword)

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

So if I try to install thin-provisioning-tools-0.9.0, Portage says the package is masked. Apparently there's some sort of "if ~amd64, do not proceed" logic in Portage that is invoked when I try to pull 0.9.0, but your "genkernel" package isn't subjected to precisely the same scrutiny once Portage has downloaded "genkernel" and begins to process it. That actually makes sense, in a way. The philosophy appears to be "Protect the stupid users from shooting themselves in the foot, but let the programmers do what they need to do". And yeah, I get it that "genkernel" downloads the 0.9.0 tarball, and not the 0.9.0 package. But that distinction is still a bit hard for me to make, because of the one-to-one corrrespondence I mentioned previously. I'll catch on someday.

Just for my own edification and amusement, I ran "emerge -v --oneshot sys-kernel/genkernel" this morning, then studied the output. I see that the "genkernel" package pulls in 38 different tarballs (I may have miscounted). I also noticed that it sets up configuration files for 17 different machine architectures, which looks like overkill. I'm not complaining. Just observing. Thanks again.Tthis has been educational.
_________________
David Bryant
Canyon Lake, Texas
Back to top
View user's profile Send private message
davidbryant
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jun 2020
Posts: 137
Location: Canyon Lake, Texas

PostPosted: Mon May 03, 2021 4:02 pm    Post subject: Reply with quote

Phoenix591 wrote:
Glibc pulls down that gcc tarball if the multilib useflag is enabled

Thanks for the explanation. I may patch that up sometime. I suppose the best place to fix that is in /etc/portage/make.conf, since I don't intend ever to use 32-bit code, and there may be other "multilib" packages / implications.
_________________
David Bryant
Canyon Lake, Texas
Back to top
View user's profile Send private message
Whissi
Retired Dev
Retired Dev


Joined: 12 Jan 2011
Posts: 222

PostPosted: Thu May 06, 2021 3:15 am    Post subject: Re: More on keeping /var/cache/distfiles clean Reply with quote

davidbryant wrote:
Just for my own edification and amusement, I ran "emerge -v --oneshot sys-kernel/genkernel" this morning, then studied the output. I see that the "genkernel" package pulls in 38 different tarballs (I may have miscounted). I also noticed that it sets up configuration files for 17 different machine architectures, which looks like overkill. I'm not complaining. Just observing. Thanks again.Tthis has been educational.
Genkernel-4.2.1 pulls in 32 additional tarball which will be stored in /usr/share/genkernel/distfiles. We call that folder "distfiles" because we mimic DISTDIR from portage but it isn't the same.
Some of them are optional. I.e. when you don't enable support for logical volume management in genkernel, LVM2, Boost and Thin-Provisioning-Tool sources are not used.

Why is genkernel not using USE flags to control that?

Because genkernel itself is an independent program, i.e. the additional tarballs are part of genkernel itself and you control usage at genkernel runtime.

Keep in mind that you can build kernel/initramfs for your host but you can also build kernel/initramfs for a different host which maybe requires a different configuration. You can even cross-compile...

That's also the reason why we provide support for 17 different architectures -- all these architectures are supported in Gentoo. But we are talking about 2.4MB in total. Not sure why this is causing you some headache. A typical generic kernel.config is ~210kb in average at the moment.

If you do not plan to re-install a package or have a fast and unlimited internet connection, you can regularly clean your DISTDIR (/var/cache/distfiles). Files in that directory are only used during emerge and your package manager would re-fetch any missing DISTFILE when needed (of course, this would require an internet connection).

For files installed by packages you don't like, you can use INSTALL_MASK to prevent your package manager from merging them to disk. However, be careful. If you remove a used/required file that way, programs maybe won't work anymore or behave different than expected. And genkernel for examples validates on start that all its "distfiles" are present (we do that to be able to error out as soon as possible so that you don't waste your time with building kernel just to find out that the last step failed because of a missing file).
_________________
Regards,
Whissi
Back to top
View user's profile Send private message
davidbryant
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jun 2020
Posts: 137
Location: Canyon Lake, Texas

PostPosted: Thu May 06, 2021 4:10 am    Post subject: Reply with quote

Thank you, Whissi. I'm not complaining. Just trying to understand. The only reasion I use genkernel at all is to build an initramfs file when I upfate the kernel. It was easier than using "dracut" from the git-go. I guess I ought to figure out how to use dracut every time.
_________________
David Bryant
Canyon Lake, Texas
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54234
Location: 56N 3W

PostPosted: Thu May 06, 2021 8:07 am    Post subject: Reply with quote

davidbryant,

You can also build a lean mean kernel agnostic initrd by hand and forget it.
My 2009 initrd still works but I've forgotten how to do it again.

The only reason that the initrd needs to be made every kernel it because it contains kernel modules that are required to mount the root filesystem.
If the initrd is userspace tools only and the kernel has all the bits needed built in, it will just work.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
davidbryant
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jun 2020
Posts: 137
Location: Canyon Lake, Texas

PostPosted: Thu May 06, 2021 12:15 pm    Post subject: Reply with quote

NeddySeagoon wrote:
davidbryant,

You can also build a lean mean kernel agnostic initrd by hand and forget it.
My 2009 initrd still works but I've forgotten how to do it again.

Too much coffee? 8O

NeddySeagoon wrote:
The only reason that the initrd needs to be made every kernel it because it contains kernel modules that are required to mount the root filesystem.
If the initrd is userspace tools only and the kernel has all the bits needed built in, it will just work.

I guess I should try that. I think I've got all the right stuff compiled into my kernel: I only support EXT4, FAT and variations, plus NTFS. No RAID, no LVM. Boot should only require VFAT (efi partition) and EXT4 (everything else). I'm a mite uncertain about "userspace tools". Thanks for the encouragement.
_________________
David Bryant
Canyon Lake, Texas
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54234
Location: 56N 3W

PostPosted: Thu May 06, 2021 12:32 pm    Post subject: Reply with quote

davidbryant,

Its unlikely you want kernel ntfs support. It used to destroy ntfs filesystem on writes.
That was 'fixed' by limiting write support to modifying existing files without changing the file size.

For full ntfs support use FUSE and ntfs3g. Kernel ntfs is OK for read only. Its also safe to have both.

Quote:
I guess I should try that. I think I've got all the right stuff compiled into my kernel

Testing is easy. Make yourself a new boot menu entry but without the initrd. Thats as well as, not in place of.

If works, you know you are good, if it breaks, you can boot the old menu entry.

The kernel does not read boot when the system boots so vfat support need not be built in. Its required by the kernel to write things to /boot.
boot contains the kernel, so the boot loader has to read boot before the kernel can be loaded to read boot.
The boot process contains lots of apparently circular dependencies like that that have to be broken.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum