Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Changing the default directory for compiled binaries
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
frozenpenguin
n00b
n00b


Joined: 04 Feb 2024
Posts: 15

PostPosted: Sun Mar 31, 2024 4:44 am    Post subject: Changing the default directory for compiled binaries Reply with quote

I am trying to emerge a package and want the emerged package to be stored on a USB. However, I do not know which variable to change in portage to make portage behave as I wish. I also do not know how to let the terminal know the commands associated with the package if the package isn't in the default package directory. Any assistance will be appreciated.
Back to top
View user's profile Send private message
Josef.95
Advocate
Advocate


Joined: 03 Sep 2007
Posts: 4558
Location: Germany

PostPosted: Sun Mar 31, 2024 6:40 am    Post subject: Reply with quote

The variable for this is PKGDIR
this is documented in "man make.conf"
Back to top
View user's profile Send private message
frozenpenguin
n00b
n00b


Joined: 04 Feb 2024
Posts: 15

PostPosted: Sun Mar 31, 2024 8:45 am    Post subject: Reply with quote

From what I understand, PKGDIR is where binary packages downloaded off the internet are stored. I am trying to find the default directory where the packages compiled from source on my machine are stored.
Back to top
View user's profile Send private message
flexibeast
Guru
Guru


Joined: 04 Apr 2022
Posts: 324
Location: Naarm/Melbourne, Australia

PostPosted: Sun Mar 31, 2024 9:15 am    Post subject: Reply with quote

The variables used during the build-and-merge process are documented in ebuild(5) man page and in the devguide. There's also the ebuild(1) man page which describes the various phases involved.

Depending on exactly what you want on the USB drive (a single binary package? a directory-and-file-tree, given that a single package might install more than one binary, and possibly various libraries and other supporting files as well?), emerge(1)'s `--buildpkg*` options, and the the quickpkg(1) program, might be relevant.

If none of the above helps, please provide more information about exactly what you require.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21651

PostPosted: Sun Mar 31, 2024 3:10 pm    Post subject: Re: Changing the default directory for compiled binaries Reply with quote

frozenpenguin wrote:
I am trying to emerge a package and want the emerged package to be stored on a USB. However, I do not know which variable to change in portage to make portage behave as I wish. I also do not know how to let the terminal know the commands associated with the package if the package isn't in the default package directory.
The third sentence leads me to believe you are trying to place the installed program at a prefix other than the typical /usr (from-source) / /opt (prebuilt). The answer to your problem there is to modify your shell's search path, traditionally $PATH, to include the directory of interest. However, that is only useful after you change the prefix, for which there is no general solution if you want Portage to change the path only for this one package.
Back to top
View user's profile Send private message
frozenpenguin
n00b
n00b


Joined: 04 Feb 2024
Posts: 15

PostPosted: Sun Apr 07, 2024 1:04 pm    Post subject: Reply with quote

My idea is to install a package on a USB. When the USB is mounted, the system should treat packages installed on the USB as if the packages were installed on the system's partition. Only the packages binaries and dependencies which are required by those but not by any packages on the system should be installed there, as I do not want packages I periodically use installed on my computer as I wish to prevent storage issues on my system before they arise.
Back to top
View user's profile Send private message
eschwartz
n00b
n00b


Joined: 29 Oct 2023
Posts: 17

PostPosted: Sun Apr 07, 2024 2:25 pm    Post subject: Reply with quote

frozenpenguin wrote:
My idea is to install a package on a USB. When the USB is mounted, the system should treat packages installed on the USB as if the packages were installed on the system's partition. Only the packages binaries and dependencies which are required by those but not by any packages on the system should be installed there, as I do not want packages I periodically use installed on my computer as I wish to prevent storage issues on my system before they arise.


This will not work because software tends to embed the filesystem paths of the resources they use. Also, shared libraries have to be integrated into the system loader.

If you wish for packages to be installed to a location other than /usr or /bin or /lib, such as a USB drive, then you need to compile the packages specific for that purpose, including nontrivial approaches to ensure they run portably. It is much the same requirements as imposed by Windows application bundles (in fact, Windows application bundles already have to do this, thus *they work by default on a USB drive*).

I think you have two options:

- don't borrow trouble from the future. Simply install the programs you need, and deal with storage issues *if* they ever come up, which they may not.

- buy a bigger hard drive
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21651

PostPosted: Sun Apr 07, 2024 2:45 pm    Post subject: Reply with quote

frozenpenguin: what are the specifications for the system where you want to do this? How much storage do you think you need for these programs you intend to relocate to USB? I concur with eschwartz that at this stage, it looks like you are trying to avoid a problem that may never come to pass, and are definitely setting yourself up for additional hassle in the attempt.
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Sun Apr 07, 2024 3:01 pm    Post subject: Reply with quote

frozenpenguin wrote:
My idea is to install a package on a USB. When the USB is mounted, the system should treat packages installed on the USB as if the packages were installed on the system's partition. Only the packages binaries and dependencies which are required by those but not by any packages on the system should be installed there, as I do not want packages I periodically use installed on my computer as I wish to prevent storage issues on my system before they arise.


What you have in mind is possible, however it is advance system admin topic. Have you thought about how you going to maintain such setup? For example when you perform a update of some system level (i.e. a xz library) and yet the USB was NOT connected at the time of update but later you wish to use a program that is on the USB that require use of the older version of system library, Do you want to perform another update just to use the program on USB?

Having said that one possible setup for your idea is using a union filesystem (OverlayFS), You create a layer on USB and map it into system hierarchy. This idea does not solve the system admin issue I made above, it just simplify the installation procedure.

Gentoo is good for customisation, however you also need to take the maintenance of the customisation info account.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20067

PostPosted: Sun Apr 07, 2024 3:50 pm    Post subject: Reply with quote

Isn't using the USB simply a sneakernet method?

Am I missing something? Why would that be meaningfully different than downloading them from a network connection?
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21651

PostPosted: Sun Apr 07, 2024 6:20 pm    Post subject: Reply with quote

As I understand the thread, OP is not merely using a USB disk as an alternative to downloading the source files over an Ethernet/Wifi connection. OP wants to use the USB as a conditionally present seamlessly available filesystem hosting finished runnable programs. For example, with USB absent, the system can only run a lightweight desktop, like Xfce4. Plug in the USB, and suddenly you can also choose GNOME or KDE Plasma as desktop options.
Back to top
View user's profile Send private message
frozenpenguin
n00b
n00b


Joined: 04 Feb 2024
Posts: 15

PostPosted: Mon Apr 08, 2024 4:36 am    Post subject: Reply with quote

My current setup on my computer is around 85GB for linux and around 383GB for windows. Now, I could add more space for linux except windows isn't willing to give any more space up from its partition using its disk management tool. I also don't want to delete windows as I may need it again in the future.
As I only want to install packages I rarely use, packages on the USB will only be updated when their use is required. The reason these packages are rarely used is because I never use these packages unless it is required for certain features(such as some sites requiring use of a certain browsers for certain functionality).
Back to top
View user's profile Send private message
e8root
n00b
n00b


Joined: 09 Feb 2024
Posts: 72

PostPosted: Mon Apr 08, 2024 3:05 pm    Post subject: Reply with quote

To allow reducing size of Win10/11 install you need to disable pagefile and hibernation, restart and then defragment your drive.
I would also recommend using folder compression on programs and files you don't really use all that often. Of course already compressed files do not gain much from compressing them again but on average gain can be quite good for typical programs. Most gains are on things like large number of small text files.

As for what you want to achieve in OP I am not sure how to achieve it, or at least in automated way as manually you could probably move some files and perhaps use LD_LIBRARY_PATH to move some .so files to other locations. Doing it manually can quickly lead to issues if you don't know exact dependencies. Automating this... should be doable but sounds like a lot of effort hacking Portage.

Probably (very...) better solution would be to use something like Flatpak or Appimage packages. Especially the latter one you should be able to just put on USB and run as portable programs. Flatpaks I am not sure but should be doable. There are also Snap packages but we neither use nor talk about Snap packages ; ) But seriously it seems you need SystemD which you might have or not and I personally am lest interrested in these even if I myself do use SystemD.

There are also other more source-based solutions like Nix and Guix package managers. The latter one is not free to use anything not free so might not have all programs you would want. Both work in similar way and allow you to build your packages. By the the looks of it it should be possible to achieve having Nix/Guix programs in separate mounted directory without needing anything on your base system. Downside it you would then need to learn new build system and there is some disk space overhead as you will need all the build tools like compilers.

Lastly it is possible to chroot in to existing Linux install and run applications from that. It can be Gentoo and in this case its like separate Gentoo environment which can be system which might not even be bootable/usable standalone but could easily be made as standalone bootable system or some other distro. It requires some tinkering if you want to run GUI apps or whole X11/Wayland servers but it is doable. It is of course abusing chroot and issues are expected, especially if you run something like Ubuntu. There for example Snaps don't work and cannot be made to work. Otherwise Especially for Gentoo itself it might be a good idea to have such system as it can be useful for testing, troubleshooting and such.

Then there are other solutions like LXC or docker.
Depending on your specific needs and compatibility either of those or others which I mentioned and didn't mention can be usable.

Most sane and "just works" solution is however something like Appimage.
Maybe create Appimage using Gentoo? Download program you need from trusted source and just run it? Lots of different ways to do it...
_________________
Unix Wars - Episode V: AT&T Strikes Back
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9537
Location: beyond the rim

PostPosted: Mon Apr 08, 2024 3:25 pm    Post subject: Reply with quote

I'd first check why you even need more than 50 GB for your linux install. Most packages only need a couple megabytes at most, so redirecting them to a USB drive wouldn't really help with space issues in the first place.
Try running
Code:
du -sh /* /usr/* /var/*

to check which filesystem locations take up the most space. I suspect there will be some candidates for cleaning up.
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1553

PostPosted: Mon Apr 08, 2024 3:57 pm    Post subject: Reply with quote

Genone wrote:
I'd first check why you even need more than 50 GB for your linux install.


What about some everyday user data. The time I lived with 85Gb is not that far. It's not enough for a desktop system. I was constantly cleaning up tmp and dist files what not to find space to decompress something.

Best Regards,
Georgi
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3348
Location: Rasi, Finland

PostPosted: Mon Apr 08, 2024 7:03 pm    Post subject: Reply with quote

I'd like to add to @Genone's comment that if you want to find "where all the stuff is" then try sys-fs/ncdu.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54274
Location: 56N 3W

PostPosted: Mon Apr 08, 2024 7:24 pm    Post subject: Reply with quote

Put distfiles, packages and /usr/src onto USB.
They are only needed at update time.
Optionally, ./repos can go to USB too.
If you must have ::gentoo available all the time, use a squashfs snapshot to save space.

Everything to run Gentoo in always available. The bits required for maintenance can be unplugged when not needed.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9537
Location: beyond the rim

PostPosted: Tue Apr 09, 2024 10:28 am    Post subject: Reply with quote

logrusx wrote:
Genone wrote:
I'd first check why you even need more than 50 GB for your linux install.


What about some everyday user data. The time I lived with 85Gb is not that far. It's not enough for a desktop system. I was constantly cleaning up tmp and dist files what not to find space to decompress something.

Best Regards,
Georgi


Was kind of my point that most space is likely taken up by user data, downloaded files, cache data and so on. If you're short on space with a >50GB install, redirecting a couple hundred megabytes of installed packages to another storage device is at best a short-lived temporary workaround. Cleaning up or outsourcing distfiles, kernel sources and build artifacts will likely yield much bigger savings.
Only exception would be if those redirected packages include behemoths that actually install gigabytes of data (e.g. some games). But then it might be worth checking their install locations and maybe simply mount /opt or whatever on USB.

TL;DR: Before working on a (perceived) solution, identify the cause of the problem.
Back to top
View user's profile Send private message
frozenpenguin
n00b
n00b


Joined: 04 Feb 2024
Posts: 15

PostPosted: Thu Apr 11, 2024 1:20 pm    Post subject: Reply with quote

So I got the source files on my flash drive, however when I try to install the package I get this:
Code:
Emerging (1 of 1) www-client/chromium-120.0.6099.129::gentoo
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
!!! File .layout.conf.distfiles.gentoo.org isn't fetched but unable to get it.
!!! File chromium-120.0.6099.129.tar.xz isn't fetched but unable to get it.
 * Fetch failed for 'www-client/chromium-120.0.6099.129', Log file:
 * '/var/tmp/portage/www-client/chromium-120.0.6099.129/temp/build.log'

>>> Failed to emerge www-client/chromium-120.0.6099.129, Log file:

The flash drive's filesystem is exfat and has chromium source files compressed in the .zip format. These files were downloaded off the chromium github mirror without using git clone as getting the source via portage isn't possible as google is blocked where I currently am. They were gotten from my phone(as it has a vpn) then moved to the flash drive.
Back to top
View user's profile Send private message
pingtoo
l33t
l33t


Joined: 10 Sep 2021
Posts: 926
Location: Richmond Hill, Canada

PostPosted: Thu Apr 11, 2024 1:32 pm    Post subject: Reply with quote

frozenpenguin wrote:
So I got the source files on my flash drive, however when I try to install the package I get this:
Code:
Emerging (1 of 1) www-client/chromium-120.0.6099.129::gentoo
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
!!! File .layout.conf.distfiles.gentoo.org isn't fetched but unable to get it.
!!! File chromium-120.0.6099.129.tar.xz isn't fetched but unable to get it.
 * Fetch failed for 'www-client/chromium-120.0.6099.129', Log file:
 * '/var/tmp/portage/www-client/chromium-120.0.6099.129/temp/build.log'

>>> Failed to emerge www-client/chromium-120.0.6099.129, Log file:

The flash drive's filesystem is exfat and has chromium source files compressed in the .zip format. These files were downloaded off the chromium github mirror without using git clone as getting the source via portage isn't possible as google is blocked where I currently am. They were gotten from my phone(as it has a vpn) then moved to the flash drive.


How is this relate to the topic you are talking about?
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9537
Location: beyond the rim

PostPosted: Fri Apr 12, 2024 10:07 am    Post subject: Reply with quote

frozenpenguin wrote:
The flash drive's filesystem is exfat

Exfat doesn't support the posix ACL model, so it isn't really suited for use under Linux for anything other than storing user data. The error you see is because under Linux FAT filesystems by default can only be modified by root or the specific user account that mounted the filesystem.
If you want to use a USB drive to store anything but user data, you'll have to use a posix compatible filesystem (like ext4). But of course then the stick will no longer be accessible by Windows without special software.
Back to top
View user's profile Send private message
frozenpenguin
n00b
n00b


Joined: 04 Feb 2024
Posts: 15

PostPosted: Mon Apr 15, 2024 11:42 pm    Post subject: Reply with quote

Quote:
How is this relate to the topic you are talking about?

I just thought that it would be relevant as as someone suggested putting source on the flash drive. I'll put it in a new thread next time.
Quote:
Exfat doesn't support the posix ACL model, so it isn't really suited for use under Linux for anything other than storing user data. The error you see is because under Linux FAT filesystems by default can only be modified by root or the specific user account that mounted the filesystem.
If you want to use a USB drive to store anything but user data, you'll have to use a posix compatible filesystem (like ext4). But of course then the stick will no longer be accessible by Windows without special software.

I mounted the flash drive as root so from this, I take it to mean that I can use it without problems as root. However, I get this problem when I try to emerge a package that has the DISTDIR variable set to point to the flash drive.

On another note, I still wish to know how to change the default directory or know where it is in case [/quote]I want packages to run on an embedded device where disk space is limited.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21651

PostPosted: Tue Apr 16, 2024 1:00 am    Post subject: Reply with quote

You originally asked to use a USB drive. I, and likely others, took this to mean a removable block storage device backed by either an SSD or a mechanical drive, not a USB flash stick. Regardless, I (and, again, likely others) assumed you would use a proper POSIX filesystem on this storage device. Whether you try to store distfiles or a compiled package on the external device, you will cause yourself considerable extra hassle if you use a filesystem that cannot represent standard POSIX features. ExFAT and NTFS are the most well known unsuitable filesystems, but there might be others.

As I wrote at the start of the thread, there is no generalized mechanism to change the prefix for individual packages to be other than the Portage default. The Gentoo Prefix project can relocate the entire resulting Gentoo system under a prefix, but that's not what you requested.

The general advice for dealing with systems that have limited internal storage is to use existing functionality to trim down which files the package installs, such as by eliminating unnecessary translations, stripping debug symbols, discarding documentation, and so on. If storage is very limited, use ROOT to install a non-self-sufficient system. Excluding compilation tools can save you substantial space.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9537
Location: beyond the rim

PostPosted: Tue Apr 16, 2024 8:04 am    Post subject: Reply with quote

frozenpenguin wrote:
I mounted the flash drive as root so from this, I take it to mean that I can use it without problems as root. However, I get this problem when I try to emerge a package that has the DISTDIR variable set to point to the flash drive.

Yes, you as root can use it. But even if you launch emerge as root, it will by default switch to another user account and therefore can't access the root-only filesystem. Also system calls like chown (to change the owner of a file) can't work on a FAT file system, the file ownership instead is determined by the mount options. That is not compatible with portage (and most other Linux/Unix applications that make use of ACLs in some way). My previous statement may have been a bit unclear regarding that.

Quote:
On another note, I still wish to know how to change the default directory or know where it is in case I want packages to run on an embedded device where disk space is limited.

There is no such thing as a "default install directory". I assume you get that concept from windows and its C:\Program Files\<appname> pattern. This doesn't apply to Linux. There are features like prefix or chroot, but they work quite different than "install location" on Windows.
In practice most package files will end up somewhere in /usr/, but that is not something you want to mess with as you can easily break your system doing so.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum