Portage local distfiles cache

Message

gtbX · Post by **gtbX** » Sun Jul 19, 2020 5:10 pm

I used to use http-replicator to provide a local cache for portage distfiles on my networks. It was last-rited some time ago, and I haven't found many reasonable alternatives.

I initially tried using net-proxy/squid in its default configuration - this worked in that I was able to use it as a proxy, but it seems kind of heavyweight, and I can't tell if it is actually caching anything.

The wiki page on the subject https://wiki.gentoo.org/wiki/Local_distfiles_cache is rather sparse on details. It suggests using apt-cacher-ng, and even configuring it so that it doesn't store multiple copies of each distfile when you run emerge on the host system (which I had http-replicator configured to do, if in a less-than-ideal way). However, it appears that this package too is on its way to retirement (https://bugs.gentoo.org/631878). The wiki also suggests nginx can be configured for this purpose, but I'm already running apache on these systems and I don't need yet another web server.

Is this even a concern with modern internet connectivity? I only have a handful of machines the cache would serve.

Thanks!

mike155 · Post by **mike155** » Sun Jul 19, 2020 8:27 pm

I have a NFS server that exports the volumes repos, distfiles and binpkgs. I mount those volumes on all Linux machines.

Ant P. · Post by **Ant P.** » Sun Jul 19, 2020 8:28 pm

Just share $DISTDIR on NFS. You can do the same for $PORTDIR too.

gtbX · Post by **gtbX** » Sun Jul 19, 2020 8:46 pm

Is there a good guide for setting up NFS securely? I last played with it more than 10 years ago, and was never comfortable with not having any authentication or encryption enabled.

fpemud · Post by **fpemud** » Mon Jul 20, 2020 4:30 am

I have written a project mirrors and mirrors-plugin-gentoo for doing this. ebuild files is located in fpemud-overlay.
Currenly it uses rsync as the wiki suggests.

mmogilvi · Post by **mmogilvi** » Mon Jul 20, 2020 4:43 am

xedakini has done a lot of work on http-replicator; maybe it could be re-added to portage, although that would probably require an official maintainer (or at least proxy maintainer). More information from viewtopic-t-1097768-postdays-0-postorde ... rt-25.html :

Status: I haven't had time to try it myself, but xedakini has apparently done a lot of upstream maintenance of version 4, and "sort of" became the upstream maintainer ( https://github.com/xedakini/replicator/pull/5 ). A quick glance suggests it already has most desired updates, including: python3 support, repcacheman updates, fixing http parsing/security concerns, and incorporating support for "X-Unique-Cache-Name" ([bug]442874[/bug]) into version 4.

There is now apparently new version 4.0alpha4 release at https://github.com/xedakini/replicator/ ... v4.0alpha4

However, xedakini is not committing to maintaining it longer term, and it has not been re-added back into portage. (It was removed in Feb/Mar via [bug]705606[/bug], although there have been a few comments added to it since then.)

szatox · Post by **szatox** » Mon Jul 20, 2020 4:51 pm

gtbX wrote:Is there a good guide for setting up NFS securely? I last played with it more than 10 years ago, and was never comfortable with not having any authentication or encryption enabled.

U S E V P N F O R T H A T

I think NFS4 supports everything you mentioned there, but it's both, application-specific AND more complicated to setup, so... Just use a VPN

Wireguard is pretty neat. It cuts some corners, but it is good enough for most ("most" not "all" - mind the corners) applications, and it serves it's designed purpose really well.