| View previous topic :: View next topic |
| Author |
Message |
fpemud
Guru
Joined: 15 Feb 2012
Posts: 349
|
 Posted: Sat Oct 24, 2020 9:57 am Post subject: distfiles server for overlays? |
 |
|
For some packages, Gentoo distfiles server are way much faster than the original source.
But for packages in overlay, fetching from the slow original source is the only way.
Is there such services for Gentoo overlays?
Or is there any way I can build a local distfiles server for some specified overlays? |
|
| Back to top |
|
 |
ff11
l33t
Joined: 10 Mar 2014
Posts: 664
|
| Posted: Sat Oct 24, 2020 12:50 pm Post subject: Re: distfiles server for overlays? |
|
|
| fpemud wrote: | For some packages, Gentoo distfiles server are way much faster than the original source.
But for packages in overlay, fetching from the slow original source is the only way.
Is there such services for Gentoo overlays?
Or is there any way I can build a local distfiles server for some specified overlays? |
Yes and No. Although you can create one and use it normally, simply adding it to your GENTOO_MIRRORS, most of the overlays ebuilds use RESTRICT="mirror", which as the documentation says:
| Quote: | | The RESTRICT="mirror" setting should be used if we cannot legally mirror certain files; files will still be downloaded from the original locations. |
So it can be done, but it will only work for some cases. If you are the owner of the overlay, you can define a local mirror more easily, as you have control over the content of ebuilds.
_________________
| Proverbs 26:12 |
| There is more hope for a fool than for a wise man that are wise in his own eyes. |
* AlphaGo - The Movie - Full Documentary "I want to apologize for being so powerless" - Lee |
|
| Back to top |
|
|
fpemud
Guru
Joined: 15 Feb 2012
Posts: 349
|
| Posted: Sat Oct 24, 2020 1:42 pm Post subject: |
|
|
Thanks for replying.
Yes, RESTRICT="mirror" is a problem that needs to be solved, but for me it is a less important one.
What I mean by "build a local distfiles server for overlay" is running a server program that auto recognizes and download SRC_URI in overlay ebuilds when new ebuild file comes, and auto remove old files when old ebuild file disappears, just like what standard Gentoo distfiles server does.
It is not possible for anyone to manually maintain the files on distfiles server.
Is there any software can do these jobs? |
|
| Back to top |
|
|
ff11
l33t
Joined: 10 Mar 2014
Posts: 664
|
| Posted: Sat Oct 24, 2020 1:57 pm Post subject: |
|
|
| fpemud wrote: | Thanks for replying.
Yes, RESTRICT="mirror" is a problem that needs to be solved, but for me it is a less important one.
What I mean by "build a local distfiles server for overlay" is running a server program that auto recognizes and download SRC_URI in overlay ebuilds when new ebuild file comes, and auto remove old files when old ebuild file disappears, just like what standard Gentoo distfiles server does.
It is not possible for anyone to manually maintain the files on distfiles server.
Is there any software can do these jobs? |
I'm not aware of this software that can do this for you working with portage.
What I've seen is people using proxy servers to do something similar.
But I fail to understand what the issue really is. Because most of the ebuilds I have seen, have hosting on mirrors with reasonable speed. And if the local internet is slow, a proxy server would be better option, as it would solve not only the issue with portage download speed, but also with normal net browsing.
_________________
| Proverbs 26:12 |
| There is more hope for a fool than for a wise man that are wise in his own eyes. |
* AlphaGo - The Movie - Full Documentary "I want to apologize for being so powerless" - Lee |
|
| Back to top |
|
|
Einstok_Fair
n00b
Joined: 12 Aug 2020
Posts: 40
|
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54262
Location: 56N 3W
|
| Posted: Tue Apr 20, 2021 10:46 pm Post subject: |
|
|
Einstok_Fair,
It was. Its python2 only, so its been removed.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Wed Apr 21, 2021 12:12 am Post subject: |
|
|
"auto recognizes" ? Not sure what you mean. If you point SRC_URI to a vaild web location, portage will download it. During ebuild development "ebuild <ebuild file name> fetch" will do the same.
If you want a local repo, just run an apache server on some machine, assign a directory under the page and store the ebuilds there.
Of course, point SRC_URI to your web page. That's what I do for my home grown ebuilds. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21642
|
| Posted: Wed Apr 21, 2021 1:30 am Post subject: |
|
|
| As I read the request (from last October), fpemud wants a way to run a web server that hosts all the distfiles needed by an overlay, and automatically clean up the server as the overlay retires out old versions. As ff11 says, I do not see the use case. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Wed Apr 21, 2021 2:59 am Post subject: |
|
|
I have never knowingly erased a distfile. Since 2004 at least.
178G.
If one is really cramped for space, try deleting every more than a year old on New Year's Day. Or two tears old.
If you need to reinstall, they will just be fetched again.
So why do i keep all those files? I'm a pack rat. Might as well confess. |
|
| Back to top |
|
|
Einstok_Fair
n00b
Joined: 12 Aug 2020
Posts: 40
|
| Posted: Sun May 16, 2021 10:19 am Post subject: |
|
|
| Hu wrote: | | As ff11 says, I do not see the use case. |
I do.
Imagine that the overlay contains many ebuilds, which refering to different locations in SRC_URIs.
An author of overlay doesn't control these source download servers.
In order to provide reliable service, he (the author of overlay) should create a source distribution mirror for his ebuilds.
# emirrordist -v --mirror --repo einflay --distfiles /usr/portage/einflay/distfiles
[INFO] success: app-emulation/lxc-4.0.0 lxc-4.0.0.tar.gz added 1349007 bytes
...
[INFO] success: sys-kernel/gentoo-sources-5.10.14 genpatches-5.10-15.experimental.tar.xz added 17516 bytes
[INFO] finished in 108 seconds
[INFO] failed to fetch 2 files
[INFO] deleted 0 files
[INFO] deletion of 0 files scheduled
[INFO] added 21 files
[INFO] added 548998318 bytes total
see also https://bugs.gentoo.org/790500
Last edited by Einstok_Fair on Sun May 16, 2021 11:26 am; edited 1 time in total |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54262
Location: 56N 3W
|
| Posted: Sun May 16, 2021 10:42 am Post subject: |
|
|
Tony,
I didn't have enough space to keep much prior to mid 2006 but I have it all too. :)
It turned out to be a valuable resource for people updating old systems.
I don't keep the online copy current (deliberately) but every now and again I add to it.
It was 278G last time I looked.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Einstok_Fair
n00b
Joined: 12 Aug 2020
Posts: 40
|
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54262
Location: 56N 3W
|
| Posted: Sun May 16, 2021 12:38 pm Post subject: |
|
|
Einstok_Fair,
You can add any mirror you like into your list.
Portage works its way down the list of mirrors in GENTOO_MIRORS= when it want to retrieve any source tarball.
Adding an overlay mirror to the end of the list means that portage will try all the other mirrors in the list and fail before it gets to the overlay mirror.
Adding the overlay mirror to the top of the list means that the mirror will get hit with a lot of requests for files that it doesn't have.
When it runs of of mirrors, it uses SRC_URI from the ebuild.
Its unlikely that overlay sources would ever be added to the Gentoo mirror system as that would require someone to do due diligence on the overlay ebuilds. At least the licence, mirror and fetch restrictions would need to be checked.
If an overlay authors wanted that they would need to become at least a proxy maintainer, so the ebuilds were included in ::gentoo. Then they would not need to be in an overlay too.
To make a public mirror, make a directory called distfiles on your web server. Fill it with distfiles. Tell everyone about it.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Einstok_Fair
n00b
Joined: 12 Aug 2020
Posts: 40
|
| Posted: Sun May 16, 2021 12:55 pm Post subject: |
|
|
| NeddySeagoon wrote: | | Einstok_Fair |
Tell all that to the topic starter. It was his/her question.
I can't modify GENTOO_MIRRORS, because mirrorselect will rewrite it. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54262
Location: 56N 3W
|
| Posted: Sun May 16, 2021 1:16 pm Post subject: |
|
|
Einstok_Fair,
Sorry for misdirecting my comment.
mirrorselect writes to wherever you tell it. From memory, it has an -o option.
I have known buggy versions make a mess of make.conf so I always inspect its output before I merge it into make.conf by hand.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Sun May 16, 2021 1:55 pm Post subject: |
|
|
| NeddySeagoon wrote: |
Portage works its way down the list of mirrors in GENTOO_MIRORS= when it want to retrieve any source tarball.
Adding an overlay mirror to the end of the list means that portage will try all the other mirrors in the list and fail before it gets to the overlay mirror.
Adding the overlay mirror to the top of the list means that the mirror will get hit with a lot of requests for files that it doesn't have.
When it runs of of mirrors, it uses SRC_URI from the ebuild. |
For my original ebuilds I add RESTRICT="mirror" to the ebuild to avoid this search that is bound to fail. The SRC_URI is on my LAN Apache server so it's very fast, but sometimes I forget to restart the apache2 daemon. I don't do this for saved ebuilds that have left the official gentoo tree (sometimes just because "there is no interest"). I think that portage won;t search for anything that already exists in distfiles. Is this correct?
Way behind you in distfiles but they may have been inadvertently cleaned by eclean once. Also you have a few more years of Gentoo on me!
The tail of "ls -lt /usr/portage/distfiles" | Code: | -rw-rw-r-- 1 portage portage 411703 Apr 13 2006 gentoo-gdm-theme-r3.tar.bz2
-rw-rw-r-- 1 portage portage 26375 Feb 9 2006 obmenu-1.0.tar.gz
-rw-rw-r-- 1 portage portage 508654 Jan 23 2006 libast-0.7.tar.gz
-rw-rw-r-- 1 portage portage 7480 Dec 6 2005 IO-String-1.08.tar.gz
-rw-rw-r-- 1 portage portage 185858 Nov 20 2005 idesk-0.7.5.tar.bz2
-rw-rw-r-- 1 portage portage 3197171 Apr 18 2005 cvs-1.12.12.tar.bz2
-rw-rw-r-- 1 portage portage 214934 Nov 7 2004 ogmtools-1.5.tar.bz2
-rw-rw-r-- 1 portage portage 24360787 Jul 11 2004 intlfonts-1.2.1.tar.gz
-rw-rw-r-- 1 portage portage 10614 Feb 13 2004 sux-1.0.1.tar.gz
-rw-rw-r-- 1 portage portage 105641 Jan 21 2004 shash-0.2.6.tar.gz
-rw-rw-r-- 1 portage portage 207697 Mar 24 2003 hfsutils-3.2.6.tar.gz
-rw-rw-r-- 1 portage portage 391028 Apr 26 2001 aalib-1.4rc5.tar.gz
-rw-rw-r-- 1 portage portage 10769 Jan 4 2001 lsb-release-1.4.tar.gz
-rw-rw-r-- 1 portage portage 49367 Jul 27 1999 xfreecell-1.0.5b.tgz
-rw-rw-r-- 1 portage portage 1212278 Apr 3 1999 MSNumbers.gz
|
|
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54262
Location: 56N 3W
|
| Posted: Sun May 16, 2021 2:19 pm Post subject: |
|
|
Tony0945,
Correct. If a distfile is already available portage will not try to fetch it.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Sun May 16, 2021 3:02 pm Post subject: |
|
|
| NeddySeagoon wrote: | Tony0945,
Correct. If a distfile is already available portage will not try to fetch it. |
Since I now share distfiles via Samba, I probably could just throw the distfile there and forget the "mirror" but it seems like a sloppy way to go. Besides, if I ever publish them I'll need to have them on a web host. I have one but haven't done anything with it except run my hosted e-mail. |
|
| Back to top |
|
|
|
Portage & Programming
