How do I fetch the sources for the entire portage tree?

creaker · l33t Joined: 14 Jul 2012 Posts: 651

Is it possible? If so, how large it may be?
Just downloading a mirror is not a good idea, there are a lot of sources that placed at different sites.
If it is possible, how do I resolve the links that points out of mirror (third party sites)?

BitJam · Posted: Thu Dec 05, 2013 10:37 am Post subject:

I would use Perl to search through all the .ebuild files, grabbing all urls and sending them to stdout. Redirect this output to a file and then use something like wget to download everything from that file. There is a bit more to it than this because the source urls usually contain variables.

Another approach would be to source each ebuild from inside a Bash script and print out the SRC_URI. You can define all the standard functions as no-ops:

John R. Graham · Posted: Thu Dec 05, 2013 10:50 am Post subject:

Too much work. :wink:

i92guboj · Posted: Thu Dec 05, 2013 10:56 am Post subject:

Why?

And anyway... You can't. There are many packages that require manual download or some other procedure to accept a license, due to legal/distribution issues.

If what you want is to fetch the files that you will need for a given installation on a machine where you don't have network access, the best bet is to run the emerge command in that machine using something like this:

szatox · Advocate Joined: 27 Aug 2013 Posts: 3133

What is the purpose of downloading everything?
you could for example clone portage repo with wget -m -k [-C] or use emerge --fetchonly fed with list of packages. And that list can be made using your /usr/portage.
and make sure you have a lot of space and good network before you attempt that.Oh, and don't do that at home. ISP will most likely limit your speed. Cheap links are cheap for a reson.

creaker · l33t Joined: 14 Jul 2012 Posts: 651

Thanks for replies.
In fact I do not need all the possible sources. I have to install a system at machine without network access. The thing is I do not know, what I'll be install. But I'm pretty sure it excludes gnome, lxde, fvwm, openbox, windowmaker and other exotic stuffs.
It should be full kde and xfce branches, full Xorg. Populating the distfiles directory with emerge --fetchonly seems as reasonable solution, but I'm not sure, will it fetch all the branch with e.g. emerge --fetchonly kde-meta? Does kde-meta contains all the kde branch?
The target machine is too far from my home, so I would like to have as much packages as possible ready-to-use (e.g. under distfiles directory).

i92guboj · Posted: Thu Dec 05, 2013 1:58 pm Post subject:

You can throw in --empty-tree and all the dependencies will be fetch as if the system hadn't a single package installed.

Note however that, even then, there might be (hopefully minor) differences in the dep tree between both systems, based on USE flags, profile, architecture and a few other parameters.

oh, and note also that this won't work at all unless both portage tress are identical, overall if you are in ~arch. Portage now is not the same than portage five minutes after now. So, a thing you might want to consider is using a portage snapshot, the same one for both machines.

creaker · l33t Joined: 14 Jul 2012 Posts: 651

ОK, portage tree will be identical since target machine has no internet acces. I can just copy my local tree for which I'll fetch packages.
Regarding profiles: what is better for fetching more stuffs - set profile to "desktop" or to "kde"? I think latter should has more dependencies, am I right?

i92guboj · Posted: Thu Dec 05, 2013 4:28 pm Post subject:

schorsch_76 · Guru Joined: 19 Jun 2012 Posts: 450

John R. Graham · Posted: Thu Dec 05, 2013 6:18 pm Post subject:

Sorry; middle of the night on the iPad. The xargs version does work, but I missed a needed parameter. Edited above.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3133

Why won't you install everything in some directory on your local machine ( mkdir newgentoo; cd newgentoo,; rsync -a /usr/portage usr/portage; mount --bind /proc proc; chroot ./; env-update && . /etc/profile && PS1=(newgentoo)$PS1 )
Once you're done with messing around, simply take image of this directory with you and rsync -a newgento <whatever the target actualy is>
(and then run grub-install

)
I've used this way successfully when I needed to install gentoo in some place with really slow net. Actualy I also use this pretty often plaing with virtual machines.

Excpet that I mount --bind portage too instead of cloning it in this latter case

If you fail with some hardware (unfortunately gentoo-sources still have atheros drivers disabled by default :/ ) kernel is most likely going to be the only thing that needs recompilation.
Xorg will by default pull allmost all video drivers possible, so this should just work.

creaker · l33t Joined: 14 Jul 2012 Posts: 651

@szatox:
I'm not a target machine user, so I do not know which software end user may wish to have installed. Thus, I can't make preinstallation, as well as I can't just pack my system into stage4 archive.

@i92guboj:
Hardware shouldn't be a problem causer, I know CPU and MoBo model, video card model - they are very similar to my desktop, moreover I assembled this box myself few months ago, so I do not expect a hardware issues. Keyboard and mouse are both simple usb devices that can be handled by usb hid.

OK, I'll try to fetch packages.
Thanks again.

dol-sen · Posted: Fri Dec 06, 2013 4:11 am Post subject:

I'm not sure if it is installed in regular portage releases, but it is with portage-9999

Genone · Posted: Fri Dec 06, 2013 7:25 am Post subject:

Are you sure you want to install Gentoo for someone else on a computer without network access? Sounds like a recipe for annoying support calls.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3133

Don't buy ring until you have girlfriend.
If the user doesn't know what he wants, he will be just as happy with ubuntu - or perhaps even more happy. If he wants to stick to gentoo, at least make him produce list of what he wants (and then make him pay extra for changes

)
There is no point in even starting if you don't know what you're doing. Too much effort, too little result.

jamapii · l33t Joined: 16 Sep 2004 Posts: 637

I'm trying to do this.

I simply do

emerge -fuD --newuse some_big_packages

The problem is, it doesn't work. There are lots of circular dependencies, bogus conflicts (The following packages depend on X: none) etc. Maybe it can be resolved with autounmask-write, but I don't want to mess with the configuration just for fetching files. Or by not just fetching, but installing, which is not what fetchonly means.

So, what is needed to make "emerge -f" work?

John R. Graham · Posted: Thu Jun 05, 2014 6:18 pm Post subject:

You should know that installing a new package doesn't work very well unless your system is up to date. If that condition is met, then --fetchonly works very well. If your system is up to date and you're still having issues, then that might very well be a bug in the Portage tree. Please post actual message and we'll help.

If you're trying to just fetch everything (but not build anything), see the method I posted earlier.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.