Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How do I fetch the sources for the entire portage tree?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
creaker
l33t
l33t


Joined: 14 Jul 2012
Posts: 651

PostPosted: Thu Dec 05, 2013 8:35 am    Post subject: How do I fetch the sources for the entire portage tree? Reply with quote

Is it possible? If so, how large it may be?
Just downloading a mirror is not a good idea, there are a lot of sources that placed at different sites.
If it is possible, how do I resolve the links that points out of mirror (third party sites)?
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2455
Location: Silver City, NM

PostPosted: Thu Dec 05, 2013 10:37 am    Post subject: Reply with quote

I would use Perl to search through all the .ebuild files, grabbing all urls and sending them to stdout. Redirect this output to a file and then use something like wget to download everything from that file. There is a bit more to it than this because the source urls usually contain variables.

Another approach would be to source each ebuild from inside a Bash script and print out the SRC_URI. You can define all the standard functions as no-ops:
Code:
name() { :;}

The idea would be to let Bash handle the variable substitution for you.

I'm updating 60 packages now. The download will be 314 Meg. The command:
Code:
find /usr/portage -name "*.ebuild" | wc
says there are about 37,000 ebuild files in Portage so I'd expect the size of your download to be roughly:
Code:
 37,000/60 * 314 Meg = 193 Gigabytes
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 8410
Location: Somewhere over Atlanta, Georgia

PostPosted: Thu Dec 05, 2013 10:50 am    Post subject: Reply with quote

Too much work. :wink:
Code:
find /usr/portage -name '*.ebuild' | xargs -n1 -I{} ebuild '{}' fetch
and done.

- John

Edit: Corrected after flaw reported by schorsch_76.
_________________
I can confirm that I have received between 0 and 999 National Security Letters.


Last edited by John R. Graham on Thu Dec 05, 2013 6:20 pm; edited 1 time in total
Back to top
View user's profile Send private message
i92guboj
Moderator
Moderator


Joined: 30 Nov 2004
Posts: 10219
Location: Córdoba (Spain)

PostPosted: Thu Dec 05, 2013 10:56 am    Post subject: Reply with quote

Why?

And anyway... You can't. There are many packages that require manual download or some other procedure to accept a license, due to legal/distribution issues.

If what you want is to fetch the files that you will need for a given installation on a machine where you don't have network access, the best bet is to run the emerge command in that machine using something like this:

Code:
emerge -pf package_name > file_list.txt


or

Code:
emerge -puDvNf @world > file_list.txt


In either case, "file_list.txt" will hold the list of files that portage would download for the intended purpose. You can later use that list with any download manager (wget will do) in some other machine to fetch the files, and later put then on $DISTDIR in the networkless machine, so portage will see them and use them.
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 673

PostPosted: Thu Dec 05, 2013 11:07 am    Post subject: Reply with quote

What is the purpose of downloading everything?
you could for example clone portage repo with wget -m -k [-C] or use emerge --fetchonly fed with list of packages. And that list can be made using your /usr/portage.
and make sure you have a lot of space and good network before you attempt that.Oh, and don't do that at home. ISP will most likely limit your speed. Cheap links are cheap for a reson.
Back to top
View user's profile Send private message
creaker
l33t
l33t


Joined: 14 Jul 2012
Posts: 651

PostPosted: Thu Dec 05, 2013 11:54 am    Post subject: Reply with quote

Thanks for replies.
In fact I do not need all the possible sources. I have to install a system at machine without network access. The thing is I do not know, what I'll be install. But I'm pretty sure it excludes gnome, lxde, fvwm, openbox, windowmaker and other exotic stuffs.
It should be full kde and xfce branches, full Xorg. Populating the distfiles directory with emerge --fetchonly seems as reasonable solution, but I'm not sure, will it fetch all the branch with e.g. emerge --fetchonly kde-meta? Does kde-meta contains all the kde branch?
The target machine is too far from my home, so I would like to have as much packages as possible ready-to-use (e.g. under distfiles directory).
Back to top
View user's profile Send private message
i92guboj
Moderator
Moderator


Joined: 30 Nov 2004
Posts: 10219
Location: Córdoba (Spain)

PostPosted: Thu Dec 05, 2013 1:58 pm    Post subject: Reply with quote

You can throw in --empty-tree and all the dependencies will be fetch as if the system hadn't a single package installed.

Note however that, even then, there might be (hopefully minor) differences in the dep tree between both systems, based on USE flags, profile, architecture and a few other parameters.

oh, and note also that this won't work at all unless both portage tress are identical, overall if you are in ~arch. Portage now is not the same than portage five minutes after now. So, a thing you might want to consider is using a portage snapshot, the same one for both machines.
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
creaker
l33t
l33t


Joined: 14 Jul 2012
Posts: 651

PostPosted: Thu Dec 05, 2013 2:27 pm    Post subject: Reply with quote

ОK, portage tree will be identical since target machine has no internet acces. I can just copy my local tree for which I'll fetch packages.
Regarding profiles: what is better for fetching more stuffs - set profile to "desktop" or to "kde"? I think latter should has more dependencies, am I right?
Back to top
View user's profile Send private message
i92guboj
Moderator
Moderator


Joined: 30 Nov 2004
Posts: 10219
Location: Córdoba (Spain)

PostPosted: Thu Dec 05, 2013 4:28 pm    Post subject: Reply with quote

creaker wrote:
ОK, portage tree will be identical since target machine has no internet acces. I can just copy my local tree for which I'll fetch packages.
Regarding profiles: what is better for fetching more stuffs - set profile to "desktop" or to "kde"? I think latter should has more dependencies, am I right?


The desktop or kde profiles should either be ok for your purpose, so I'd say go with either of those. One piece of advice is that, if the place is far enough, you should locate some public internet service near there, just in case you need to fetch a couple of files or something. That or a mobile phone that can do the job.

Note that even if you have all the files to get kde working right, the hardware [might] will be different in their system. That means that the Xorg video drivers will be different, the input drivers might also differ, etc. etc. etc.

In a related note, you will save yourself some problem if you can get them to boot a linux livecd and send you the output of "lspci".
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
schorsch_76
Apprentice
Apprentice


Joined: 19 Jun 2012
Posts: 242

PostPosted: Thu Dec 05, 2013 5:26 pm    Post subject: Reply with quote

John R. Graham wrote:
Too much work. :wink:
Code:
find /usr/portage -name '*.ebuild' | xargs -n1 ebuild '{}' fetch
and done.

- John

Sorry, i need to correct you ;)

Code:

find /usr/portage -name *.ebuild -exec ebuild {} fetch \;


Your version did not work ;)
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 8410
Location: Somewhere over Atlanta, Georgia

PostPosted: Thu Dec 05, 2013 6:18 pm    Post subject: Reply with quote

Sorry; middle of the night on the iPad. The xargs version does work, but I missed a needed parameter. Edited above.

- John
_________________
I can confirm that I have received between 0 and 999 National Security Letters.
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 673

PostPosted: Thu Dec 05, 2013 6:54 pm    Post subject: Reply with quote

Why won't you install everything in some directory on your local machine ( mkdir newgentoo; cd newgentoo,; rsync -a /usr/portage usr/portage; mount --bind /proc proc; chroot ./; env-update && . /etc/profile && PS1=(newgentoo)$PS1 )
Once you're done with messing around, simply take image of this directory with you and rsync -a newgento <whatever the target actualy is>
(and then run grub-install :) )
I've used this way successfully when I needed to install gentoo in some place with really slow net. Actualy I also use this pretty often plaing with virtual machines. :) Excpet that I mount --bind portage too instead of cloning it in this latter case

If you fail with some hardware (unfortunately gentoo-sources still have atheros drivers disabled by default :/ ) kernel is most likely going to be the only thing that needs recompilation.
Xorg will by default pull allmost all video drivers possible, so this should just work.
Back to top
View user's profile Send private message
creaker
l33t
l33t


Joined: 14 Jul 2012
Posts: 651

PostPosted: Thu Dec 05, 2013 7:34 pm    Post subject: Reply with quote

@szatox:
I'm not a target machine user, so I do not know which software end user may wish to have installed. Thus, I can't make preinstallation, as well as I can't just pack my system into stage4 archive.

@i92guboj:
Hardware shouldn't be a problem causer, I know CPU and MoBo model, video card model - they are very similar to my desktop, moreover I assembled this box myself few months ago, so I do not expect a hardware issues. Keyboard and mouse are both simple usb devices that can be handled by usb hid.

OK, I'll try to fetch packages.
Thanks again.
Back to top
View user's profile Send private message
dol-sen
Developer
Developer


Joined: 30 Jun 2002
Posts: 2689
Location: Richmond, BC, Canada

PostPosted: Fri Dec 06, 2013 4:11 am    Post subject: Reply with quote

I'm not sure if it is installed in regular portage releases, but it is with portage-9999
Code:
/usr/bin/emirrordist


It is one of the tools designed specifically for the mirrors to fetch and verify all the distfiles for the tree.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9013
Location: beyond the rim

PostPosted: Fri Dec 06, 2013 7:25 am    Post subject: Reply with quote

Are you sure you want to install Gentoo for someone else on a computer without network access? Sounds like a recipe for annoying support calls.
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 673

PostPosted: Fri Dec 06, 2013 8:53 am    Post subject: Reply with quote

Don't buy ring until you have girlfriend.
If the user doesn't know what he wants, he will be just as happy with ubuntu - or perhaps even more happy. If he wants to stick to gentoo, at least make him produce list of what he wants (and then make him pay extra for changes :D )
There is no point in even starting if you don't know what you're doing. Too much effort, too little result.
Back to top
View user's profile Send private message
jamapii
Guru
Guru


Joined: 16 Sep 2004
Posts: 528

PostPosted: Thu Jun 05, 2014 6:06 pm    Post subject: Reply with quote

I'm trying to do this.

I simply do

emerge -fuD --newuse some_big_packages

The problem is, it doesn't work. There are lots of circular dependencies, bogus conflicts (The following packages depend on X: none) etc. Maybe it can be resolved with autounmask-write, but I don't want to mess with the configuration just for fetching files. Or by not just fetching, but installing, which is not what fetchonly means.

So, what is needed to make "emerge -f" work?
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 8410
Location: Somewhere over Atlanta, Georgia

PostPosted: Thu Jun 05, 2014 6:18 pm    Post subject: Reply with quote

You should know that installing a new package doesn't work very well unless your system is up to date. If that condition is met, then --fetchonly works very well. If your system is up to date and you're still having issues, then that might very well be a bug in the Portage tree. Please post actual message and we'll help.

If you're trying to just fetch everything (but not build anything), see the method I posted earlier.

- John
_________________
I can confirm that I have received between 0 and 999 National Security Letters.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum