Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Running multiple instances of emerge
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
Mr. Atoz
Tux's lil' helper
Tux's lil' helper


Joined: 06 May 2002
Posts: 84
Location: Colorado

PostPosted: Wed May 08, 2002 5:28 pm    Post subject: Running multiple instances of emerge Reply with quote

How does emerge handle emerging multiple packages at the same time (separate shell windows) when one or more of the packages SHARE dependancies? In my limited trials, I had mixed success. Sometime it would seem to work, and sometimes the packages sharing the dependancies would both fail.

IMO, it would be handy to see a locking mechanism in emerge. Each emerge instance that was started would update and read an "in progress" section in the pkg db. The "in progress" section would contain a list of packages and all dependancy packages CURRENTLY in the queue. So...

Package A has dependancies X, Y, Z and package B has dependancies W, X, Y. Say I start an emerge on package B. Emerge would load W, X, Y, B into the "in progress" section, to "lock" them. Then I start an emerge on package A. This emerge would check and understand that two of its dependancies, X, Y, are locked by another instance of emerge.

Emerge would then either exit stating that it cannot continue with Package A, or it would wait until W, X, Y, B are finished and then resume emerging.
_________________
---Atoz
Back to top
View user's profile Send private message
arkane
l33t
l33t


Joined: 30 Apr 2002
Posts: 918
Location: Phoenix, AZ

PostPosted: Wed May 08, 2002 5:31 pm    Post subject: Reply with quote

Emerge was created as a light tool. The lightness makes it quick, agile, and able to be added to as needed. Putting those types of things into the code would start a trend to make it start thinking for the user. Why would you want to do that, anyway? Do you have multiple processors? If so, use the -j<number> flag in the make options to use them all on one emerge and just slap a few packages after the one emerge. Compiling two things at once is just silly. Maybe we should put in code to keep two instances of emerge from installing anything at the same time, just allow queries. :)

Just wondering, if your that worried about dependencies during concurrent emerges, why not write a wrapper script to check the dependencies of each and just compile the dependencies first, then you can run any concurrent compiles until your hearts content.
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Wed May 08, 2002 5:42 pm    Post subject: Re: Running multiple instances of emerge Reply with quote

Mr. Atoz wrote:
How does emerge handle emerging multiple packages at the same time (separate shell windows) when one or more of the packages SHARE dependancies?


You don't want to do this. Ever.

Mr. Atoz wrote:
IMO, it would be handy to see a locking mechanism in emerge


Yes, but the locking mechanism should be binary -- either everything is locked or nothing is locked. Trying to lock certain programs only if they have shared dependencies is a Bad Idea, IMO.

You know, don't you, that you can install multiple packages like this:

Code:
emerge xfsprogs bitchx vim links


Granted, it does things in sequence, rather than in parallel, but that's the safer way to do things. (and I believe Gentoo developers are working on a parallelized version of emerge as well)

BTW, this thread probably belongs in Gentoo Suggestions. Moving it there.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
Mr. Atoz
Tux's lil' helper
Tux's lil' helper


Joined: 06 May 2002
Posts: 84
Location: Colorado

PostPosted: Thu May 09, 2002 9:08 am    Post subject: Reply with quote

arkane wrote:
Why would you want to do that, anyway? Do you have multiple processors? If so, use the -j<number> flag in the make options to use them all on one emerge and just slap a few packages after the one emerge. Compiling two things at once is just silly. Maybe we should put in code to keep two instances of emerge from installing anything at the same time, just allow queries. :)

Just wondering, if your that worried about dependencies during concurrent emerges, why not write a wrapper script to check the dependencies of each and just compile the dependencies first, then you can run any concurrent compiles until your hearts content.


Nope, I only have one processor. I don't think compiling multiple things at once is sillly at all. Even if it was, I wouldn't mind because I'm a silly person. :D :lol: :D :lol: I understand that not only am I going to NOT gain anything in the way of speed, but that the whole system will bog down to a crawl. I only do this when I am going to walk away for a few hours or before I go to bed.

Writing a wrapper script is not a bad idea If I can scrape the time together (Yeah, right. There I go dreaming again) to do it. When/if I do, I will share it with ya'all. I've been looking for something to make a good learning project for Python...

Quote:

Maybe we should put in code to keep two instances of emerge from installing anything at the same time, just allow queries. :)

That is actually not a bad idea either and probably wouldn't be too difficult to implement. I think RPM (Which, BTW PALES] in comparison to Portage, IMHO) already does this.

klieber wrote:

Yes, but the locking mechanism should be binary -- either everything is locked or nothing is locked. Trying to lock certain programs only if they have shared dependencies is a Bad Idea, IMO.

Yes, I agree, binary locking is the way to go for performance and accuracy.
klieber wrote:

You know, don't you, that you can install multiple packages like this:
Code:

emerge xfsprogs bitchx vim links


Actually, that had slipped my mind. Thanks for unslipping it. :oops:

However, I have never gotten this to work. It just does world and then exits.
Code:

emerge --update world system


Are you not allowed to pass in multiple packages with the --update flag?
_________________
---Atoz
Back to top
View user's profile Send private message
Mr. Atoz
Tux's lil' helper
Tux's lil' helper


Joined: 06 May 2002
Posts: 84
Location: Colorado

PostPosted: Thu May 09, 2002 9:29 am    Post subject: Re: Running multiple instances of emerge Reply with quote

klieber wrote:

Yes, but the locking mechanism should be binary -- either everything is locked or nothing is locked. Trying to lock certain programs only if they have shared dependencies is a Bad Idea, IMO.


After giving this more thought, I don't think it would be much of a big deal to lock individual packages, especially if it was a binary locking system. It would simply be an extension of the existing dependancy analysis. A 10,000 mile view:

Code:

1 - Analyze dependancies
2 - Check lock flag for package and each dependancy.  If ANY dependancy is locked, place this emerge in the queue and recheck the lock flags every N seconds/minutes.
3 - Once locks are cleared, re-analyze dependancies, just in case whatever held the locks failed.
4 - Resume the emerge.

Step 2 might want to emerge dependancies up to, but NOT including the locked package. Also, since the lock would simply be a binary flag, this should work for nested dependancies as well. Emerge could print some text stating, "emerge is pausing until {insert ebuild(s)) who hold the locks here} have completed. Please stand by. Do not adjust your television set"

Anyway, just a thought.... :)
_________________
---Atoz
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Thu May 09, 2002 11:35 am    Post subject: Reply with quote

Mr. Atoz wrote:
Code:

emerge --update world system

Are you not allowed to pass in multiple packages with the --update flag?


World is just that. Everything. It's inclusive of system, so there's no need to pass both world and system as arguments.

And yes, you can pass multiple packages to emerge, even when using the update flag.

Mr. Atoz wrote:
After giving this more thought, I don't think it would be much of a big deal to lock individual packages, especially if it was a binary locking system. It would simply be an extension of the existing dependancy analysis.


Not sure I agree, but for the sake of argument, let's say it isn't a big deal. What does it gain? Given that you can already install multiple programs with one emerge command, and given that compiling multiple programs simultaneously does not decrease the overall compile time, then what additional benefit is provided from this type of a locking system?

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
Mr. Atoz
Tux's lil' helper
Tux's lil' helper


Joined: 06 May 2002
Posts: 84
Location: Colorado

PostPosted: Thu May 09, 2002 9:44 pm    Post subject: Reply with quote

Point well taken. And, after giving your question some thought as to what it would gain, I cannot come up with a good answer.

Locking the whole DB during an emerge might still be a good idea, as long as you could still query it. This would provide a simple safety mechanism preventing one from running multiple emerges in the first place.
_________________
---Atoz
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Thu May 09, 2002 10:28 pm    Post subject: Reply with quote

Mr. Atoz wrote:
Locking the whole DB during an emerge might still be a good idea, as long as you could still query it.


I agree -- you probably want to file this as a feature request on bugs.gentoo.org.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
itzdandy
Guest





PostPosted: Thu May 09, 2002 10:52 pm    Post subject: thanks for the random accidental advice Reply with quote

emerge foo foo2 fooa fooc foo9
whod a thought,
ive been making a script for this:

nano -w /home/"me"/emscript.sh
{
emerge foo
emerge foo2
emerge fooa
emerge fooc
emerge foo9
}
sh /home/"me"/emscript.sh

ya learn something new everyday
Back to top
kang
Guest





PostPosted: Thu May 16, 2002 3:10 pm    Post subject: something you didn't though of.... Reply with quote

Well appart from the "hey i forgot to put that install in queue before going to bed"
you can have multiple admins of the same system logged as root who want to upgrade some stuff at the same time (happens to me OFTEN since i work on computers which are usually >900km away).
You won't ps aux | grep emerge all the time before doing it, and even then, someone could emerge while you are doing to because, eh, we are juts humans and can't ps aux that fast :D
so yes a lock would be usefull, ala apt, which touch a lock file (so if ever u had to unlock just delete the file and you can emerge again, as ypou could happen to have to do 2 emerge at the same time (if u know what u are doing, that's not mandrake, right ? we need power of doing anything we want!) or if the first emerge died and you cannot remove it from memory before the kernel does it for you ([defect]) as it can sometimes happen (even if rare)


just some thoughs :)
Back to top
joshua
Tux's lil' helper
Tux's lil' helper


Joined: 19 Jun 2002
Posts: 134
Location: Wiesbaden

PostPosted: Wed Sep 25, 2002 1:49 pm    Post subject: Reply with quote

for me it makes sense to run multiple instances of emerge:

actually right now i am emerging mozilla. while it builds i badly need to install eg. the very important xmms-themes (or any small app i really need at that moment, maybe ddclient or anything).

so please continue allowing us to emerge multiple things at the same time.

moreover i did that a lot, even if it was not necessary, and never experienced any problems
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Wed Sep 25, 2002 7:03 pm    Post subject: Reply with quote

joshua wrote:
so please continue allowing us to emerge multiple things at the same time.

moreover i did that a lot, even if it was not necessary, and never experienced any problems

I don't think it's a question of "continue", AFAIK it was never supported, and carpaski is on record here saying it's a bad idea.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
klieber
Administrator
Administrator


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Wed Sep 25, 2002 7:24 pm    Post subject: Reply with quote

One trick that I've used with good results is to have one instance of emerge compiling a particular program and then, in a separate console, have a second instance of emerge fetching packages for the other programs that I want to install. (emerge -f)

That also ensures that emerge doesn't bomb out in the middle of a compile because it can't find a dependency package on any of the mirrors.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
Nedo
n00b
n00b


Joined: 30 Apr 2002
Posts: 7

PostPosted: Wed Sep 25, 2002 10:46 pm    Post subject: Reply with quote

Wouldn't it be an idea to have emerge lists all the packages it is going to emerge in a queue file and if there already is an emerge running on the computer the second started one should search throw the queue file and see what packages thats not in the queue and are needed for the current emerge that would stop two admins from emergine packages at the same time.

And if you have some special package maybe there could be a system for setting priority on the packages that needs to be emerged.

Another thing that I would like ot see in the emerge program is an possibuility to see what package it's currently beeining emerged I end up writing.
Code:
emerge -p gnome

all the time to see how much is left. An way of passing -s to make would be nice to because I really don't need to see all the output of make.
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Wed Sep 25, 2002 10:54 pm    Post subject: Reply with quote

Feel free to submit any or all of these as enhancement requests at https://bugs.gentoo.org.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
gsfgf
Veteran
Veteran


Joined: 08 May 2002
Posts: 1266

PostPosted: Thu Sep 26, 2002 1:13 am    Post subject: Reply with quote

When i set up a gentoo sys i run about 10 emerges at once. all the gtk apps fail since they all try to emerge gtk at once. I then emerge gtk and reemerge the rest. all is good. An openoffice can go along in the background for a month or whatever it takes. :)
Back to top
View user's profile Send private message
Curious
Bodhisattva
Bodhisattva


Joined: 13 May 2002
Posts: 395
Location: Sydney, Australia

PostPosted: Thu Sep 26, 2002 1:34 am    Post subject: Reply with quote

gsfgf wrote:
When i set up a gentoo sys i run about 10 emerges at once. all the gtk apps fail since they all try to emerge gtk at once. I then emerge gtk and reemerge the rest. all is good. An openoffice can go along in the background for a month or whatever it takes. :)


Careful! Doing it this way places GTK itself in your 'world' preferred packages list, instead of making it a lowly dependancy like it should be ;-) If thats what you want, fine - but you may not want to update GTK evertime a new version is released, instead updating it only when a package you chose to deliberately install needs the updated version.

Eh, YMMV.

-- Curious.
_________________
Are you down with the Hawk?
Back to top
View user's profile Send private message
BLASTER_
Tux's lil' helper
Tux's lil' helper


Joined: 17 May 2002
Posts: 105
Location: .:Sea - WA - USA

PostPosted: Sun Sep 29, 2002 1:14 pm    Post subject: Reply with quote

they locked a thred about fetching packages during compile time, I think thats a GREAT idea.. corse it adds bloat to emerge.

so, something like.. on a bfresh syatem..

emerge kde & emerge kde -f

hurrm..
Back to top
View user's profile Send private message
gsfgf
Veteran
Veteran


Joined: 08 May 2002
Posts: 1266

PostPosted: Sun Sep 29, 2002 4:10 pm    Post subject: Reply with quote

Curious wrote:
gsfgf wrote:
When i set up a gentoo sys i run about 10 emerges at once. all the gtk apps fail since they all try to emerge gtk at once. I then emerge gtk and reemerge the rest. all is good. An openoffice can go along in the background for a month or whatever it takes. :)


Careful! Doing it this way places GTK itself in your 'world' preferred packages list, instead of making it a lowly dependancy like it should be ;-) If thats what you want, fine - but you may not want to update GTK evertime a new version is released, instead updating it only when a package you chose to deliberately install needs the updated version.

Eh, YMMV.

-- Curious.


I mean, i emerge 1 gtk pkg and then reemerge all the rest after the one has finished.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum