Forums

Skip to content

Advanced search
  • Quick links
    • Unanswered topics
    • Active topics
    • Search
  • FAQ
  • Login
  • Register
  • Board index Assistance Unsupported Software
  • Search

Download files and compile/emerge simultaneously

This forum covers all Gentoo-related software not officially supported by Gentoo. Ebuilds/software posted here might harm the health and stability of your system(s), and are not supported by Gentoo developers. Bugs/errors caused by ebuilds from overlays.gentoo.org are covered by this forum, too.
Post Reply
Advanced search
23 posts • Page 1 of 1
Author
Message
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

Download files and compile/emerge simultaneously

  • Quote

Post by dreamer3 » Tue Feb 25, 2003 5:32 am

Ok, this is hot off the presses. I've heard some of the mods say that downloading files while compiling can really glitch things up but I think they are referring to the "emerge -f kde & ", wait a little while then "emerge kde" and crossing your fingering hoping the downloads stay ahead of the compiles (otherwise you'll have two emerges trying to download the same file, messy).

I for one have had 4 or 5 different emerge processes running all download/compiling different things in the past in seperate windows with NO problems whatsoever.

This BASH script simply makes sure that the compiles don't get ahead of the file downloads.

Known Problems (as of v0.16*)
Each package that you pass is emerged invididually and added to the world file, dependencies are NOT handled as emerge itself handles them (ie, not adding them to world file). This is a side-effect of the looping that I just discovered (ie, it's a bug in qemerge) but haven't though of an easy fix for yet. Meanwhile if you're only emerging dependencies you can pass the --oneshot parameter which qemerge now does understand... Sorry for this, any thoughts on an easy way to fix this would be welcome. Update 3-06: Solution is in my head... just need to turn it into code...

Updates
v0.02 - now will die if the compiling emerge process ends with an error
v0.10 - added status bar to top of screen detailing emerge and download progress
v0.15
  • added name and version to status bar
    can now be used as a wrapper script for emerge (ie, figures out if you're wanting emerge for some of it's options and calls it, otherwises does what it does best, emerge and download)

v0.151 - fix bug in sed editing of emerge output

v0.152 - add "info" to options that need to go directly to emerge
v0.16 - understands --oneshot option now
v0.161 - understand all options in man emerge and when to handle compiles/downloads and when to let emerge do the work (ie, when you've passed options that don't really have to do with compiling or downloading and using semerge as a full-fledged wrapper for emerge)

Everyone, let me know if you like it, want more, etc...

semerge v0.161

Code: Select all

#!/bin/bash

source /sbin/functions.sh

# the folder to store working files
TEMP=~/.semerge
EMERGE=/usr/bin/emerge

set_scroll()
{	
	#echo "from: $1 to: $2"
	echo -en "\e[${1};${2}r\e[?4h";
}
save_cursor() { echo -en "\e7"; }
restore_cursor() { echo -en "\e8"; }
cursor_to() { echo -en "\e[$2;$1H"; }
erase_line() { echo -en "\e[2K"; }

download_files()
{
	i=1;
	total=`count_ebuilds $@`;

	for ebuild in $@; do
		update_dl_status "($i of $total) Fetching files for ${ebuild}..."
		$EMERGE -f "=${ebuild}" 2>/dev/null >/dev/null
		touch $TEMP/${ebuild/\//_}.complete
		i=$(($i+1))
	done
	update_dl_status "Downloads complete..."
}

is_downloaded()
{
	#convert the slash into an underscore
	local ebuild=${1/\//_};
	if [ -e $TEMP/${ebuild}.complete ]; then
		return 0;
	else
		return 1;
	fi
}

count_ebuilds()
{
	i=0;
	for x in $@; do
		i=$(($i+1));
	done
echo "$i"
}

setup_status_bar()
{
	echo; echo;
	save_cursor
	set_scroll 4 80
	cursor_to 1 1;	erase_line
	cursor_to 1 2;	erase_line
	cursor_to 1 3;	erase_line
	echo -en "--------------------------------\e[7m[ semerge v0.16 ]\e[0m--------------------------------"
	restore_cursor
}

update_dl_status()
{
	save_cursor
	cursor_to 1 1
	erase_line
	einfo $@
	restore_cursor
}

update_emerge_status()
{
	save_cursor
	cursor_to 1 2
	erase_line
	einfo $@
	restore_cursor
}

start_compiles()
{
	i=1;
	total=`count_ebuilds $@`;
	
	for ebuild in $@; do
		# are the files for the ebuild downloaded
		is_downloaded ${ebuild}
		while [ $? != 0 ]; do
			sleep 2s;
			is_downloaded ${ebuild}
		done
		update_emerge_status "($i of $total) Emerging ${ebuild}..."
		einfo "$EMERGE $EMERGE_OPTS \"=${ebuild}\""
		einfo "($i of $total) Emerging ${ebuild}..."
		$EMERGE $EMERGE_OPTS "=${ebuild}"
		if [ $? != 0 ]; then
			eerror "emerge ${ebuild}"
			eerror " - has exited with an error"
			echo
			eerror "qemerge is exiting as well"
			exit 1
		fi
		i=$(($i+1));
	done 
}

get_ebuilds()
{
all=`$EMERGE $@ -p | grep "\[ebuild " | sed -e "s/\[ebuild[^]]*] //" | awk '{print $1}'`
echo ${all}
return ${result}
}

clean_up()
{
	for ebuild in $@; do
		rm -f $TEMP/${ebuild/\//_}.complete
	done
	save_cursor; set_scroll 1 1000; restore_cursor
}

EMERGE_OPTS=""

is_emerge_options()
{
	result=0
	while [ ${#} -gt 0 ]; do
		a=${1}
		shift
		case "${a}" in
			--oneshot|--nodeps|--buildpkg|-b|--usepkg|-k|-verbose|-v)
				# these options should be passed each time emerge is executed
				EMERGE_OPTS="$EMERGE_OPTS ${a}"
				;;
			--deep|--emptytree|-e|--onlydeps|-o)
				# these simply possibly add or change which packages
				# emerge returns, these are allowed
				;;
			clean|depclean|help|info|inject|prune|rsync|search|sync|unmerge)
				result=1
				;;
			-*)
				result=1
				;;
		esac
	done
	return ${result}
}

is_emerge_options $@
if [ $? != 0 ]; then
	$EMERGE $@
	exit
fi

# make sure working folder it exists
mkdir -p $TEMP

# get the ebuilds we'll be building
ebuilds=`get_ebuilds $@`
if [ -z "${ebuilds}" ]; then
	$EMERGE $@
	exit
fi

# startup stuff
clean_up ${ebuilds}
setup_status_bar

# start the downloads in the background (one at a time)
download_files ${ebuilds} &

# start compiling
start_compiles ${ebuilds}

# clean up our state tracking files
clean_up ${ebuilds}
Last edited by dreamer3 on Fri Mar 07, 2003 4:21 am, edited 8 times in total.
Top
AlterEgo
Veteran
Veteran
User avatar
Posts: 1619
Joined: Thu Apr 25, 2002 2:51 pm

  • Quote

Post by AlterEgo » Tue Feb 25, 2003 3:38 pm

Great, I'm trying it now.
Really stupid question: how do you keep track of both processes (comping and downloading). I use prozilla, so the compiling process happens where I cannot see it.
The only indication I have that both things are running, comes from gkrellm (great tool BTW).
Top
vaevictus
n00b
n00b
User avatar
Posts: 20
Joined: Mon Feb 24, 2003 9:11 pm

glad I could be of inspiration.

  • Quote

Post by vaevictus » Tue Feb 25, 2003 10:13 pm

think there's any chance of someone pythonizing this type of setup into the portage standard way of doing things?

n8
Use FDISK for WIN32 today!
Top
Kijutsu
n00b
n00b
User avatar
Posts: 16
Joined: Mon Feb 17, 2003 2:51 am
Location: Romulus, MI
Contact:
Contact Kijutsu
Website

n33t

  • Quote

Post by Kijutsu » Tue Feb 25, 2003 10:33 pm

When I start compiling minor stuff I'll give it a runthru... great idea dude! 8)
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

Re: glad I could be of inspiration.

  • Quote

Post by dreamer3 » Wed Feb 26, 2003 11:37 pm

vaevictus wrote:think there's any chance of someone pythonizing this type of setup into the portage standard way of doing things?

n8
It's been on the list of "wanted" Portage features for some time I think, but until then I think it can be done just as well with this type of approach...

Now, if I could just come up with a cool way to allow the user to see BOTH output streams (downloading and compiling) I seriously think that with better error handling this approach is perfect.

I will think about it... seems like one could nohup the wget and then read the file, pass it through grep and sed and print download "status" lines every once in a while during the buildling... would anyone even like this kind of download status though?
Top
vaevictus
n00b
n00b
User avatar
Posts: 20
Joined: Mon Feb 24, 2003 9:11 pm

yeah

  • Quote

Post by vaevictus » Thu Feb 27, 2003 3:37 pm

I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :)
Use FDISK for WIN32 today!
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

Re: yeah

  • Quote

Post by dreamer3 » Fri Feb 28, 2003 1:27 am

vaevictus wrote:I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :)
Could probably do all that with BASH ;-) In one of my CS classes in college we had to write a file viewer (kinda like nano but without edit functionality) with a toolbar and I learned how to do all that kinda of stuff with VT100 escape sequences... not sure how ncurses does it, but I bet what we learned would work with most consoles...

I will look into those sorts of glitzy things if I see people start using the script...
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

Re: yeah

  • Quote

Post by dreamer3 » Sat Mar 01, 2003 1:31 pm

vaevictus wrote:I'd love it if portage would have a curses tagline on the bottom with: "Emerge package X out of Y total" and a download line would be good too.
Ask you shall recieve... semerge now has both (at the top cause I like it that way). (uses VT100 terminal escape codes which work fine with eterm at least, should work with most term emus).

Give it a try... i use semerge hdparm hddtemp unzip to test it...

Edit: Ok, it works fine in Eterm, but not from a console... evidentally Eterm doesn't mind my inflating the # of lines in the VT100 code that sets the scroll area, but the console does... it only works when you specify the correct # of lines that are actually VISIBLE on the screen. Anyone know how to get this easily with a command line utility? Of course if you ALWAYS used only the console you can just hack the utility...
Top
wyrickre
n00b
n00b
User avatar
Posts: 27
Joined: Wed Jul 10, 2002 12:13 am
Location: Colorado Springs, CO

  • Quote

Post by wyrickre » Wed Mar 05, 2003 2:07 am

try

Code: Select all

$ stty size
or

Code: Select all

$ resize
Geek used to be a 4 letter word; Now it's a 6 figure one.
Top
pjp
Administrator
Administrator
User avatar
Posts: 20668
Joined: Tue Apr 16, 2002 10:35 pm

  • Quote

Post by pjp » Wed Mar 05, 2003 2:20 am

Don't know why I didn't mention this sooner. Also of interest might be the portage version (under development) of this feature. More info in GF4: Can portage build one package while downloading others?.
Quis separabit? Quo animo?
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

  • Quote

Post by dreamer3 » Fri Mar 07, 2003 3:48 am

wyrickre wrote:try

Code: Select all

$ stty size
or

Code: Select all

$ resize
Looking into using this. Also, just updated the version to 0.16 to allow it to work with --oneshot.
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

Re: yeah

  • Quote

Post by dreamer3 » Fri Mar 07, 2003 4:22 am

vaevictus wrote:I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
Still waiting to see what you think of recent updates. :-)
Top
Sastraxi
Apprentice
Apprentice
User avatar
Posts: 258
Joined: Tue Feb 25, 2003 9:23 pm

  • Quote

Post by Sastraxi » Fri Mar 07, 2003 4:45 am

Is there any way to pause part of the console? Aka. have the top 4 lines have the wget status (ex. 1 of 8, pbar, etc.), and the rest the compile output.
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

  • Quote

Post by dreamer3 » Fri Mar 07, 2003 8:57 am

Sastraxi wrote:Is there any way to pause part of the console? Aka. have the top 4 lines have the wget status (ex. 1 of 8, pbar, etc.), and the rest the compile output.
Pause???

I think what you're asking for is something like "split screen"... and no, I don't think that is possible... what I'm doing is redefining the "scroll" window of the screen and then moving the cursor to the top, updating status, and moving it back quickly... just thinking about it it seems that eventually it should "screw up" when in the middle of a big emerge or something and put some emerge output on the top, but I haven't seen that happen yet.

Anyone have any ideas for "split screen" mode? I was just wondering about logging both the download and the emerge (running in the BG) and having the BASH script read the output files very often and draw the screen itself... sounds like a lot of work though if it would work at all ;-)

Would be cool to see an implimentation though.

Edit: Is it possible to "pause" a process in BASH from script?
Top
TheCoop
Veteran
Veteran
User avatar
Posts: 1814
Joined: Sat Jun 15, 2002 5:20 pm
Location: Where you least expect it
Contact:
Contact TheCoop
Website

  • Quote

Post by TheCoop » Fri Mar 07, 2003 9:29 am

can you do a split screen thing with framebuffers? (like the gentoo logo on bootup)
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

  • Quote

Post by dreamer3 » Fri Mar 07, 2003 9:50 am

TheCoop wrote:can you do a split screen thing with framebuffers? (like the gentoo logo on bootup)
Maybe, but that would be outside the scope of what I want to accomplish as I don't want a certain thing to have to be compiled into your kernel for semerge to work properly...
Top
Ari Rahikkala
Guru
Guru
Posts: 370
Joined: Wed Oct 02, 2002 7:29 pm
Location: Finland

  • Quote

Post by Ari Rahikkala » Wed Aug 20, 2003 10:30 am

dreamer3 wrote:Edit: Is it possible to "pause" a process in BASH from script?
Well, it's possible to `kill -STOP` one, but I think you mean the output, not the whole process...
<laurentius> gentoo linux?
<ari> Yesh.
<laurentius> they look horny
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

  • Quote

Post by dreamer3 » Thu Aug 21, 2003 8:37 pm

It's been so long I have no idea what I was asking...
Top
Hackeron
Guru
Guru
Posts: 307
Joined: Fri Nov 01, 2002 2:44 pm

  • Quote

Post by Hackeron » Sat Feb 28, 2004 2:53 am

why is script not being updated?
Top
dreamer3
Guru
Guru
Posts: 553
Joined: Tue Sep 24, 2002 6:15 am

  • Quote

Post by dreamer3 » Sat Feb 28, 2004 6:32 am

Becuase I have a dial-up connection and get tired of downloading/compiling source all the time so I switched to Debian. :-) *ducks*

That being said, my intended answer to the problem:

The get_ebuilds step needs to be broken into two... the first command run is exected with --no-deps and this list is stored as packages that should be emerged and ADDED to the world file.

Then the same emerge needs to be called with --deps-only to return a list of which ebuilds should be emerged with --one-time-only (ie, not added to the world file cause they are only depends).

Now that one has BOTH these lists it might be necessary to call emerge again with neither parameter so that a COMPLETE list is created with the correct installation order. Incase one of the dependencies requires one of the non-deps of a different package, etc...

Only problem with this is it might significantly increase the startup time of the script as emerge needs to be run 3 times... If emerge outputed more detailed info (say if package was gonna be added to world or not) you could just parse that, but it doesn't.

If I still ran Gentoo I'd be happy to write this up but I've honestly seen little interest in this script except for the occasoinal passerby, so I'll leave it as an excersize for the reader... Sorry. :-)

Feel free to hack something together and paste here and I"ll come look at it... I love BASH scripting, but I really don't have the time or inclination to do more at this point.
Top
Hackeron
Guru
Guru
Posts: 307
Joined: Fri Nov 01, 2002 2:44 pm

  • Quote

Post by Hackeron » Sat Feb 28, 2004 11:58 am

Thanks for giving this info, I'm going to try to make a hack for what you just said. PS, debian is a nice distro and has many advantages to gentoo, no need to duck, there are many simultaneous debian/gentoo users here ;)

EDIT: Think I'm going to turn this into a full size wrapper for portage actually, I have a way in my head for something like the 'ehush' script, but with better output, and shows information about how much lines compiled, with ability to make a bug report and display both compile and download information.
Top
meowsqueak
Veteran
Veteran
User avatar
Posts: 1549
Joined: Tue Aug 26, 2003 6:46 am
Location: New Zealand

  • Quote

Post by meowsqueak » Sat Jul 03, 2004 11:15 pm

What's the current status of this enhancement please? Is anyone still actively working on it?
Top
kamagurka
Veteran
Veteran
User avatar
Posts: 1026
Joined: Sun Jan 25, 2004 1:55 am
Location: /germany/munich
Contact:
Contact kamagurka
Website

Re: yeah

  • Quote

Post by kamagurka » Thu Jul 08, 2004 1:55 pm

vaevictus wrote:I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :)
yes please. want please.
If you loved me, you'd all kill yourselves today.
--Spider Jerusalem, the Word
Top
Post Reply

23 posts • Page 1 of 1

Return to “Unsupported Software”

Jump to
  • Assistance
  • ↳   News & Announcements
  • ↳   Frequently Asked Questions
  • ↳   Installing Gentoo
  • ↳   Multimedia
  • ↳   Desktop Environments
  • ↳   Networking & Security
  • ↳   Kernel & Hardware
  • ↳   Portage & Programming
  • ↳   Gamers & Players
  • ↳   Other Things Gentoo
  • ↳   Unsupported Software
  • Discussion & Documentation
  • ↳   Documentation, Tips & Tricks
  • ↳   Gentoo Chat
  • ↳   Gentoo Forums Feedback
  • ↳   Duplicate Threads
  • International Gentoo Users
  • ↳   中文 (Chinese)
  • ↳   Dutch
  • ↳   Finnish
  • ↳   French
  • ↳   Deutsches Forum (German)
  • ↳   Diskussionsforum
  • ↳   Deutsche Dokumentation
  • ↳   Greek
  • ↳   Forum italiano (Italian)
  • ↳   Forum di discussione italiano
  • ↳   Risorse italiane (documentazione e tools)
  • ↳   Polskie forum (Polish)
  • ↳   Instalacja i sprzęt
  • ↳   Polish OTW
  • ↳   Portuguese
  • ↳   Documentação, Ferramentas e Dicas
  • ↳   Russian
  • ↳   Scandinavian
  • ↳   Spanish
  • ↳   Other Languages
  • Architectures & Platforms
  • ↳   Gentoo on ARM
  • ↳   Gentoo on PPC
  • ↳   Gentoo on Sparc
  • ↳   Gentoo on Alternative Architectures
  • ↳   Gentoo on AMD64
  • ↳   Gentoo for Mac OS X (Portage for Mac OS X)
  • Board index
  • All times are UTC
  • Delete cookies

© 2001–2026 Gentoo Foundation, Inc.

Powered by phpBB® Forum Software © phpBB Limited

Privacy Policy