Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Download files and compile/emerge simultaneously
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Tue Feb 25, 2003 5:32 am    Post subject: Download files and compile/emerge simultaneously Reply with quote

Ok, this is hot off the presses. I've heard some of the mods say that downloading files while compiling can really glitch things up but I think they are referring to the "emerge -f kde & ", wait a little while then "emerge kde" and crossing your fingering hoping the downloads stay ahead of the compiles (otherwise you'll have two emerges trying to download the same file, messy).

I for one have had 4 or 5 different emerge processes running all download/compiling different things in the past in seperate windows with NO problems whatsoever.

This BASH script simply makes sure that the compiles don't get ahead of the file downloads.

Known Problems (as of v0.16*)
Each package that you pass is emerged invididually and added to the world file, dependencies are NOT handled as emerge itself handles them (ie, not adding them to world file). This is a side-effect of the looping that I just discovered (ie, it's a bug in qemerge) but haven't though of an easy fix for yet. Meanwhile if you're only emerging dependencies you can pass the --oneshot parameter which qemerge now does understand... Sorry for this, any thoughts on an easy way to fix this would be welcome. Update 3-06: Solution is in my head... just need to turn it into code...

Updates
v0.02 - now will die if the compiling emerge process ends with an error
v0.10 - added status bar to top of screen detailing emerge and download progress
v0.15
    added name and version to status bar
    can now be used as a wrapper script for emerge (ie, figures out if you're wanting emerge for some of it's options and calls it, otherwises does what it does best, emerge and download)

v0.151 - fix bug in sed editing of emerge output

v0.152 - add "info" to options that need to go directly to emerge
v0.16 - understands --oneshot option now
v0.161 - understand all options in man emerge and when to handle compiles/downloads and when to let emerge do the work (ie, when you've passed options that don't really have to do with compiling or downloading and using semerge as a full-fledged wrapper for emerge)

Everyone, let me know if you like it, want more, etc...

semerge v0.161
Code:
#!/bin/bash

source /sbin/functions.sh

# the folder to store working files
TEMP=~/.semerge
EMERGE=/usr/bin/emerge

set_scroll()
{   
   #echo "from: $1 to: $2"
   echo -en "\e[${1};${2}r\e[?4h";
}
save_cursor() { echo -en "\e7"; }
restore_cursor() { echo -en "\e8"; }
cursor_to() { echo -en "\e[$2;$1H"; }
erase_line() { echo -en "\e[2K"; }

download_files()
{
   i=1;
   total=`count_ebuilds $@`;

   for ebuild in $@; do
      update_dl_status "($i of $total) Fetching files for ${ebuild}..."
      $EMERGE -f "=${ebuild}" 2>/dev/null >/dev/null
      touch $TEMP/${ebuild/\//_}.complete
      i=$(($i+1))
   done
   update_dl_status "Downloads complete..."
}

is_downloaded()
{
   #convert the slash into an underscore
   local ebuild=${1/\//_};
   if [ -e $TEMP/${ebuild}.complete ]; then
      return 0;
   else
      return 1;
   fi
}

count_ebuilds()
{
   i=0;
   for x in $@; do
      i=$(($i+1));
   done
echo "$i"
}

setup_status_bar()
{
   echo; echo;
   save_cursor
   set_scroll 4 80
   cursor_to 1 1;   erase_line
   cursor_to 1 2;   erase_line
   cursor_to 1 3;   erase_line
   echo -en "--------------------------------\e[7m[ semerge v0.16 ]\e[0m--------------------------------"
   restore_cursor
}

update_dl_status()
{
   save_cursor
   cursor_to 1 1
   erase_line
   einfo $@
   restore_cursor
}

update_emerge_status()
{
   save_cursor
   cursor_to 1 2
   erase_line
   einfo $@
   restore_cursor
}

start_compiles()
{
   i=1;
   total=`count_ebuilds $@`;
   
   for ebuild in $@; do
      # are the files for the ebuild downloaded
      is_downloaded ${ebuild}
      while [ $? != 0 ]; do
         sleep 2s;
         is_downloaded ${ebuild}
      done
      update_emerge_status "($i of $total) Emerging ${ebuild}..."
      einfo "$EMERGE $EMERGE_OPTS \"=${ebuild}\""
      einfo "($i of $total) Emerging ${ebuild}..."
      $EMERGE $EMERGE_OPTS "=${ebuild}"
      if [ $? != 0 ]; then
         eerror "emerge ${ebuild}"
         eerror " - has exited with an error"
         echo
         eerror "qemerge is exiting as well"
         exit 1
      fi
      i=$(($i+1));
   done
}

get_ebuilds()
{
all=`$EMERGE $@ -p | grep "\[ebuild " | sed -e "s/\[ebuild[^]]*] //" | awk '{print $1}'`
echo ${all}
return ${result}
}

clean_up()
{
   for ebuild in $@; do
      rm -f $TEMP/${ebuild/\//_}.complete
   done
   save_cursor; set_scroll 1 1000; restore_cursor
}

EMERGE_OPTS=""

is_emerge_options()
{
   result=0
   while [ ${#} -gt 0 ]; do
      a=${1}
      shift
      case "${a}" in
         --oneshot|--nodeps|--buildpkg|-b|--usepkg|-k|-verbose|-v)
            # these options should be passed each time emerge is executed
            EMERGE_OPTS="$EMERGE_OPTS ${a}"
            ;;
         --deep|--emptytree|-e|--onlydeps|-o)
            # these simply possibly add or change which packages
            # emerge returns, these are allowed
            ;;
         clean|depclean|help|info|inject|prune|rsync|search|sync|unmerge)
            result=1
            ;;
         -*)
            result=1
            ;;
      esac
   done
   return ${result}
}

is_emerge_options $@
if [ $? != 0 ]; then
   $EMERGE $@
   exit
fi

# make sure working folder it exists
mkdir -p $TEMP

# get the ebuilds we'll be building
ebuilds=`get_ebuilds $@`
if [ -z "${ebuilds}" ]; then
   $EMERGE $@
   exit
fi

# startup stuff
clean_up ${ebuilds}
setup_status_bar

# start the downloads in the background (one at a time)
download_files ${ebuilds} &

# start compiling
start_compiles ${ebuilds}

# clean up our state tracking files
clean_up ${ebuilds}


Last edited by dreamer3 on Fri Mar 07, 2003 4:21 am; edited 8 times in total
Back to top
View user's profile Send private message
AlterEgo
Veteran
Veteran


Joined: 25 Apr 2002
Posts: 1619

PostPosted: Tue Feb 25, 2003 3:38 pm    Post subject: Reply with quote

Great, I'm trying it now.
Really stupid question: how do you keep track of both processes (comping and downloading). I use prozilla, so the compiling process happens where I cannot see it.
The only indication I have that both things are running, comes from gkrellm (great tool BTW).
Back to top
View user's profile Send private message
vaevictus
n00b
n00b


Joined: 24 Feb 2003
Posts: 20

PostPosted: Tue Feb 25, 2003 10:13 pm    Post subject: glad I could be of inspiration. Reply with quote

think there's any chance of someone pythonizing this type of setup into the portage standard way of doing things?

n8
_________________
Use FDISK for WIN32 today!
Back to top
View user's profile Send private message
Kijutsu
n00b
n00b


Joined: 17 Feb 2003
Posts: 16
Location: Romulus, MI

PostPosted: Tue Feb 25, 2003 10:33 pm    Post subject: n33t Reply with quote

When I start compiling minor stuff I'll give it a runthru... great idea dude! 8)
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Wed Feb 26, 2003 11:37 pm    Post subject: Re: glad I could be of inspiration. Reply with quote

vaevictus wrote:
think there's any chance of someone pythonizing this type of setup into the portage standard way of doing things?

n8

It's been on the list of "wanted" Portage features for some time I think, but until then I think it can be done just as well with this type of approach...

Now, if I could just come up with a cool way to allow the user to see BOTH output streams (downloading and compiling) I seriously think that with better error handling this approach is perfect.

I will think about it... seems like one could nohup the wget and then read the file, pass it through grep and sed and print download "status" lines every once in a while during the buildling... would anyone even like this kind of download status though?
Back to top
View user's profile Send private message
vaevictus
n00b
n00b


Joined: 24 Feb 2003
Posts: 20

PostPosted: Thu Feb 27, 2003 3:37 pm    Post subject: yeah Reply with quote

I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :)
_________________
Use FDISK for WIN32 today!
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Fri Feb 28, 2003 1:27 am    Post subject: Re: yeah Reply with quote

vaevictus wrote:
I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :)

Could probably do all that with BASH ;-) In one of my CS classes in college we had to write a file viewer (kinda like nano but without edit functionality) with a toolbar and I learned how to do all that kinda of stuff with VT100 escape sequences... not sure how ncurses does it, but I bet what we learned would work with most consoles...

I will look into those sorts of glitzy things if I see people start using the script...
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Sat Mar 01, 2003 1:31 pm    Post subject: Re: yeah Reply with quote

vaevictus wrote:
I'd love it if portage would have a curses tagline on the bottom with: "Emerge package X out of Y total" and a download line would be good too.

Ask you shall recieve... semerge now has both (at the top cause I like it that way). (uses VT100 terminal escape codes which work fine with eterm at least, should work with most term emus).

Give it a try... i use semerge hdparm hddtemp unzip to test it...

Edit: Ok, it works fine in Eterm, but not from a console... evidentally Eterm doesn't mind my inflating the # of lines in the VT100 code that sets the scroll area, but the console does... it only works when you specify the correct # of lines that are actually VISIBLE on the screen. Anyone know how to get this easily with a command line utility? Of course if you ALWAYS used only the console you can just hack the utility...
Back to top
View user's profile Send private message
wyrickre
n00b
n00b


Joined: 10 Jul 2002
Posts: 27
Location: Colorado Springs, CO

PostPosted: Wed Mar 05, 2003 2:07 am    Post subject: Reply with quote

try
Code:
$ stty size

or
Code:
$ resize

_________________
Geek used to be a 4 letter word; Now it's a 6 figure one.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16114
Location: Colorado

PostPosted: Wed Mar 05, 2003 2:20 am    Post subject: Reply with quote

Don't know why I didn't mention this sooner. Also of interest might be the portage version (under development) of this feature. More info in GF4: Can portage build one package while downloading others?.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Fri Mar 07, 2003 3:48 am    Post subject: Reply with quote

wyrickre wrote:
try
Code:
$ stty size
or
Code:
$ resize

Looking into using this. Also, just updated the version to 0.16 to allow it to work with --oneshot.
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Fri Mar 07, 2003 4:22 am    Post subject: Re: yeah Reply with quote

vaevictus wrote:
I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.

Still waiting to see what you think of recent updates. :-)
Back to top
View user's profile Send private message
Sastraxi
Apprentice
Apprentice


Joined: 25 Feb 2003
Posts: 258

PostPosted: Fri Mar 07, 2003 4:45 am    Post subject: Reply with quote

Is there any way to pause part of the console? Aka. have the top 4 lines have the wget status (ex. 1 of 8, pbar, etc.), and the rest the compile output.
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Fri Mar 07, 2003 8:57 am    Post subject: Reply with quote

Sastraxi wrote:
Is there any way to pause part of the console? Aka. have the top 4 lines have the wget status (ex. 1 of 8, pbar, etc.), and the rest the compile output.

Pause???

I think what you're asking for is something like "split screen"... and no, I don't think that is possible... what I'm doing is redefining the "scroll" window of the screen and then moving the cursor to the top, updating status, and moving it back quickly... just thinking about it it seems that eventually it should "screw up" when in the middle of a big emerge or something and put some emerge output on the top, but I haven't seen that happen yet.

Anyone have any ideas for "split screen" mode? I was just wondering about logging both the download and the emerge (running in the BG) and having the BASH script read the output files very often and draw the screen itself... sounds like a lot of work though if it would work at all ;-)

Would be cool to see an implimentation though.

Edit: Is it possible to "pause" a process in BASH from script?
Back to top
View user's profile Send private message
TheCoop
Veteran
Veteran


Joined: 15 Jun 2002
Posts: 1814
Location: Where you least expect it

PostPosted: Fri Mar 07, 2003 9:29 am    Post subject: Reply with quote

can you do a split screen thing with framebuffers? (like the gentoo logo on bootup)
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Fri Mar 07, 2003 9:50 am    Post subject: Reply with quote

TheCoop wrote:
can you do a split screen thing with framebuffers? (like the gentoo logo on bootup)

Maybe, but that would be outside the scope of what I want to accomplish as I don't want a certain thing to have to be compiled into your kernel for semerge to work properly...
Back to top
View user's profile Send private message
Ari Rahikkala
Guru
Guru


Joined: 02 Oct 2002
Posts: 370
Location: Finland

PostPosted: Wed Aug 20, 2003 10:30 am    Post subject: Reply with quote

dreamer3 wrote:
Edit: Is it possible to "pause" a process in BASH from script?


Well, it's possible to `kill -STOP` one, but I think you mean the output, not the whole process...
_________________
<laurentius> gentoo linux?
<ari> Yesh.
<laurentius> they look horny
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Thu Aug 21, 2003 8:37 pm    Post subject: Reply with quote

It's been so long I have no idea what I was asking...
Back to top
View user's profile Send private message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Sat Feb 28, 2004 2:53 am    Post subject: Reply with quote

why is script not being updated?
Back to top
View user's profile Send private message
dreamer3
Guru
Guru


Joined: 24 Sep 2002
Posts: 553

PostPosted: Sat Feb 28, 2004 6:32 am    Post subject: Reply with quote

Becuase I have a dial-up connection and get tired of downloading/compiling source all the time so I switched to Debian. :-) *ducks*

That being said, my intended answer to the problem:

The get_ebuilds step needs to be broken into two... the first command run is exected with --no-deps and this list is stored as packages that should be emerged and ADDED to the world file.

Then the same emerge needs to be called with --deps-only to return a list of which ebuilds should be emerged with --one-time-only (ie, not added to the world file cause they are only depends).

Now that one has BOTH these lists it might be necessary to call emerge again with neither parameter so that a COMPLETE list is created with the correct installation order. Incase one of the dependencies requires one of the non-deps of a different package, etc...

Only problem with this is it might significantly increase the startup time of the script as emerge needs to be run 3 times... If emerge outputed more detailed info (say if package was gonna be added to world or not) you could just parse that, but it doesn't.

If I still ran Gentoo I'd be happy to write this up but I've honestly seen little interest in this script except for the occasoinal passerby, so I'll leave it as an excersize for the reader... Sorry. :-)

Feel free to hack something together and paste here and I"ll come look at it... I love BASH scripting, but I really don't have the time or inclination to do more at this point.
Back to top
View user's profile Send private message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Sat Feb 28, 2004 11:58 am    Post subject: Reply with quote

Thanks for giving this info, I'm going to try to make a hack for what you just said. PS, debian is a nice distro and has many advantages to gentoo, no need to duck, there are many simultaneous debian/gentoo users here ;)

EDIT: Think I'm going to turn this into a full size wrapper for portage actually, I have a way in my head for something like the 'ehush' script, but with better output, and shows information about how much lines compiled, with ability to make a bug report and display both compile and download information.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Sat Jul 03, 2004 11:15 pm    Post subject: Reply with quote

What's the current status of this enhancement please? Is anyone still actively working on it?
Back to top
View user's profile Send private message
kamagurka
Veteran
Veteran


Joined: 25 Jan 2004
Posts: 1026
Location: /germany/munich

PostPosted: Thu Jul 08, 2004 1:55 pm    Post subject: Re: yeah Reply with quote

vaevictus wrote:
I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :)


yes please. want please.
_________________
If you loved me, you'd all kill yourselves today.
--Spider Jerusalem, the Word
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum