Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
eprogress: track an ebuild's progress
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Gherald2
Guru
Guru


Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA

PostPosted: Wed Jul 14, 2004 4:03 am    Post subject: eprogress: track an ebuild's progress Reply with quote

Ever wondered how long an ebuild is going to take? Here's a quick script to determine how far along it is.

Or perhaps you'd like to see a list of all previous failed/interrupted ebuilds? Just run it w/o arguements and you'll get a pretty good idea.

Code:
#!/bin/bash
#eprogress v0.0.1, July 13, 2004

#When given a package name this scipt is will track the progress of an ebuild by counting the number of .c and .cpp files in the portage temp directory and comparing it to the number of .o files that have been compiled so far.

#When run without arguements it's useful for locating previous failed or interrupted ebuilds so they can be attempted again or deleted manually to free space.

cd /var/tmp/portage
printf "%40s%8s%8s%8s%9s" "PACKAGE" "TOTAL" "LEFT" "DONE" "PERCENT"; echo
for i in $1*; do
   A=`find $i/ -iname "*.c*" | wc -l`
   if test $A -eq 0; then continue; fi
   B=`find $i/ -iname "*.o*" | wc -l`
   C=$[$A-$B]
   D=`expr $B \* 100 / $A`
   if test $D -gt 100; then D="??"; fi
   printf "%40s%8s%8s%8s%7s" "$i" "$A" "$C" "$B" "$D"; echo
done


TODO:

-add java and ".C" file support
-add an interactive mode that rescans every n seconds.
-rename variables

Note that someone attempted a full progress bar here but the project seems abandoned :(

If you know a better way to get this info, let us know!
_________________
Unregistered Linux User #17598363


Last edited by Gherald2 on Wed Jul 14, 2004 3:01 pm; edited 1 time in total
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16102
Location: Colorado

PostPosted: Wed Jul 14, 2004 4:58 am    Post subject: Reply with quote

I've wanted to see something like this. Just some options to consider:
  • Listing file sizes of object being compiled. Perhaps a list of current, plus next 3 (or 5, 10, etc.).
  • A (1 of 35) type counter.


As for knowing a better way to get the info, perhaps using python and the portage system would be more handy. I've only poked at it a little though, so I can't be specific.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
neenee
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1786

PostPosted: Wed Jul 14, 2004 10:36 am    Post subject: Reply with quote

here is a thread with two other scripts which show progress.

but i like how the output of your script is more detailed :wink:
_________________
proud to be a scout and a chronic penguin hugger
Legion of Lore - site
Back to top
View user's profile Send private message
nelz
n00b
n00b


Joined: 24 Apr 2003
Posts: 19
Location: Warrington, UK

PostPosted: Tue Jul 20, 2004 11:04 am    Post subject: Reply with quote

This script fails if you have PORTAGE_TMPDIR set to anywhere other than /var/tmp.

Replace
Code:
cd /var/tmp/portage

with
Code:
source /etc/make.conf
cd $PORTAGE_TMPDIR/portage

[/code]
Back to top
View user's profile Send private message
Gherald2
Guru
Guru


Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA

PostPosted: Tue Jul 20, 2004 11:25 am    Post subject: Reply with quote

Good idea. I'll just leave the post as-is though, unless more people are interested in seeing the script improved.

For such a quick hack, to me it seems usable enough as-is.

But feel free to post any other modifications you think of :)
_________________
Unregistered Linux User #17598363
Back to top
View user's profile Send private message
xchris
Advocate
Advocate


Joined: 10 Jul 2003
Posts: 2824
Location: 45.488291,9.186094

PostPosted: Thu Jul 29, 2004 2:24 pm    Post subject: Reply with quote

i really like this script!!!
10x
_________________
while True:Gentoo()
Back to top
View user's profile Send private message
dub.wav
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2003
Posts: 149
Location: Norway

PostPosted: Tue Aug 10, 2004 1:42 pm    Post subject: Reply with quote

I've written a python version of the bash script above, which does the same thing, except that:
* it adds the following arguments: -i for interactive mode, and -d for number of seconds between updates.
* the TOTAL header comes after DONE.
* it's much faster than the bash script.
example usage:
Code:

eprogress.py gedit
eprogress.py gedit -i -d2 # default delay is 5 seconds
eprogress.py \* # works like the bash script without args.

Btw, I think that it would be a better idea to use a list of known source extensions/object extensions instead of wildcards.
How many different extensions are there? (For objects I've personally only seen .o, and for source .c, .cpp.)
The script is probably a bit too large to post, but I don't have any webspace to use, so here goes.
Code:
import portage, glob, sys, os, fnmatch, time
from optparse import OptionParser

def err(msg, fatal=True, ret=1):
   print >>sys.stderr, "Error:", msg
   if fatal:
      sys.exit(ret)

if os.getuid() != 0:
   err("You have to be root to run this script.")

parser = OptionParser(usage="%prog [options] progname",
      version="%prog 0.0.1")
parser.add_option("-i", action="store_true",
      dest="interactive", default=False,
      help="Interactive mode")
parser.add_option("-d", type="int", dest="delay", default=5,
      help="Number of seconds between updates in interactive mode")

options, args = parser.parse_args()
if len(args) != 1:
   parser.print_help()
   sys.exit(1)
else:
   progname = args[0]

for cfg_file in "/etc/make.conf", "/etc/make.globals":
   try:
      portage_tmpdir = portage.getconfig(cfg_file)["PORTAGE_TMPDIR"]
      break
   except:
      portage_tmpdir = False
      
if not portage_tmpdir:
   err("Could not find PORTAGE_TMPDIR.")

portage_tmpdir += "/portage"

p = os.popen("clear")
clear = p.read()
if p.close() != None:
   clear = False

os.chdir(portage_tmpdir)

#
header_format = "%40s%8s%8s%8s%9s"
result_format = "%40s%8s%8s%8s%7s"
# Borrowed from Gherald's bash script.

while True:
   dirs = []
   dirs_tmp = glob.glob("%s*/work/*" % progname)
   for dir in dirs_tmp:
      if os.path.isdir(dir):
         dirs.append(dir)
   if clear:
      sys.stdout.write(clear)
   for dir in dirs:
      package = dir.split('/')[0]
      source = []
      object = []
      print header_format % ("PACKAGE", "DONE", "TOTAL", "LEFT", "PERCENT")
      for root, dirs, files in os.walk(dir):
         for file in files:
            ext = os.path.splitext(file)[1][1:].lower()
            if fnmatch.fnmatch(ext, "c*"):
               source.append(file)
            elif fnmatch.fnmatch(ext, "o"):
               object.append(file)
                  
      total = len(source)
      left = total - len(object)
      done = len(object)
      try:
         percent = done * 100 / total
      except:
         pass
      print result_format % (package, done, total, left, percent)
   if not options.interactive:
      break
   else:
      time.sleep(options.delay)
Back to top
View user's profile Send private message
discomfitor
l33t
l33t


Joined: 21 Feb 2003
Posts: 927
Location: None

PostPosted: Tue Aug 10, 2004 8:35 pm    Post subject: Reply with quote

Seems like it could use a bit more work; on gnumeric I supposedly had 33 files left to compile and was at 91% as it was unmerging the old one and finishing. I like how it works though...reminds me of this project, but without the cool little bar. Then again, the other one would break a lot of emerges.

Keep up the good ideas!
_________________
There is no substitute for experience.
Imperfection indicates a lack of effort.
Back to top
View user's profile Send private message
Gherald2
Guru
Guru


Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA

PostPosted: Wed Aug 11, 2004 4:48 am    Post subject: Reply with quote

Nice going dub.wav, I've been toying with python lately and your version looks like it'll be fun to play around with.

Darckness, these scripts are based on the assumption that the ratio of .o to .c* files will give a crude indication of the progress. There will always be some ebuilds that don't behave this way, but with portage-ng (which will likely have a full-fledged plugin interface) on the roadmap those special cases aren't really worth investigating. And I did mention the project you linked to.. as you said, it breaks a lot of emerges and of course involves editing portage. I wanted something less obtrusive :)
_________________
Unregistered Linux User #17598363
Back to top
View user's profile Send private message
discomfitor
l33t
l33t


Joined: 21 Feb 2003
Posts: 927
Location: None

PostPosted: Wed Aug 11, 2004 6:17 am    Post subject: Reply with quote

Yeah, my eyes have been failing me in the color spectrum recently, so I missed your reference to the deceased project. It seems that for larger packages it becomes increasingly inaccurate. Mozilla was an entertaining one; it finished at 33%.
_________________
There is no substitute for experience.
Imperfection indicates a lack of effort.
Back to top
View user's profile Send private message
fallow
Bodhisattva
Bodhisattva


Joined: 08 Jan 2004
Posts: 2206
Location: Poland

PostPosted: Mon Sep 06, 2004 7:57 am    Post subject: Reply with quote

I like the script from Gherald , nice 4 me .
Thx & Greetings ;)
_________________
"Time is a companion that goes with us on a journey. It reminds us to cherish each moment, because it will never come again. What we leave behind is not as important as how we have lived" J-L. Picard ;)
Back to top
View user's profile Send private message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Tue Mar 15, 2005 11:12 am    Post subject: Reply with quote

dub.wav wrote:
I've written a python version of the bash script above, which does the same thing, except that:
* it adds the following arguments: -i for interactive mode, and -d for number of seconds between updates.
* the TOTAL header comes after DONE.
* it's much faster than the bash script.
What are you smoking? -- 99% of the work is done by os.walk which is about 40x slower than using find. I ran some benchmarks and for a larger build dir like glibc, os.walk takes up to 3 seconds while find takes 0.1 seconds!
Back to top
View user's profile Send private message
roo_
Tux's lil' helper
Tux's lil' helper


Joined: 07 Feb 2004
Posts: 91

PostPosted: Sun Apr 17, 2005 3:18 pm    Post subject: Reply with quote

Hackeron wrote:
dub.wav wrote:
I've written a python version of the bash script above, which does the same thing, except that:
* it adds the following arguments: -i for interactive mode, and -d for number of seconds between updates.
* the TOTAL header comes after DONE.
* it's much faster than the bash script.
What are you smoking? -- 99% of the work is done by os.walk which is about 40x slower than using find. I ran some benchmarks and for a larger build dir like glibc, os.walk takes up to 3 seconds while find takes 0.1 seconds!


The bash script is a ton faster.
Back to top
View user's profile Send private message
radarsat1
n00b
n00b


Joined: 13 Jul 2005
Posts: 32

PostPosted: Wed Jul 13, 2005 2:14 pm    Post subject: Reply with quote

I like this bash script!
I ran it with the following command line:

Code:
watch "sudo ./eprogress \`grep === /var/log/emerge.log | tail -n 1 | cut -f 3 -d / | cut -f 1 -d -\`"


there's probably a better way to do it. but anyways it lets me watch the progress of the current package being compiled, whatever that happens to be. could probably parse the progress better and pipe it to zenity for a graphical progress bar.
(needs root access unfortunately)
Back to top
View user's profile Send private message
radarsat1
n00b
n00b


Joined: 13 Jul 2005
Posts: 32

PostPosted: Wed Jul 13, 2005 2:32 pm    Post subject: Reply with quote

hm, actually this one seems to work better:

Code:

watch "sudo ./eprogress \`grep === /var/log/emerge.log | tail -n 1 | cut -f 3 -d / | cut -f 1 -d :\` 2>&1"
Back to top
View user's profile Send private message
Earthwings
Administrator
Administrator


Joined: 14 Apr 2003
Posts: 7751
Location: Karlsruhe, Germany

PostPosted: Wed Jul 13, 2005 2:43 pm    Post subject: Reply with quote

See also genlop (emerge genlop), especially
Code:
genlop --current

_________________
KDE 4.13 - Get It While It's Hot!
Back to top
View user's profile Send private message
Cinder6
l33t
l33t


Joined: 05 Aug 2004
Posts: 767
Location: California

PostPosted: Thu Jul 14, 2005 3:51 am    Post subject: Reply with quote

I have always wished that the -q option in emerge would just show a progressbar and write the standard output to a virtual screen, so if the emerge fails, it can still show you the error message.
_________________
Knowledge is power.
Power corrupts.
Study hard.
Be evil.

Ugly Overload
Back to top
View user's profile Send private message
cruise
n00b
n00b


Joined: 01 Dec 2003
Posts: 48

PostPosted: Tue Sep 06, 2005 10:15 am    Post subject: Okay, my tweaked version Reply with quote

Code:

#!/bin/bash
#eprogress v0.0.2, Sep 06, 2005

#When given a package name this scipt is will track the progress of an ebuild by counting the number of .c and .cpp files in the portage temp directory and comparing it to the number of .o files that have been compiled so far.

#When run without arguments it's useful for locating previous failed or interrupted ebuilds so they can be attempted again or deleted manually to free space.

PORTAGE_TMPDIR=/var/tmp
source /etc/make.conf
cd ${PORTAGE_TMPDIR}/portage

printf "%40s%8s%8s%8s%9s" "PACKAGE" "TOTAL" "LEFT" "DONE" "PERCENT"; echo
for i in $1*; do
   A=`find $i/ -iname "*.c*" | wc -l`
   if test $A -eq 0; then continue; fi
   B=`find $i/ -iname "*.o" | egrep '/\.' | wc -l`
   C=$[$A-$B]
   D=`expr $B \* 100 / $A`
   if test $D -gt 100; then D="??"; fi
   printf "%40s%8s%8s%8s%7s" "$i" "$A" "$C" "$B" "$D"; echo
done


Added the check for a different tmp dir, and included an egrep to ignore anything with a leading '.' The CVS e17 version of edje, for example, seems to have copies of it's .o files in .lib directories, which really screws up the progress counting :P

Again, quick and dirty, but it works on mine system for the moment, so what the heck :D
_________________
"quantam sufficit"
[ cruise / casual-tempest.net / transference.org ]
Back to top
View user's profile Send private message
iplayfast
l33t
l33t


Joined: 08 Jul 2002
Posts: 642
Location: Cambridge On,CA

PostPosted: Fri Sep 15, 2006 7:29 pm    Post subject: Reply with quote

Oh man I have to say. I really like what you've done with this. Seems like the original thread still has some activity on it as well. 8O http://forums.gentoo.org/viewtopic-t-139562-postdays-0-postorder-asc-start-25.html
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum