Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Emerge -e world. A script to examine previous compile times.
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Ant
Tux's lil' helper
Tux's lil' helper


Joined: 31 Oct 2002
Posts: 129
Location: Australia

PostPosted: Sat Jul 12, 2003 9:31 pm    Post subject: Emerge -e world. A script to examine previous compile times. Reply with quote

Hey hey!! Success!!

Well after a slight screw up with moving to gnome 2.2 I have finally attempted a rebuild of everything. It worked perfectly the first time, and gnome now runs beautifully. So here's my journey and a helpful little script for all you people interested in know how long your compile has taken for previous packages.

In total, it took 26hours on my P4 1.7Gig Dell Inspiron 8200 to recompile 419 packages. (X, Gnome, OpenOffice, celestia, xephem, mozilla, qt, etc)

I only had a few glitches, libmpeg failed until I down graded my CFLAGS. Then I resumed (emerge -e world --resume) I'm glad I read the forums first, --resume is the best invention since sliced bread.

I also had a problem with OpenOffice filling my hard drive half way through, but since it was the 5th last pacakge I skipped it untill I can make more space.

So, with 26 hours to kill I wrote a script which I hope you'll all like.

First a disclaimer: This is my first python script and I had no documentation to refer to, so I've had to guess the syntax, types and functions from reading the emerge code and python include files. In short, feel free to correct my code.

Anyway, what it does is scan the emerge log file looking for two strings
Code:

>>> emerge
and
>>> AUTOCLEAN

that reference the same package.

It then uses the timestamp to calculate the time it took to compile.

It will also remember the last partial package to give you a running time of the currently compiling package.

Cut the script from below and save it (ie as compileTime)
The usage is
Code:

./compileTime [-f logfile] pkgname

where the package name can be a valid package, package category, 'world' for all packages or 'latest' for the currently compiling package.

Again the code is not perfect and is only based on the log file, however, it seems to work quite well.

Anyway, let me know what you all think and if anyone wants I can post some package compile times to give you an idea of how long it will take.

Ant.

Code:

#!/usr/bin/env python2.2
# Author: Anthony Higgins.
# Gentoo Forum ID: Ant
# Purpose : Script to parse the emerge log file to calculate package
#         : compile times
#
# Disclaimer : This is my first python script written with no python
#            : documentation while my Gentoo system rebuilt. While I
#            : don't see how this could damage anything, it is still
#            : a case of User Beware!!

import sys,string,time

# Global variables
startstring=">>> emerge"
stopstring=">>> AUTOCLEAN:"
emergelog="/var/log/emerge.log"

# Parse the command line
cmdline=sys.argv

if len(cmdline)==2:
   searchstr=cmdline[1]
elif len(cmdline)==4:
   if cmdline[1]=="-f":
      emergelog=cmdline[2]
      searchstr=cmdline[3]
   else:
      emergelog=cmdline[3]
      searchstr=cmdline[1]
else:
   print "Usage:",cmdline[0],"[-f emergeLog] pkg"
   print "     : pkg can be any valid package name, category name, 'world' for all packages, or 'latest' for the currently compiling package."
   sys.exit(-1)

print "Scanning file",emergelog,"for",searchstr
print

# open logfile
try:
   mylogfile=open(emergelog, "r")
except Exception, e:
   print "Error:",e

# Function to print package data
def printPkg(pkgdata):
   print "Date\t\tHH:MM:SS\tPackage"
   print "------------------------------------------------------------------------------"
   for set in pkgdata:
      if set[5]>0:
         t=set[5]-set[4]
         hrs=t/60/60
         min=t/60-hrs*60
         sec=t-min*60-hrs*60*60
         timedata=[0,0,0,hrs,min,sec,0,0,0]
         if set[3]=="":
            print time.strftime("%d/%m/%y",time.localtime(set[4])),"\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]
         else:
            print time.strftime("%d/%m/%y",time.localtime(set[4])),"\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]+"-"+set[3]
      else:
         t=long(time.time())-set[4]
         hrs=long(t/60/60)
         min=long(t/60-hrs*60)
         sec=long(t-min*60-hrs*60*60)
         timedata=[0,0,0,hrs,min,sec,0,0,0]
         if set[3]=="":
            print "In progress\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]
         else:
            print "In progress\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]+"-"+set[3]
   print

# Returns the data only for the currently compiling package
def queryLatest(data):
   for set in data:
      if set[5]==-1:
         return set
   return []

# Returns the data only for a specific package/category
def queryData(data, pkg):
   querydata=[]
   for set in data:
      if set[1]==pkg or set[0]==pkg:
         querydata.append(set)
   return querydata

# Scans the next complete set of package data from the input file
def scanNextPkg(file):
   startline=""
   stopline=""
   starttime=0
   stoptime=0
   pkgInfo=["","","",""]

   while startline=="" or stopline=="":
      line = file.readline()
      if line=="":
         if starttime!=0:
         # for use to query the currently compiling package
            return pkgInfo+[starttime,-1]
         else:
            return []

      #find start line
      if line.find(startstring)!=-1:
         pkgInfo=packageBreakDown(line.split(" ")[7])
         startline=line
         starttime=string.atoi(startline.split(":")[0])
         stopline=""

      #find stop line that matches start line package
      elif line.find(stopstring)!=-1:
         if line.find(pkgInfo[1])!=-1:
            stopline=line
            stoptime=string.atoi(stopline.split(":")[0])


   return pkgInfo+[starttime, stoptime]


# Builds the total package data by repeatedly calling scanNextFile
def scanFile(file):
   data=[]
   while 1:
      pkgdata=scanNextPkg(file)
      if pkgdata==[]:
         return data
      else:
         data.append(pkgdata)

def packageBreakDown(str):
   cat=str.split("/")[0]
   pkgData=str.split("/")[1].split("-")
   if pkgData[len(pkgData)-1][0]=="r":
      rev=pkgData[len(pkgData)-1]
      ver=pkgData[len(pkgData)-2]
      j=len(pkgData)-2
   else:
      rev=""
      ver=pkgData[len(pkgData)-1]
      j=len(pkgData)-1
   i=1
   pkg=pkgData[0]
   while i<j:
      pkg=pkg+"-"+pkgData[i]
      i=i+1
   return [cat, pkg, ver, rev]

# Main code starts here
printdata=[]
data=scanFile(mylogfile)

if searchstr=="world": #special case
   printdata=data
elif searchstr=="latest":
   cur=queryLatest(data)
   if len(cur)==6:
      printdata=queryData(data, cur[1])
   else:
      print "Couldn't find a currently compiling package in the log."
else:
   printdata=queryData(data, searchstr)

if printdata==[]:
   print "Search string",searchstr,"not found."
else:
   printPkg(printdata)

mylogfile.close()
Back to top
View user's profile Send private message
Ant
Tux's lil' helper
Tux's lil' helper


Joined: 31 Oct 2002
Posts: 129
Location: Australia

PostPosted: Sat Jul 12, 2003 9:50 pm    Post subject: Reply with quote

Had a thought.

Since I can only connect to the internet via modem *sigh* I always emerge -f pkg before doing a emerge pkg.

Hence the times from my log are only for the compile. However if you download and then compile all in one command, the times the above script produces will be download plus compile time. Keep this in mind.

If emerge logged the fetch stage for each package, I would then have a different start string to scan for.

Ant.
Back to top
View user's profile Send private message
Pythonhead
Developer
Developer


Joined: 16 Dec 2002
Posts: 1801
Location: Redondo Beach, Republic of Calif.

PostPosted: Sat Jul 12, 2003 9:57 pm    Post subject: Reply with quote

Here are some similar tools:

http://pollycoke.org/genlop.html
http://www.l8nite.net/projects/splat/
Back to top
View user's profile Send private message
Ant
Tux's lil' helper
Tux's lil' helper


Joined: 31 Oct 2002
Posts: 129
Location: Australia

PostPosted: Sat Jul 12, 2003 10:08 pm    Post subject: Reply with quote

Oh well, :roll: at least I got a crash course in python and a greater understanding or emerge.

They both cover what my script does except the 'latest' option.

Code:

./compileTimes latest
Scanning file /var/log/emerge.log for latest
 
Date            HH:MM:SS        Package
------------------------------------------------------------------------------
21/06/03        00:02:00        net-www/apache-1.3.27-r3
12/07/03        00:01:42        net-www/apache-1.3.27-r3
In progress     00:02:22        net-www/apache-2.0.47

Back to top
View user's profile Send private message
Giorgio
Tux's lil' helper
Tux's lil' helper


Joined: 18 Jan 2003
Posts: 77
Location: Milano, Italy

PostPosted: Fri Jul 18, 2003 6:42 pm    Post subject: Reply with quote

What does this "latest" option do?
I can add it into the next genlop release (soon).
_________________
app-portage/genlop
A nice emerge.log parser
Back to top
View user's profile Send private message
Giorgio
Tux's lil' helper
Tux's lil' helper


Joined: 18 Jan 2003
Posts: 77
Location: Milano, Italy

PostPosted: Fri Jul 18, 2003 6:47 pm    Post subject: Reply with quote

Uh.. ok, I understand it by myself. :D
_________________
app-portage/genlop
A nice emerge.log parser
Back to top
View user's profile Send private message
Ant
Tux's lil' helper
Tux's lil' helper


Joined: 31 Oct 2002
Posts: 129
Location: Australia

PostPosted: Wed Aug 06, 2003 9:13 am    Post subject: Reply with quote

8)

Sorry about the lack of reply, been ofline for a while.

As you worked out, it just gives you the time taken so far for the currently compiling package.

Ant.
Back to top
View user's profile Send private message
JustJoe
Tux's lil' helper
Tux's lil' helper


Joined: 30 Jul 2005
Posts: 80

PostPosted: Tue Oct 25, 2005 10:07 pm    Post subject: Reply with quote

Hi Ant,

I tried your script but it errors out:
Code:
./CompileTime latest
Scanning file /var/log/emerge.log for latest

Date            HH:MM:SS        Package
------------------------------------------------------------------------------
05/08/05        Traceback (most recent call last):
  File "./CompileTime", line 167, in ?
    printPkg(printdata)
  File "./CompileTime", line 57, in printPkg
    print time.strftime("%d/%m/%y",time.localtime(set[4])),"\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]
ValueError: month out of range


Any ideas ?
Back to top
View user's profile Send private message
babykart
Guru
Guru


Joined: 08 Oct 2004
Posts: 415

PostPosted: Tue Oct 25, 2005 11:21 pm    Post subject: Reply with quote

why build another tool? :? genlop make it so easy
Code:
# emerge -pve world | genlop -p
....
Estimated update time: 15 hours, 1 minute.



PS - sorry for my very bad english :(
_________________
>> Gentoo-FR <<
-----
Back to top
View user's profile Send private message
JustJoe
Tux's lil' helper
Tux's lil' helper


Joined: 30 Jul 2005
Posts: 80

PostPosted: Wed Oct 26, 2005 2:17 am    Post subject: Reply with quote

I'm just curious why this error occurs.


btw: just now i notice i bumped a 2 yr. old thread... sorry for that. (i guess i'd better send Ant a PM. :lol: )
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum