View previous topic :: View next topic |
Author |
Message |
Ant Tux's lil' helper
Joined: 31 Oct 2002 Posts: 129 Location: Australia
|
Posted: Sat Jul 12, 2003 9:31 pm Post subject: Emerge -e world. A script to examine previous compile times. |
|
|
Hey hey!! Success!!
Well after a slight screw up with moving to gnome 2.2 I have finally attempted a rebuild of everything. It worked perfectly the first time, and gnome now runs beautifully. So here's my journey and a helpful little script for all you people interested in know how long your compile has taken for previous packages.
In total, it took 26hours on my P4 1.7Gig Dell Inspiron 8200 to recompile 419 packages. (X, Gnome, OpenOffice, celestia, xephem, mozilla, qt, etc)
I only had a few glitches, libmpeg failed until I down graded my CFLAGS. Then I resumed (emerge -e world --resume) I'm glad I read the forums first, --resume is the best invention since sliced bread.
I also had a problem with OpenOffice filling my hard drive half way through, but since it was the 5th last pacakge I skipped it untill I can make more space.
So, with 26 hours to kill I wrote a script which I hope you'll all like.
First a disclaimer: This is my first python script and I had no documentation to refer to, so I've had to guess the syntax, types and functions from reading the emerge code and python include files. In short, feel free to correct my code.
Anyway, what it does is scan the emerge log file looking for two strings
Code: |
>>> emerge
and
>>> AUTOCLEAN
|
that reference the same package.
It then uses the timestamp to calculate the time it took to compile.
It will also remember the last partial package to give you a running time of the currently compiling package.
Cut the script from below and save it (ie as compileTime)
The usage is
Code: |
./compileTime [-f logfile] pkgname
|
where the package name can be a valid package, package category, 'world' for all packages or 'latest' for the currently compiling package.
Again the code is not perfect and is only based on the log file, however, it seems to work quite well.
Anyway, let me know what you all think and if anyone wants I can post some package compile times to give you an idea of how long it will take.
Ant.
Code: |
#!/usr/bin/env python2.2
# Author: Anthony Higgins.
# Gentoo Forum ID: Ant
# Purpose : Script to parse the emerge log file to calculate package
# : compile times
#
# Disclaimer : This is my first python script written with no python
# : documentation while my Gentoo system rebuilt. While I
# : don't see how this could damage anything, it is still
# : a case of User Beware!!
import sys,string,time
# Global variables
startstring=">>> emerge"
stopstring=">>> AUTOCLEAN:"
emergelog="/var/log/emerge.log"
# Parse the command line
cmdline=sys.argv
if len(cmdline)==2:
searchstr=cmdline[1]
elif len(cmdline)==4:
if cmdline[1]=="-f":
emergelog=cmdline[2]
searchstr=cmdline[3]
else:
emergelog=cmdline[3]
searchstr=cmdline[1]
else:
print "Usage:",cmdline[0],"[-f emergeLog] pkg"
print " : pkg can be any valid package name, category name, 'world' for all packages, or 'latest' for the currently compiling package."
sys.exit(-1)
print "Scanning file",emergelog,"for",searchstr
print
# open logfile
try:
mylogfile=open(emergelog, "r")
except Exception, e:
print "Error:",e
# Function to print package data
def printPkg(pkgdata):
print "Date\t\tHH:MM:SS\tPackage"
print "------------------------------------------------------------------------------"
for set in pkgdata:
if set[5]>0:
t=set[5]-set[4]
hrs=t/60/60
min=t/60-hrs*60
sec=t-min*60-hrs*60*60
timedata=[0,0,0,hrs,min,sec,0,0,0]
if set[3]=="":
print time.strftime("%d/%m/%y",time.localtime(set[4])),"\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]
else:
print time.strftime("%d/%m/%y",time.localtime(set[4])),"\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]+"-"+set[3]
else:
t=long(time.time())-set[4]
hrs=long(t/60/60)
min=long(t/60-hrs*60)
sec=long(t-min*60-hrs*60*60)
timedata=[0,0,0,hrs,min,sec,0,0,0]
if set[3]=="":
print "In progress\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]
else:
print "In progress\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]+"-"+set[3]
print
# Returns the data only for the currently compiling package
def queryLatest(data):
for set in data:
if set[5]==-1:
return set
return []
# Returns the data only for a specific package/category
def queryData(data, pkg):
querydata=[]
for set in data:
if set[1]==pkg or set[0]==pkg:
querydata.append(set)
return querydata
# Scans the next complete set of package data from the input file
def scanNextPkg(file):
startline=""
stopline=""
starttime=0
stoptime=0
pkgInfo=["","","",""]
while startline=="" or stopline=="":
line = file.readline()
if line=="":
if starttime!=0:
# for use to query the currently compiling package
return pkgInfo+[starttime,-1]
else:
return []
#find start line
if line.find(startstring)!=-1:
pkgInfo=packageBreakDown(line.split(" ")[7])
startline=line
starttime=string.atoi(startline.split(":")[0])
stopline=""
#find stop line that matches start line package
elif line.find(stopstring)!=-1:
if line.find(pkgInfo[1])!=-1:
stopline=line
stoptime=string.atoi(stopline.split(":")[0])
return pkgInfo+[starttime, stoptime]
# Builds the total package data by repeatedly calling scanNextFile
def scanFile(file):
data=[]
while 1:
pkgdata=scanNextPkg(file)
if pkgdata==[]:
return data
else:
data.append(pkgdata)
def packageBreakDown(str):
cat=str.split("/")[0]
pkgData=str.split("/")[1].split("-")
if pkgData[len(pkgData)-1][0]=="r":
rev=pkgData[len(pkgData)-1]
ver=pkgData[len(pkgData)-2]
j=len(pkgData)-2
else:
rev=""
ver=pkgData[len(pkgData)-1]
j=len(pkgData)-1
i=1
pkg=pkgData[0]
while i<j:
pkg=pkg+"-"+pkgData[i]
i=i+1
return [cat, pkg, ver, rev]
# Main code starts here
printdata=[]
data=scanFile(mylogfile)
if searchstr=="world": #special case
printdata=data
elif searchstr=="latest":
cur=queryLatest(data)
if len(cur)==6:
printdata=queryData(data, cur[1])
else:
print "Couldn't find a currently compiling package in the log."
else:
printdata=queryData(data, searchstr)
if printdata==[]:
print "Search string",searchstr,"not found."
else:
printPkg(printdata)
mylogfile.close()
|
|
|
Back to top |
|
|
Ant Tux's lil' helper
Joined: 31 Oct 2002 Posts: 129 Location: Australia
|
Posted: Sat Jul 12, 2003 9:50 pm Post subject: |
|
|
Had a thought.
Since I can only connect to the internet via modem *sigh* I always emerge -f pkg before doing a emerge pkg.
Hence the times from my log are only for the compile. However if you download and then compile all in one command, the times the above script produces will be download plus compile time. Keep this in mind.
If emerge logged the fetch stage for each package, I would then have a different start string to scan for.
Ant. |
|
Back to top |
|
|
Pythonhead Developer
Joined: 16 Dec 2002 Posts: 1801 Location: Redondo Beach, Republic of Calif.
|
|
Back to top |
|
|
Ant Tux's lil' helper
Joined: 31 Oct 2002 Posts: 129 Location: Australia
|
Posted: Sat Jul 12, 2003 10:08 pm Post subject: |
|
|
Oh well, at least I got a crash course in python and a greater understanding or emerge.
They both cover what my script does except the 'latest' option.
Code: |
./compileTimes latest
Scanning file /var/log/emerge.log for latest
Date HH:MM:SS Package
------------------------------------------------------------------------------
21/06/03 00:02:00 net-www/apache-1.3.27-r3
12/07/03 00:01:42 net-www/apache-1.3.27-r3
In progress 00:02:22 net-www/apache-2.0.47
|
|
|
Back to top |
|
|
Giorgio Tux's lil' helper
Joined: 18 Jan 2003 Posts: 77 Location: Milano, Italy
|
Posted: Fri Jul 18, 2003 6:42 pm Post subject: |
|
|
What does this "latest" option do?
I can add it into the next genlop release (soon). _________________ app-portage/genlop
A nice emerge.log parser |
|
Back to top |
|
|
Giorgio Tux's lil' helper
Joined: 18 Jan 2003 Posts: 77 Location: Milano, Italy
|
Posted: Fri Jul 18, 2003 6:47 pm Post subject: |
|
|
Uh.. ok, I understand it by myself. _________________ app-portage/genlop
A nice emerge.log parser |
|
Back to top |
|
|
Ant Tux's lil' helper
Joined: 31 Oct 2002 Posts: 129 Location: Australia
|
Posted: Wed Aug 06, 2003 9:13 am Post subject: |
|
|
Sorry about the lack of reply, been ofline for a while.
As you worked out, it just gives you the time taken so far for the currently compiling package.
Ant. |
|
Back to top |
|
|
JustJoe Tux's lil' helper
Joined: 30 Jul 2005 Posts: 80
|
Posted: Tue Oct 25, 2005 10:07 pm Post subject: |
|
|
Hi Ant,
I tried your script but it errors out: Code: | ./CompileTime latest
Scanning file /var/log/emerge.log for latest
Date HH:MM:SS Package
------------------------------------------------------------------------------
05/08/05 Traceback (most recent call last):
File "./CompileTime", line 167, in ?
printPkg(printdata)
File "./CompileTime", line 57, in printPkg
print time.strftime("%d/%m/%y",time.localtime(set[4])),"\t",time.strftime("%H:%M:%S",timedata),"\t",set[0]+"/"+set[1]+"-"+set[2]
ValueError: month out of range |
Any ideas ? |
|
Back to top |
|
|
babykart Guru
Joined: 08 Oct 2004 Posts: 415
|
Posted: Tue Oct 25, 2005 11:21 pm Post subject: |
|
|
why build another tool? genlop make it so easy
Code: | # emerge -pve world | genlop -p
....
Estimated update time: 15 hours, 1 minute. |
PS - sorry for my very bad english _________________ >> Gentoo-FR <<
----- |
|
Back to top |
|
|
JustJoe Tux's lil' helper
Joined: 30 Jul 2005 Posts: 80
|
Posted: Wed Oct 26, 2005 2:17 am Post subject: |
|
|
I'm just curious why this error occurs.
btw: just now i notice i bumped a 2 yr. old thread... sorry for that. (i guess i'd better send Ant a PM. ) |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|