Python Newbie looking for Gentoo related Python examples.

[syn_ack]

I've just began learning how to use/program in Python. Running Python 2.3.3 and have all of the Python 2.3.3 related pdf's downloaded, "Learning Python" by Oreily and a ton of "beginning Python" tutorials to keep me busy for a while.

I was wondering if someone can point me to a Python script/program that is used in Gentoo or one that pertains/relates to, used by Gentoo or Portage,....that I could examine as a real world example. Preferrably something somewhat simple if this exists.

If this doesn't make sense then let me know. I'm new, so I might not even know if what I'm asking for has been correctly stated. I've been running Gentoo for about 6 mnths now and Luv it. Only Hickup was the new Genkernel which forced me to learn how to compile manually. Great experience and will never have the need for Genkernel again. Heh..

I guess what I'm really wanting are some python scripts/programs that people have made or developed in regards to real world Gentoo operations that I can use to examine as examlpes to help speed up the learning process. Again. Something preferrably simple that a newb can digest.

Out of work right now so Python and Gentoo are going to be my main focus.
Forgot to mention that I have found some Python/Gentoo related scripts. Looking for more

Much appreciated.

[dr_strange]

I am also a Python newbie, three weeks now... experimenting with simple scripts and simple wxPython applications. We could share our experiences maybe.
As for Gentoo related Python stuff, portage.py and esearch.py spring to mind at once, although portage.py is not the easiest source to stomach (at least for me).

[syn_ack]

I am also a Python newbie, three weeks now... experimenting with simple scripts and simple wxPython applications. We could share our experiences maybe.

I'm game. I'm still going through the tutorials making sure that I have atleast a grasp of the basics before I allow myself to get too lost in the advanced stuff thereby shooting myself in the foot.

As for Gentoo related Python stuff, portage.py and esearch.py spring to mind at once, although portage.py is not the easiest source to stomach (at least for me).

Here's a good link to a post with some good info and examples:
http://forums.gentoo.org/viewtopic.php? ... n+examples

Just do a search on those users that you see in that post and I'm sure you'll find a ton of stuff. Right now I'm going through some basic tutorials. I've never programmed before in anything so I'm trying to be a good student. Heh..

I'll email you my email address if you would like. Just let me know.

Thanks

[dr_strange]

Yes, I remember that topic, nice thread.

We could post our little scripties here and comment on each other and maybe get someadvice from the gurus out there, hm? I'll start with one when I get home.

[dr_strange]

Be sure to download Dive into Python from http://diveintopython.org - it is very good IMHO.

[dr_strange]

OK, so here's a very basic script I wrote in the weekend; basically it takes the current directory and returns the sum of the size of files in it plus size of all subdirectories - sort of a Python du. Please feel free to comment what could be better - I am sure it is clumsy in many ways.


pydu.py

Code: Select all

#!/usr/bin/env python

import os
from os.path import join, getsize, isfile
cwd = os.getcwd()
dirlist = os.listdir(cwd)
dirdir = {}
dirsum = 0
dirlist2 = []
filesonly = 0
print "Calculating..."
for basedir in dirlist:
    filesum = 0
    if basedir in ["proc", "sys", "dev", "mnt"]:
        continue
    if isfile(join(cwd, basedir)):
        filesum = filesum + getsize(join(cwd, basedir))
	filesonly = filesonly + getsize(join(cwd, basedir))
    else:	
        for root, dirs, files in os.walk(join(cwd, basedir)):
            try:
                filesum = filesum + sum([getsize(join(root, name)) for name in files])
		
	    except OSError:
	        continue    
        dirlist2.append(basedir) 
    dirdir[basedir] = filesum 
    dirsum = dirsum + filesum
    

print "\x1b[34;01mFiles: \x1b[0m", filesonly, "bytes"
a = int(float(filesonly)/float(dirsum)*100)
print a*"#"
for basedir in dirlist2:
    print "\x1b[34;01m" + basedir + ": \x1b[0m" + str(dirdir[basedir]) + " bytes"
    a = float(dirdir[basedir])
    b = int(float(a/dirsum)*100)
    print b*"#" 
print "============================"
print "\x1b[31;01mTotal: ", dirsum/1024, "KBytes\x1b[0m"

[far]

The whitespace was screwed up when you did copy paste. This is what I think you meant:

Code: Select all

#!/usr/bin/env python

import os
from os.path import join, getsize, isfile
cwd = os.getcwd()
dirlist = os.listdir(cwd)
dirdir = {}
dirsum = 0
dirlist2 = []
filesonly = 0
print "Calculating..."
for basedir in dirlist:
    filesum = 0
    if basedir in ["proc", "sys", "dev", "mnt"]:
        continue
    if isfile(join(cwd, basedir)):
        filesum = filesum + getsize(join(cwd, basedir))
        filesonly = filesonly + getsize(join(cwd, basedir))
    else:   
        for root, dirs, files in os.walk(join(cwd, basedir)):
            try:
                filesum = filesum + sum([getsize(join(root, name))
                                         for name in files])
            except OSError:
                continue   
    dirlist2.append(basedir)
    dirdir[basedir] = filesum
    dirsum = dirsum + filesum
   

print "\x1b[34;01mFiles: \x1b[0m", filesonly, "bytes"
a = int(float(filesonly)/float(dirsum)*100)
print a*"#"
for basedir in dirlist2:
    print "\x1b[34;01m" + basedir + ": \x1b[0m" \
          + str(dirdir[basedir]) + " bytes"
    a = float(dirdir[basedir])
    b = int(float(a/dirsum)*100)
    print b*"#"
print "============================"
print "\x1b[31;01mTotal: ", dirsum/1024, "KBytes\x1b[0m"

[dr_strange]

yes, thank you

[syn_ack]

Be sure to download Dive into Python from http://diveintopython.org - it is very good IMHO.

Sweet. Thanks for the Link and the Code.

"dr_strange", How long did it take you to code this? Are you using other examples and just doing a trial and error type approach? I look at some of the Python scripts that people have wrote and I'm just mezmorized at how they got from Start to Finish.

?? I also noticed that you use "#!/usr/bin/env python" Is this a standard practice? I thought you only needed this if python wasn't in your path? The only reason that I can think of for doing this would be if your giving your scripts to other people. Any other reasoning??

Thanks

[dr_strange]

It took me maybe an hour of experimenting to get the output as I wanted. The base of the code is a short example from the Python Library Reference at python.org. Basically I wanted to see how the os.walk() function works in case I need it in other scripts.

I always put #!/usr/bin/env python in there, just in case. I don't know if that's a desirable practice, I'm just a newbie still...

[dr_strange]

As for "debugging" my code I usually scatter some print statements across the code printing out relevant variables to see if they get the correct values, trying to narrow down errors if any.

[dr_strange]

Here's the very first Python script I wrote (not that I've written many, mind you ), which I'd love to tweak to "perfection". Here is how it looks:

Code: Select all

#!/usr/bin/env python

import sys,os,string
from time import sleep

while 1:
   os.system("ps axh -o pid= -o comm=  -o stat=fbprocmenu > /usr/share/commonbox/fbproc.txt")
   os.system("touch /usr/share/commonbox/fbprocmenu.txt")
   fh = open('/usr/share/commonbox/fbproc.txt', 'r+')
   ft = open('/usr/share/commonbox/fbprocmenu.txt', 'r+')
   ft.write('[begin]  (procinfo) {}\n')
   for line in fh.readlines():
      line = line[:-1]
      line = "[submenu]  "+"("+line+") {}\n    [exec] (kill) {kill -9 "+line+"}\n   [exec]"\
      +" (restart) {kill -18 "+line+"}\n    [exec] (terminate) {kill -15 "+line+"}\n[end]\n"
      ft.write(line)
   ft.write('[end]')   
   fh.close()
   ft.close()
   sleep(5)
   os.system("rm /usr/share/commonbox/fbprocmenu.txt")

What it does is parses the output of ps axh and creates a procinfo submenu for the fluxbox menu (you have to create a submenu with the path to fbprocmenu.txt in your regular flux menu for it to work), kind of pekwm-fashion. Now it uses os calls to the ps program to accomplish this, but I am sure there is a more elegant way of getting process info out of the system, maybe directly from /proc. I'll try to work out something. Any advices welcome.

[far]

?? I also noticed that you use "#!/usr/bin/env python" Is this a standard practice? I thought you only needed this if python wasn't in your path? The only reason that I can think of for doing this would be if your giving your scripts to other people.

"#!/usr/bin/env python" works only if python is in your path. Otherwise you would use "#!/usr/bin/python" or whatever, but then you have to know the path to the Python interpreter. The env program can usually be assumed to be in /usr/bin, but it is by no means certain.

If you are doing a bigger project and using the python distutils, they will automatically substitute the "shebang" (as it's called) with the location of the python executable upon installation.

[dr_strange]

I have made some progress with the procinfo script; I have defined 3 functions that get out relevant info (pid, name, status) out of /proc in a nice list of dictionaries:

Code: Select all

#!/usr/bin/env python

#------------------------ sample process info script -------------------------

import sys, os
from os.path import isdir, isfile, join
def getprocesses():
    """Sort out process ids from /proc"""
    proccontent = os.listdir("/proc")
    dirsonly = [d for d in proccontent if isdir(join("/proc", d))]
    proclist = [elem for elem in dirsonly if elem[0] in ['1', '2', '3', '4', '5', '6', '7', '8',
'9']]
    proclist.sort()
    return proclist

def getprocinfo(procid):
    """Get info about an individual process."""
    procinfo = {}
    procpath = "/proc/%s/status" % procid
    statfile = open(procpath, 'r')
    procinfo["pid"] = procid
    for line in statfile.readlines():
        if "Name" in line:
            procinfo["Name"] = line.split(":")[1].strip()
        elif "State" in line:
            procinfo["State"] = line.split(":")[1].strip()
        else:
            break
    return procinfo

def makeprocdictlist(proclist):
    """Construct list of procinfo dictionaries"""
    procdictlist = [getprocinfo(pid) for pid in proclist] 
    return procdictlist

procs = getprocesses()
print str(procs)
info = makeprocdictlist(procs)
print str(info)

(the print statements at the bottom are just for checking)
Now all I have to do is make a nicely formatted fluxbox submenu from the output of makeprocdictlist().

[dr_strange]

Okay, works now. This is how it looks:

Code: Select all

#!/usr/bin/env python

#------------------------ Process info script for fluxbox menu-------------------

import sys, os
from os.path import isdir, isfile, join
from time import sleep
def getprocesses():
    """Sort out process ids from /proc"""
    proccontent = os.listdir("/proc")
    dirsonly = [d for d in proccontent if isdir(join("/proc", d))]
    proclist = [elem for elem in dirsonly if elem[0] in ['1', '2', '3', '4', '5', '6', '7', '8', '9']]
    proclist.sort()
    return proclist

def getprocinfo(procid):
    """Get info about an individual process."""
    procinfo = {}
    procpath = "/proc/%s/status" % procid
    statfile = open(procpath, 'r')
    procinfo["pid"] = procid
    for line in statfile.readlines():
        if "Name" in line:
            procinfo["Name"] = line.split(":")[1].strip()
        elif "State" in line:
            procinfo["State"] = line.split(":")[1].split(" ")[0].strip()
        else:
            break
    return procinfo

def makeprocdictlist(proclist):
    """Construct list of procinfo dictionaries"""
    procdictlist = [getprocinfo(pid) for pid in proclist] 
    return procdictlist

while 1:
    procs = getprocesses()
    info = makeprocdictlist(procs)
    os.remove("/usr/share/commonbox/fbprocmenu.txt")
    try:
        outfile = open('/usr/share/commonbox/fbprocmenu.txt', 'w+')
        outfile.write('[begin]  (procinfo) {}\n')
        for elem in info:
            line = "[submenu]  ("+elem["Name"]+" "+elem["pid"]+" "+elem["State"]+") {}\n    [exec] (kill) {kill -9 "+elem["pid"]+"}\n   [exec]"\
		+" (restart) {kill -18 "+elem["pid"]+"}\n    [exec] (terminate) {kill -15 "+elem["pid"]+"}\n[end]\n"
            outfile.write(line)
        outfile.close()
        sleep(5)
    except IOError:
        print "Cannot create fbprocmenu file."
        sys.exit(0)

I think I'll post it in the Tips&Tricks section, maybe a few will find it interesting enough to try out.

[syn_ack]

"#!/usr/bin/env python" works only if python is in your path. Otherwise you would use "#!/usr/bin/python" or whatever, but then you have to know the path to the Python interpreter. The env program can usually be assumed to be in /usr/bin, but it is by no means certain.

If you are doing a bigger project and using the python distutils, they will automatically substitute the "shebang" (as it's called) with the location of the python executable upon installation.

Hmmm. Ok. I guess I'm alittle confused then. I'm sure you can help clear this up though. Two things I don't quite understand fully is where's the best place to place my python scripts and should I be creating all my scripts as "root" or as a normal user? So far I've done everything as a normal user. (kind of thinking of security here but not really to worried about..just wondering)

This is what I see in "/usr/bin":

Code: Select all

 /usr/bin/env and /usr/bin/python -> python2.3.

This is what I've done and would like to do. Please let me know if this is correct or what's correct if possible.

1) I would like to beable to have the ability to execute any of the python scripts that I create, by name, from any given directory so that I don't have to put the full complete path to the script in the terminal everytime I wana execute a python script as a normal user or as root.

2) Rightnow I've paritially fullfilled part of #1 above by giving my *.py scripts +x permissions and putting them in the "/usr/local/bin" directory. Right now they all start with "#!/usr/bin/python".
So from a terminal as a normal user and being in any given directory for example, I type "./junk.py" and it will execute. (not as root though) (thats easily fixed I just don't know whats proper. To have full root access or full user access or both?)

But from the same terminal session I cannot simply type in "python junk.py" without putting the full path to "junk.py" to execute the said script unless ofcourse I'm actually in the same directory.

So I would think that I would need to know the path that Python searches and then just adjust my user and root path to point to the same search path that Python looks in, put all my *.py scripts there, and all would be well.

The only other thing that I can think of is what's listed below but I'm unsure of which path below is correct and if any of what I'm explaining is correct or not.

Code: Select all

>>> import sys
>>> sys.path
['', '/usr/lib/python23.zip', '/usr/lib/python2.3', '/usr/lib/python2.3/plat-linux2', '/usr/lib/python2.3/lib-tk', '/usr/lib/python2.3/lib-dynload', '/usr/lib/python2.3/site-packages', '/usr/lib/portage/pym']

I'm sure I'll find most of the answers to my own questions throughout the day. Any opinions are appreciated though.

Thanks

[dr_strange]

I also have a question. I have read in an article about optimizing Python code by G. van Rossum that in cases like this:

Code: Select all

def makeprocdictlist(proclist):
    """Construct list of procinfo dictionaries"""
    procdictlist = [getprocinfo(pid) for pid in proclist]
    return procdictlist 

it is better to use the map function instead of for loops, as mapping pushes the code into C and that speeds up executions as opposed to calling the function for each item in the list. But how can I refer to the members of the list without iterating over the list? p = map(getprocinfo(pid), proclist) does not work, it gives an error because pid is unreferenced. One solution I can think of is to subclass the string class and make getprocinfo() a method of it - then map(pid.getprocinfo(), proclist) would work. Is there any other/better way to do this?

[syn_ack]

Hey dr_strange,

I joined up to this list today and you have some real pro's that help moderate in a very kind fasion might I add.. I was very impressed with the responses.

Your way beyond where I'm at but I do know that you will get some topnotch help in a very kind and timely manner. Not that we don't get that here. I was just really impressed throughout the whole day that as I read all the different emails as they came in. I actually had the Author of the book that I'm currently using respond to one of my posts. Kind of funny. He was very open minded and honest. You will get your question answered and then some.

http://mail.python.org/mailman/listinfo/tutor

Thats where you join and here's a list of the Archives.

http://mail.python.org/pipermail/tutor/

[dr_strange]

Thanks, I'll check it out.

[syn_ack]

Thanks, I'll check it out.

I look forward to your post and the response that you receive.

Do you have an IM client like Ymessenger?

[dr_strange]

I do have ymessenger, yes, although I rarely use it nowadays. I'm on freenode on irc (#gentoo, #gentoo-doc and #gentoo-hu channels) in the evenings.
On yahoo my nick is percival64.

[far]

p = map(getprocinfo(pid), proclist) does not work,

try treating getprocinfo as a variable:

Code: Select all

p = map(getprocinfo, proclist)

[far]

'Two things I don't quite understand fully is where's the best place to place my python scripts and should I be creating all my scripts as "root" or as a normal user?

If you are the only one who is supposed to run these scripts, you might as well put them in a subdir of your home dir (for instance, ~/bin/) and add it to your $PATH. You can also let other users run your scripts by giving them read and execute rights to ~/bin and ~/bin/yourscript.py.

If the scripts are supposed to be seen as "part of the system" the /usr/local/bin is as good a place as any. Install them as root, don't run them as root unless necessary.

1) I would like to beable to have the ability to execute any of the python scripts that I create, by name, from any given directory so that I don't have to put the full complete path to the script in the terminal everytime I wana execute a python script as a normal user or as root.

That is what $PATH is for.

So from a terminal as a normal user and being in any given directory for example, I type "./junk.py" and it will execute.

No, you type "junk.py". If you type "./junk.py" it will execute the junk.py in the current dir, if it exists.

But from the same terminal session I cannot simply type in "python junk.py"

No, but you put in #!/usr/bin/python, so you don't have to do that, right?

So I would think that I would need to know the path that Python searches and then just adjust my user and root path to point to the same search path that Python looks in, put all my *.py scripts there, and all would be well.

The only other thing that I can think of is what's listed below but I'm unsure of which path below is correct and if any of what I'm explaining is correct or not.

Code: Select all

>>> import sys
>>> sys.path
['', '/usr/lib/python23.zip', '/usr/lib/python2.3', '/usr/lib/python2.3/plat-linux2', '/usr/lib/python2.3/lib-tk', '/usr/lib/python2.3/lib-dynload', '/usr/lib/python2.3/site-packages', '/usr/lib/portage/pym']

sys.path is the paths that python looks for modules in when doing import. I don't thing it affects where python looks for scripts, it looks just in the current directory. But as I said, you don't have to bother with this, just put #!/usr/bin/python or #!/usr/bin/env python in your scripts and make sure your scripts are executable and in your path and you will be just fine.

[dr_strange]

p = map(getprocinfo(pid), proclist) does not work,

try treating getprocinfo as a variable:

Code: Select all

p = map(getprocinfo, proclist)

Okay, thanks. I'm not sure I understand it, but will try out in the evening.

[dr_strange]

What I don't get is how will python know that the variable getprocinfo refers to getprocinfo(). Or do you mean I should do

getprocinfo = getprocinfo()
p = map(getprocinfo, proclist)?