[HOWTO] Speeding up portage with cdb -- UPDATE

[tobidope]

[HOWTO] Speeding up portage with cdb -- UPDATE

Changelog

• 19.12.2004 -- New portage_db_cdb.py module. It's a workaround to accelerate the building of the database after an emerge sync. Hopefully portage will never use jython, I need the destructor. On my ancient computer emerge metadata now last 10 minutes instead of 50!
  
• 19.12.2004 -- Second try. Note to myself "Also making tiny changes can break your system"
  
• 04.02.2005 -- Deleted the advice to use emerge --rege
  
• 16.02.2005 -- Added some lines about copyright
  

When it comes to speed the database backend of portage is rather slow because it's implemented as a raw abstraction in the filesystem. It is possible to replace this with a mysql backend which is not ideal, if you don't want to start a full-blown RDBMS. My solution is rather tiny, if you compare the code and the packages you need for that:

1. You'll need the python interface python-cdb for DJB's cdb
   

   Code:

   emerge python-cdb

   
   
2. Create /usr/lib/portage/pym/portage_db_cdb.py and put this in it:
   

   Code:

   # Copyright 2004, 2005 Tobias Bell <tobias.bell@web.de> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA import portage_db_template import os import os.path import cPickle import cdb class _data(object):     def __init__(self, path, category, uid, gid):         self.path = path         self.category = category         self.uid = uid         self.gid = gid         self.addList = {}         self.delList = []         self.modified = False         self.cdbName = os.path.normpath(os.path.join(             self.path, self.category) + ".cdb")         self.cdbObject = None     def __del__(self):         if self.modified:             self.realSync()         self.closeCDB()     def realSync(self):         if self.modified:             self.modified = False             newDB = cdb.cdbmake(self.cdbName, self.cdbName + ".tmp")                         for key, value in iter(self.cdbObject.each, None):                 if key in self.delList:                     if key in self.addList:                         newDB.add(key, cPickle.dumps(self.addList[key], cPickle.HIGHEST_PROTOCOL))                         del self.addList[key]                 elif key in self.addList:                                        newDB.add(key, cPickle.dumps(self.addList[key], cPickle.HIGHEST_PROTOCOL))                     del self.addList[key]                 else:                     newDB.add(key, value)                             self.closeCDB()             for key, value in self.addList.iteritems():                 newDB.add(key, cPickle.dumps(value, cPickle.HIGHEST_PROTOCOL))                         newDB.finish()             del newDB                         self.addList = {}             self.delList = []             self.openCDB()     def openCDB(self):         prevmask = os.umask(0)                 if not os.path.exists(self.path):             os.makedirs(self.path, 02775)             os.chown(self.path, self.uid, self.gid)                     if not os.path.isfile(self.cdbName):             maker = cdb.cdbmake(self.cdbName, self.cdbName + ".tmp")             maker.finish()             del maker             os.chown(self.cdbName, self.uid, self.gid)             os.chmod(self.cdbName, 0664)         os.umask(prevmask)                     self.cdbObject = cdb.init(self.cdbName)     def closeCDB(self):         if self.cdbObject:             self.cdbObject = None class _dummyData:     cdbName = ""     def realSync():         pass     realSync = staticmethod(realSync) _cacheSize = 4 _cache = [_dummyData()] * _cacheSize class database(portage_db_template.database):        def module_init(self):         self.data = _data(self.path, self.category, self.uid, self.gid)         for other in _cache:             if other.cdbName == self.data.cdbName:                 self.data = other                 break         else:             self.data.openCDB()             _cache.insert(0, self.data)                        _cache.pop().realSync()                 def has_key(self, key):         self.check_key(key)         retVal = 0         if self.data.cdbObject.get(key) is not None:             retVal = 1         if self.data.modified:             if key in self.data.delList:                 retVal = 0             if key in self.data.addList:                 retVal = 1                     return retVal     def keys(self):         myKeys = self.data.cdbObject.keys()         if self.data.modified:             for k in self.data.delList:                 myKeys.remove(k)             for k in self.data.addList.iterkeys():                 if k not in myKeys:                     myKeys.append(k)                             return myKeys     def get_values(self, key):         values = None                 if self.has_key(key):             if key in self.data.addList:                 values = self.data.addList[key]             else:                 values = cPickle.loads(self.data.cdbObject.get(key))         return values         def set_values(self, key, val):         self.check_key(key)         self.data.modified = True         self.data.addList[key] = val     def del_key(self, key):         retVal = 0                 if self.has_key(key):             self.data.modified = True             retVal = 1             if key in self.data.addList:                 del self.data.addList[key]             else:                 self.data.delList.append(key)         return retVal                         def sync(self):         pass         def close(self):         pass if __name__ == "__main__":     import portage     uid = os.getuid()     gid = os.getgid()     portage_db_template.test_database(database,"/tmp", "sys-apps", portage.auxdbkeys, uid, gid)

   
   
3. Create /etc/portage/modules and fill in
   

   Code:

   portdbapi.auxdbmodule = portage_db_cdb.database eclass_cache.dbmodule = portage_db_cdb.database

   
   
4. Now you'll have to regenerate the portage cache with
   

   Code:

   emerge metadata

Now try some searches with emerge especially with --searchdesc

[steveb]

nice

[rojaro]

Oh yes ... this is really nice, this should be included in portage!
_________________
A mathematician is a machine for turning coffee into theorems. ~ Alfred Renyi (*1921 - †1970)

[TheCoop]

ooooooh this is good. prod the devs to put it in
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock

[PrakashP]

I like this. Painless and good results.

[tobidope]

You can also benchmark it against the portage_db_anydbm module by putting

[AlterEgo]

Cool stuff! Works great for me. Thanks!

[John-Boy]

Nice

[PrakashP]

BTW, where are the cdb files kept? In /usr/portage/metadata?

[tobidope]

[GentooBox]

Can i see a benchmark before switching to cdb... please
_________________
Encrypt, lock up everything and duct tape the rest

[matroskin]

normal
% time emerge -S mozilla > /dev/null
real 2m39.952s
user 0m21.770s
sys 0m4.580s

cdb
% time emerge -S mozilla > /dev/null
real 0m23.169s
user 0m19.770s
sys 0m2.190s

[UberLord]

Is there any speed increase updating the cache after syncing?

[PrakashP]

@tobidope

Is the other stuff in /var/cache/edb/dep/usr/portage actually needed after using cdb? I see files with .cpickle and various dirs with files which occupy a hunk of space...

[tobidope]

[tobidope]

[PrakashP]

Hmm, it seems after an emerge sync the got removed by itself.

[s4kk3]

Very nice. It took almost 3 minutes more to complete emerge --searchdesc python without cdb. This kind of speedups I love
_________________
My own filemanager project

[tobidope]

@PrakashKC

And? Has the update of the portage cache been faster? I hope so, but my PC
I'm working with at the moment is rather slow. Both syncs (portage_db_flat and portage_db_cdb) took a long amount of time and I forgot to put a time in front of the emerge sync.

[PrakashP]

I have not timed it actually. It seemed slower with cdb, but it could be just a subjective impression because of the % rising slowly whereas with default portage you don't see the %...

But searching is now really fast - now competing with portage-c, though not quite as fast.

[UberLord]

Default portage

[HackingM2]

[PrakashP]

@UberLord

I think there is something wrong with your sytem:

time emerge -S mozilla
real 0m21.519s
user 0m15.361s
sys 0m1.082s

with cdb.

AthlonXP 2.2 GHz, 1 GB RAM, / on RAID0

[tobidope]

I think, the problem with the really slow update after

[Leffe]

Hm, I wonder which is faster SQLite or CDB