Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Introducing GEMS... Gentoo Enterprise Management System
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
xgregx
Apprentice
Apprentice


Joined: 17 Dec 2004
Posts: 181
Location: Cheshire, CT

PostPosted: Sun Jun 04, 2006 2:11 am    Post subject: Introducing GEMS... Gentoo Enterprise Management System Reply with quote

I'm a Gentoo Junkie, and I've been working on a project for the last few months to make it easier to manage a large group of Gentoo computers, like in a lab or lan center. The project is called GEMS, and its an acronym for Gentoo Enterprise Management System.
GEMS aims to be similar to the "Red Hat Network" provided by Red Hat Linux. It gives you a view of your environment hardware, as well as an inventory of Portage installed software, and which packages have GLSA (Gentoo Security Announcments) associated with them. It also allows you to update Portage software on many machines at once with a few simple mouse clicks. This first release has many great features, with many more to come. So if you have more than a few Gentoo boxes to maintain, check out the web site. It's GPL Licensed and totally free. Tested on x86 Gentoo, but should work on any Gentoo supported arch.
http://www.gamehound.net/gems/
Thanks!
Greg :)
_________________
Half-life/Source/Urban Terror/Unreal control panel -> www.opengamepanel.org
Linux user #347899
Gentoo has USE flags for a highly configurable system.
Microsoft has SUE flags to stifle innovation
Back to top
View user's profile Send private message
chovy
Guru
Guru


Joined: 03 Dec 2004
Posts: 453

PostPosted: Sun Jun 04, 2006 6:03 am    Post subject: Reply with quote

cool diagrams. Your web design could use some work, specifically the homepage.

i've been wanting to discuss gentoo and distributed package installation at work, but until now I haven't really seen a good way to do it.

One thing I'm curious about is if something broke, how easy would it be to flip a switch and revert back?

When I have done 'emerge -uDp world' in the past, it has not once that I can remember gone smoothly. Basically what we would need here is a baseline or a tag like in SCM that we (as a large company deploying updates to 100's of servers and client machines) that we could revert back to immediately, should something crucial come to a grinding halt.
_________________
Woof, Woof! Add "[solved]" to the title! Woof, Woof!
Back to top
View user's profile Send private message
Maedhros
Bodhisattva
Bodhisattva


Joined: 14 Apr 2004
Posts: 5511
Location: Durham, UK

PostPosted: Sun Jun 04, 2006 8:03 am    Post subject: Reply with quote

Cool. You may be interested in talking to the people behind the SCIRE project, as their aim is similar, I think. :)

http://www.gentoo.org/proj/en/scire/
_________________
No-one's more important than the earthworm.
Back to top
View user's profile Send private message
xgregx
Apprentice
Apprentice


Joined: 17 Dec 2004
Posts: 181
Location: Cheshire, CT

PostPosted: Mon Jun 05, 2006 12:14 am    Post subject: Reply with quote

Chovy,
Thanks man. I know the website is a lacking. I'm not the most creative. I'll be making it a little nicer later on. I just wanted to get it up so I could release the project.
GEMS is very simple, it only does what portage can do. I've never done an emerge world for that reason, there's too much stuff that could get broken. If you do use GEMS to install a package that ends up causing problems, you could just log into the box and do emerge unmerge packagename The next version of GEMS will have a package unmerge feature.
Keep in mind that you do need to test the packages you wish to install before you use GEMS to install them. It doesn't do anything special in terms of package conflicts (yet, maybe later). The GEMS endpoint just runs emerge commands and responds to the errors that emerge throws. If you think you want to use it for your company, feel free to ask me questions about deploying it, and I'll help you the best I can. Thanks for checking it out.

Maedros,
Thanks man, I'll check it out.


Greg :D
_________________
Half-life/Source/Urban Terror/Unreal control panel -> www.opengamepanel.org
Linux user #347899
Gentoo has USE flags for a highly configurable system.
Microsoft has SUE flags to stifle innovation
Back to top
View user's profile Send private message
zatalian
Apprentice
Apprentice


Joined: 27 Aug 2002
Posts: 179
Location: Gent, Belgium

PostPosted: Mon Jun 05, 2006 7:37 am    Post subject: Reply with quote

where did you find those nice icons to make the diagrams?
Back to top
View user's profile Send private message
chovy
Guru
Guru


Joined: 03 Dec 2004
Posts: 453

PostPosted: Mon Jun 05, 2006 3:48 pm    Post subject: Reply with quote

xgregx wrote:
Chovy,
Thanks man. I know the website is a lacking. I'm not the most creative. I'll be making it a little nicer later on. I just wanted to get it up so I could release the project.
GEMS is very simple, it only does what portage can do. I've never done an emerge world for that reason, there's too much stuff that could get broken. If you do use GEMS to install a package that ends up causing problems, you could just log into the box and do emerge unmerge packagename The next version of GEMS will have a package unmerge feature.
Keep in mind that you do need to test the packages you wish to install before you use GEMS to install them. It doesn't do anything special in terms of package conflicts (yet, maybe later). The GEMS endpoint just runs emerge commands and responds to the errors that emerge throws. If you think you want to use it for your company, feel free to ask me questions about deploying it, and I'll help you the best I can. Thanks for checking it out.

Maedros,
Thanks man, I'll check it out.


Greg :D


would emerge --unmerge pkgname revert back to the previously installed version? I would assume it would remove that package all together.
_________________
Woof, Woof! Add "[solved]" to the title! Woof, Woof!
Back to top
View user's profile Send private message
xgregx
Apprentice
Apprentice


Joined: 17 Dec 2004
Posts: 181
Location: Cheshire, CT

PostPosted: Mon Jun 05, 2006 4:36 pm    Post subject: Reply with quote

Chovy,
No, that would remove it. You would have to emerge the specific version. I don't remember the exact syntax, you can look it up. But it would be something like this:
Code:
emerge '=category/package-v1.2.3'

That would put you back to the desired version. Thanks.
Greg
_________________
Half-life/Source/Urban Terror/Unreal control panel -> www.opengamepanel.org
Linux user #347899
Gentoo has USE flags for a highly configurable system.
Microsoft has SUE flags to stifle innovation
Back to top
View user's profile Send private message
chovy
Guru
Guru


Joined: 03 Dec 2004
Posts: 453

PostPosted: Mon Jun 05, 2006 5:25 pm    Post subject: Reply with quote

yeah, that works on an individual basis, but I think a corporate solution would need the ability to rollback easily on failed deploys. just my experience speaking here, but it's one reason why I haven't even bothered suggesting it at work.

becuse of thevast difference in technilogical expertise at mid-large sized corporations, a distributed development envrionement would have to be easily deployed, ie - the big *green* button (with an *red* button for "undo").

The command line is used quite a bit, but we also support windows for the most part (as a large chunk of our developers simply do not use the CLI, instead we mount the filesystem via samba, and write Win32 scripts that do the same on the nix command line), so it's difficult, there is talk of each developer having their own Linux development box vs. working/testing on a few overloaded servers. Regardless, the problem we face is major breakages in a software update, and having it impact developer productivity.

I think portage is a perfect candidate for deploying internal software packages and upgrades to our developers.

I don't know how well a win32 script to updateportage would work, or if it's even possible. My gut feeling is there would have to be a gui for it for the non-cli users.
_________________
Woof, Woof! Add "[solved]" to the title! Woof, Woof!
Back to top
View user's profile Send private message
xgregx
Apprentice
Apprentice


Joined: 17 Dec 2004
Posts: 181
Location: Cheshire, CT

PostPosted: Mon Jun 05, 2006 8:34 pm    Post subject: Reply with quote

Zatalian,
The icons are from fasticon.com. They are free to use personally, but $10 to use on a public website or application. I forget the exact terms of the license, but I decided to just pay it. It's only $10, so I considered it like a donation. They are really sweet, I could never make something like that.


Chovy,
You just game me another "TODO" for the next version... Roll back functionality. I'll have to figure that one out.
_________________
Half-life/Source/Urban Terror/Unreal control panel -> www.opengamepanel.org
Linux user #347899
Gentoo has USE flags for a highly configurable system.
Microsoft has SUE flags to stifle innovation
Back to top
View user's profile Send private message
chovy
Guru
Guru


Joined: 03 Dec 2004
Posts: 453

PostPosted: Mon Jun 05, 2006 9:08 pm    Post subject: Reply with quote

xgregx wrote:
Zatalian,
The icons are from fasticon.com. They are free to use personally, but $10 to use on a public website or application. I forget the exact terms of the license, but I decided to just pay it. It's only $10, so I considered it like a donation. They are really sweet, I could never make something like that.


Chovy,
You just game me another "TODO" for the next version... Roll back functionality. I'll have to figure that one out.


Is there a way to dump the currently installed packages and versions list, perhaps into a well-formed xml document? That could be enough to write a simple script that could parse it and revert back if it was generated before the update took place.
_________________
Woof, Woof! Add "[solved]" to the title! Woof, Woof!
Back to top
View user's profile Send private message
SirYes
Apprentice
Apprentice


Joined: 15 Jan 2006
Posts: 282
Location: Lodz, Poland

PostPosted: Mon Jun 05, 2006 10:30 pm    Post subject: Reply with quote

chovy wrote:
Is there a way to dump the currently installed packages and versions list, perhaps into a well-formed xml document?

Provided that you have gentoolkit installed first, it's:
Code:
equery list

The output is not an XML file, but rather one package per line (aka atoms) including category, name of the package and its exact version. Would it suffice?
_________________
My blog: In search for ultimate programming language
Back to top
View user's profile Send private message
chovy
Guru
Guru


Joined: 03 Dec 2004
Posts: 453

PostPosted: Mon Jun 05, 2006 10:37 pm    Post subject: Reply with quote

SirYes wrote:
chovy wrote:
Is there a way to dump the currently installed packages and versions list, perhaps into a well-formed xml document?

Provided that you have gentoolkit installed first, it's:
Code:
equery list

The output is not an XML file, but rather one package per line (aka atoms) including category, name of the package and its exact version. Would it suffice?


Yeah, that would work, xml only makes sense if there's lots of data to present.

Code:
[I--] [  ] dev-perl/Digest-SHA1-2.11 (0)
[I--] [ ~] dev-perl/Finance-Quote-1.11 (0)
[I--] [  ] dev-perl/HTML-Element-Extended-1.13 (0)
[I--] [  ] dev-perl/HTML-Parser-3.48 (0)
[I--] [  ] dev-perl/HTML-TableExtract-2.06 (0)
[I--] [  ] dev-perl/HTML-Tagset-3.10 (0)
[I--] [  ] dev-perl/HTML-Tree-3.19.01 (0)
[I--] [  ] dev-perl/IO-Socket-SSL-0.97 (0)
[I--] [  ] dev-perl/IO-String-1.08 (0)
[I--] [  ] dev-perl/IO-Zlib-1.04 (0)
[I--] [  ] dev-perl/Locale-gettext-1.05 (0)
[I--] [ ~] dev-perl/Net-DNS-0.55 (0)
[I--] [  ] dev-perl/Net-Daemon-0.38 (0)
[I--] [  ] dev-perl/Net-IP-1.24 (0)



what do the [I--] [~] and (0) mean?
_________________
Woof, Woof! Add "[solved]" to the title! Woof, Woof!
Back to top
View user's profile Send private message
xgregx
Apprentice
Apprentice


Joined: 17 Dec 2004
Posts: 181
Location: Cheshire, CT

PostPosted: Tue Jun 06, 2006 1:28 am    Post subject: Reply with quote

I use eix. It allows you to format the output of equery and emerge. I wrote it's output to file, and then parse that for doing inventory collection in GEMS.
Greg
_________________
Half-life/Source/Urban Terror/Unreal control panel -> www.opengamepanel.org
Linux user #347899
Gentoo has USE flags for a highly configurable system.
Microsoft has SUE flags to stifle innovation
Back to top
View user's profile Send private message
xgregx
Apprentice
Apprentice


Joined: 17 Dec 2004
Posts: 181
Location: Cheshire, CT

PostPosted: Wed Jun 14, 2006 7:27 pm    Post subject: Reply with quote

I put a GEMS demo up, check out the webpage.
www.gamehound.net/gems/
Click on the demo link at the top. The login is user:guest password:guest. It's prefilled.
Thanks!
Greg :)
_________________
Half-life/Source/Urban Terror/Unreal control panel -> www.opengamepanel.org
Linux user #347899
Gentoo has USE flags for a highly configurable system.
Microsoft has SUE flags to stifle innovation


Last edited by xgregx on Mon Jun 19, 2006 3:58 pm; edited 1 time in total
Back to top
View user's profile Send private message
FishB8
l33t
l33t


Joined: 17 Mar 2003
Posts: 820

PostPosted: Mon Jun 19, 2006 4:33 am    Post subject: Reply with quote

Here's an idea for managing networks that I've been thinking of that might be of interest:

Image Sets and Client Groups.

Image-Set: An image of all the files installed on a system. Sort of like how when building packages the install is dumped into an image folder and then copied over onto the actual system. An Image Set would be an image of an entire installation. You could delete, duplicate, and create new Image Sets. The sets are stored on a central image server.

Client-Groups: A way of dividing client computers on a LAN into groups for purposes of package management.

Here's an example of how this would be useful. (Examples tend to work better than trying to explain it):

Say you have a network that, like most networks, has a variety of different types of computers on it. You divide them up into client-groups depending upon what compiler settings, USE flags you want used, or what that particular set of computers is used for, etc.

Lets say you have a dozen computers that have athlon processors and are used for general office work and so you have assigned them to a client-group named AMD-Office-Grp. The contents of the computers in this client-group are generally synced to the image-set named AMD-Office-Img on the image server.

You want to update a package on these computers only. So your GEMS starts to build that package using the settings for that client-group. All computers on your network that are in client-groups that have distcc enabled in their config, work together to build the package. (Goes really fast on a big network!!) You don't want to install this pacakage right away though because you want to test it first. So before building it, you duplicated the AMD-Office-Img image-set and named it AMD-Office-TEST. Once the package is built you have it installed to that AMD-Office-TEST image-set on the image server.

To test it you take one the the computers from AMD-Office-Grp and change it's client-group to a new one called AMD-TEST-DUMMY. You then sync that client-group to the AMD-Office-TEST image-set. You can then test on that computer to make sure it works. Let's assume it's all working ok.

Then you move the test computer back to the AMD-Office-Grp client-group. You also delete the AMD-Office-Img image set. It's old now and you don't need it. Now you can rename the AMD-Office-TEST image-set to AMD-Office-Img. This is the new default image-set for the AMD-Office-Grp client-group that has the updated package on it. You then sync the AMD-Office-Grp to the new AMD-Office-Img.



Bleh, :P that was a little over-stated, but I think I get my idea across of an automated system to manage packages on groups of computers while having distributed compiling.

Does this make sense? Would this be feasible? Are there other management packages that already do this?
_________________
"...as we enjoy great advantages from the inventions of others, we should be glad of an opportunity to serve others by any invention of ours, and this we should do freely and generously." -Benjamin Franklin
Back to top
View user's profile Send private message
bianco
n00b
n00b


Joined: 21 Jun 2005
Posts: 54

PostPosted: Sat Feb 27, 2010 3:33 pm    Post subject: Reply with quote

Is this project still alive? :?:
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun Feb 28, 2010 1:29 am    Post subject: Reply with quote

bianco wrote:
Is this project still alive? :?:


you necromancer ! :wink:


rolling back packages could be implemented by previously running:

quickpkg --include-config=y on the client-side

on the server-side (assuming you using a similar environment) you could prepare packages by emerge -B
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
augury
l33t
l33t


Joined: 22 May 2004
Posts: 722
Location: philadelphia

PostPosted: Sun Feb 28, 2010 11:57 pm    Post subject: Reply with quote

I used to maintain boxes with packages. The problem was portage was slow. And if portage sync changes from system to system, then packages get built anyways. Clients are too slow to compile.

I used ntfs for portage and the package across all systems. If the portage system was maintained eternally (/var/db/pkg) for all systems and a fast package emerging script was uploaded to each system it would stabilize and facilitate upgrades. And not require either a dumb chron to run "emerge --world" but also not require logging into every system.

Sloppy. Slow. Everything breaks.

There is still an element of unpredictability. For this reason I would compile a stable installation and then not touch it ever. That goes for other distros as well. :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum