Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Goodbye my Gentoo servers, I hardly knew you
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
kashani
Advocate
Advocate


Joined: 02 Sep 2002
Posts: 2032
Location: San Francisco

PostPosted: Thu Oct 05, 2006 2:38 am    Post subject: Goodbye my Gentoo servers, I hardly knew you Reply with quote

Intro:
It all started in Dec '05. I arrived at my new job at a new startup as their jack of all trades IT guy for http://jumpcut.com/ They were running on three dev boxes with CentOS and having a number of samba issues, cvs, and apache issues. I'd been running Gentoo for about three and half years at this point and felt it could do everything we wanted and make my life easier. As the only sys admin, network engineer, desktop support, etc guy I took a vote and Gentoo won.


I: Fix the office stuff
Consolidated dev into one CentOS box. Installed Gentoo onto sys01 which was now free to be CVS with cvsd, which wasn't in the Centos package, Bugzilla, Cacti, net-snmp, RT, Nagios, Samba 3.0.22c or whatever fixed what Centos' Samba 3.0.14 couldn't do with the latests XP patches. Additionally followed the nice how-to the Gentoo Wiki to integrate Samba into the existing active directory.

Number of packages I didn't have to build RPMs for: 150+ mostly because of RT and Bugzilla deps.
Total time: two weeks for building, configuring packages, testing, and releasing to all employees


II: Moving more to Gentoo
Dev had standardized on PHP and then used a home built version of PHP 5.1.1 and Mysql 5.0.11 since neither existed in our CentOS. Additionally their original consultant did not setup software RAID in any of the dev machines. It was a bit of a mess, but we were early in the dev cycle so weird dependencies on custom builds had not crept in.

Built dev01 with LVM and software RAID, Apache 2.0, PHP 5.1.2 w/hardened patch and limited subset of functions to reduce binary size, current cvs ffmpeg and assorted libs, etc. Moving devs over and set each up with their own Apache instance. No one noticed a difference other than they really liked the default Gentoo colors in their terminal.

Number of packages I didn't have to build RPMs for: 50+ mostly due to ffmpeg and PHP required libs.
Total time: one week to build, configure, test, and release.


III: Dev and staging fully on Gentoo
dev02 and dev03 rebuilt with LVM and software RAID. dev02 becomes dev Mysql 5.0 and dev03 becomes dev NFS storage.

Number of packages I didn't have to build RPMs for: none
Total time: four hours as I used a stage4 I made from dev01


IV : Setting up production on Gentoo
We used what we called minimal stage4s generated from to dev to build the eventual twenty machines that ran the site. Each class of machine eventually got it's own stage4 like db-stage4, www-stage4, which allowed us to install a single machine in one hour or roughly six machines in one and a half hours which is the most we ever received at one time.
We rebuilt the database servers as x86_64 eight months later as well as the dev db server, but otherwise updated machines as packages became available. All new ebuilds were tested on dev, then staging, and then made it into production. Generally updates made it into production within three weeks of the initial package update hitting portage. Some hit portage quicker if they were security issues.

Total number of additional packages I never had to build: 30+
web stats, Postfix, postgrey are some of the highlights.
Total number of Gentoo caused problems in production: 0
Uptime: 99.99%, stupid power breaker in our colo.


V: How are we doing?
I've worked in a few large shops of 100+ machines and some small shops. As a one man shop I do far less work and update faster with Gentoo than I ever did with Redhat, Fedora, or Solaris in other < 50 machine shops. Remember as a startup being able to get new versions, especially versions that fix issues quickly and easily is key. Additionally using Gentoo ebuilds was vastly superior to any package I could build on my own. I don't have the time to QA in house packages and even if I did Gentoo builds are used by thousands of people rather than just one.
I don't come out so well against a large shop. However without the infrastructure or the budget to pay for large shop tools we did quite well. Additionally I am most comfortable in Gentoo and there is something to be said for using the tools you're most familiar with.
Where Gentoo has really shines is in projects that fail. No really. We've all done that "hey lets try upgrading to Apache 2.2 and see how well it works." In Gentoo you change a few lines, emerge apache, run some tests, realize it's not quite there, change a few lines, emerge apache again, and you're back to where you started. Total time about two hours.
Or even projects that go somewhere. "Hey I need X packages for testing." Gentoo installs, some minor tweaks, and hand off to the dev. When we go to production I know I can get the same package because I let Gentoo do the work rather than half ass a build because I didn't have time for non production issues when the project had no priority. Naturally there some changes in config in production, but we can go to production faster without having to repackage, re QA, and then release.


VI: Goodbye Gentoo servers
As of last week we are now owned by Yahoo! which uses Redhat. We're starting the process of moving to their infrastructure and it looks to be straightforward. I'm sad to see our Gentoo servers go, but trying to convince Yahoo! to run a distro they have no experience with just for us isn't going to happen.
I did however get geek points from a number of Yahoo engineers who thought it was very cool that we used Gentoo to build our system. Of course I never said things like, "Gentoo rulez" when I describe our infrastructure and I did say "We were relying on a number of bleeding edge packages so we picked a distro that catered to that. There are some quirks, but if you're prepared to deal with that it can be great."

kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape.
Back to top
View user's profile Send private message
andrewd18
Guru
Guru


Joined: 11 Apr 2004
Posts: 364
Location: Wisconsin, USA

PostPosted: Thu Oct 05, 2006 3:39 am    Post subject: Reply with quote

I was thinkin' this was gonna be another "Wah, I hate Gentoo" thread. Glad to see you enjoyed your experience. :D
_________________
Keep Your Toolchain Stable! - emwrap.sh

There's no place like ::1
Back to top
View user's profile Send private message
padoor
Advocate
Advocate


Joined: 30 Dec 2005
Posts: 4185
Location: india

PostPosted: Thu Oct 05, 2006 5:10 am    Post subject: Reply with quote

it is a pity yahoo does not want part of gentoo.
_________________
reach out a little bit more to catch it (DON'T BELIEVE the advocate part under my user name)
Back to top
View user's profile Send private message
anello
Guru
Guru


Joined: 17 Jul 2005
Posts: 557
Location: EU -> DE -> Stuttgart

PostPosted: Thu Oct 05, 2006 5:32 am    Post subject: Reply with quote

Thank you kashani!

That's how a "goodbye/I'm back" post gotta look like.

Finally something that's worth the time reading.

It's sad to hear that yahoo is insisting on RedHat. I would not expect a conservative attitude from such a big company. On the other side, it makes the work of the IT Dep. much easier.
_________________
Antonino Catinello | http://catinello.eu
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Oct 05, 2006 6:10 am    Post subject: Reply with quote

The problem with large companies is that most decisions are made on the top and is unchangable. When you move to a structure like that it hurts performance on the lower levels, because the leaves hate to be dragged in the wind. The larger the structure, the more chance for resistance, expecially when leaves are transplanted.

Gentoo Servers!!!! I love gentoo; The server I set up at my high school (im now in college) is nearing its one year mark, as in uptime and age :D

Red hat is nice when you have the tech support, let them fix it when you dont know something. Of course this leads to lazy admins, but then again, its better then lazy windows admins.
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
mark_alec
Bodhisattva
Bodhisattva


Joined: 11 Sep 2004
Posts: 6066
Location: Melbourne, Australia

PostPosted: Thu Oct 05, 2006 8:00 am    Post subject: Reply with quote

:cry: Sad to see that your Gentoo boxen have been migrated. Glad you have had a good experience using Gentoo all these years.
_________________
www.gentoo.org.au || #gentoo-au
Back to top
View user's profile Send private message
uxbod
Guru
Guru


Joined: 30 Dec 2002
Posts: 474
Location: UK

PostPosted: Thu Oct 05, 2006 8:12 am    Post subject: Reply with quote

I hate to say but personally I feel that Gentoo will not easily make it into the Blue Chip arena until such time one of the large vendors ie. HP, Dell provide commercial support for it. RedHat and Suse have the commercial ties, and upper management look far more favourable on these distros because of that. Commercial acceptance and support does not equal a better product though ;)
Back to top
View user's profile Send private message
BastianBalthazarBux
Retired Dev
Retired Dev


Joined: 10 Dec 2004
Posts: 78

PostPosted: Thu Oct 05, 2006 10:38 am    Post subject: Reply with quote

Happy to see you had good time with gentoo.
I've also apreciated your frequent answers on the server mailing list, hope that you decide to keep at least some gentoo box and still lurk on our mail list.
Having prepared ppl like you help us is a great bonus to the community.

krgds, Francesco R.
Back to top
View user's profile Send private message
kierans
n00b
n00b


Joined: 07 Apr 2005
Posts: 51

PostPosted: Thu Oct 05, 2006 11:04 am    Post subject: Reply with quote

I'm in my final year of university (down in the land of Australia), and I've been using Gentoo as my distro for a few years. I even got a guy that I did group projects with into it. In our final year this year, myself, my Gentoo friend and some other guys had to do a full year project for a client external to university. Turns out our project wasn't going to be the standard software development stuff that they usually throw at you, to see if you've being paying attention the last few years. The project was more a research project, which means we needed a server box that was highly configurable and easy to rollback in case something went wrong. So my friend and I turned out to be the main code monkeys. Where did we turn - Gentoo of course

Turns out my old man was upgrading, so I got his old machine, installed Gentoo within a couple of days, whacked in on the back of the uni network, and we were off. Subversion, apache2, and all the Java tools (being the language of the project) were all installed quickly and efficiently. When we wanted to try something, we let portage do the work, had a play and if it didn't work, a couple of hours to undo the damage. If it did work, it stayed. In terms of software configuration the biggest hurdle was documentation of some of the software, but that's not Gentoo's fault.

All in all the group agreed that if it wasn't for Gentoo, we would have never gotten the project done in the allocated time span. Thanks to portage, we didn't have to spend huge amounts of time dealing with dependencies, or getting the damn thing compiled. No we sat back while Gentoo did it's thing, and got the important stuff done.
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Oct 05, 2006 5:23 pm    Post subject: Reply with quote

Doesnt portage have an option to grab precompiled packages? It would be interesting if someone were to start a commercialized branch of gentoo that had the option to use bin packages for most of the packages (or atleast a system to drop request for precompiled).

I dont mind the other distros, but I love gentoo, it doesnt complain, it works great, and I think businesses would love it IF they were to know about it.

I only figured out about gentoo because I heard about it from the 3rd person from one of my friends. Maybe we should start a gentoo awarness initiative?
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
amne
Bodhisattva
Bodhisattva


Joined: 17 Nov 2002
Posts: 6378
Location: Graz / EU

PostPosted: Thu Oct 05, 2006 5:33 pm    Post subject: Reply with quote

Nice to hear Gentoo doesn't suck. :D
_________________
Dinosaur week! (Ok, this thread is so last week)
Back to top
View user's profile Send private message
kashani
Advocate
Advocate


Joined: 02 Sep 2002
Posts: 2032
Location: San Francisco

PostPosted: Thu Oct 05, 2006 6:10 pm    Post subject: Reply with quote

I figured it was about time Gentoo Chat had a good sucess story rather than some of those *other* posts.

In regards to Yahoo I still remain the Jumpcut Tech Guy or whatever my title ends up being, but Servers, Storage, Networking, etc are all individual groups outside of us at Y! They install and manage the servers that us, Flickr, de.lic.io.us, Yahoo Mail, etc all use. Trying to add Gentoo to that infrastructure would be impossible since they're already managing thousands of RedHat servers. However a few Y! engineers expressed intrest in setting up a few boxes to play with.

Of course some of the local boxes in the office won't be changing like sys01 and my workstation. I'll still be on the mailing lists and hanging around in the forums. I also keep meaning to sign up for x86 and amd64 arch testing, but have never had the time as you might imagine. My personal webserver will also stay Gentoo so I'm not going anywhere, just biding me time until I can Gentoo-ize another shop. :)

kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape.
Back to top
View user's profile Send private message
Zepp
Veteran
Veteran


Joined: 15 Mar 2004
Posts: 1246
Location: Ontario, Canada

PostPosted: Thu Oct 05, 2006 7:14 pm    Post subject: Reply with quote

Finally a nice goodbye (sorta) story :P. Sorry to hear, but glad it worked out so well while it lasted.

and well at least they didn't make ya switch to windows ;)
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sun Oct 08, 2006 1:16 pm    Post subject: Reply with quote

Yeah good to hear a success story, even if that company's changed it's infrastructure.
Back to top
View user's profile Send private message
dundas
Guru
Guru


Joined: 16 Dec 2004
Posts: 317
Location: China, Earth

PostPosted: Thu Oct 12, 2006 7:13 am    Post subject: Reply with quote

The hardest thing for me is also to convince other engineers to realize the beauty of Gentoo, to overcome the compiling fear and linux fear in general.

But damn it, I'm trying and keeping myself updated wiz the latest Gentoo stuff.
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer
Back to top
View user's profile Send private message
mudrii
l33t
l33t


Joined: 26 Jun 2003
Posts: 789
Location: Singapore

PostPosted: Thu Oct 12, 2006 2:40 pm    Post subject: Reply with quote

Yup really nice story
_________________
www.gentoo.ro
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Thu Oct 12, 2006 8:45 pm    Post subject: Reply with quote

most people dont even know what compiling is... people fear the unknown...
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
renrutal
Tux's lil' helper
Tux's lil' helper


Joined: 26 Mar 2005
Posts: 135
Location: Brazil

PostPosted: Thu Oct 12, 2006 9:50 pm    Post subject: Reply with quote

kashani, sir, you really should set a page, a blog, or a very complete guide about how to manage 100+ Gentoo machines.

Many people here, including me, know Gentoo is awesome, but they don't have a clue about how to do that miraculous feat.
Back to top
View user's profile Send private message
deanpence
Apprentice
Apprentice


Joined: 08 Nov 2004
Posts: 158
Location: Earth

PostPosted: Thu Oct 12, 2006 10:16 pm    Post subject: Reply with quote

renrutal wrote:
kashani, sir, you really should set a page, a blog, or a very complete guide about how to manage 100+ Gentoo machines.


At my last job, I actually did manage close to 1,000 servers all running Gentoo. It was no picnic, but I've since realized that there really are few tools out there in any distro to manage a huge enterprise like that. I'm at a much smaller shop now that runs CentOS, and the problems of maintaining the few dozen machines we have are really not very different from maintaining 1,000.

With Gentoo, you really have to treat it as a meta-distribution and, at least to some extent, create your own distro (or at least image of binary packages) to roll out; i.e., unless you want to devote all your machines to being a distcc cluster, compiling every app for every machine just isn't practical. The really big problem, one which I still haven't completely solved, is that of configuration file management. Running etc-update or dispatch-conf on 1,000 machines is not only impractical; it's impossible. Software solutions to synchronize config files across a network become very attractive.

What really keeps me with Gentoo--and the reason I'm going to push to migrate to Gentoo at my new job--is that it's elegant. Even if Gentoo was not a meta-distribution with installation from source, I would still be in love with its rc initialization system (with dependencies!!!) and elegant management scripts. CentOS just can't compare in this regard.
Back to top
View user's profile Send private message
vibrokatana
Guru
Guru


Joined: 09 Feb 2006
Posts: 328
Location: o0o0oo

PostPosted: Fri Oct 13, 2006 12:03 am    Post subject: Reply with quote

Whenever someone asks me how to make an init script I always ask which distro they are running, if its gentoo ill gladly help them, but red hat needs to despirately do something with their startup, its a royal pain without a gui
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."
Back to top
View user's profile Send private message
karsten_s
n00b
n00b


Joined: 19 Jan 2006
Posts: 33
Location: Berlin, Germany

PostPosted: Sat Oct 14, 2006 12:01 pm    Post subject: Reply with quote

y'know, it's been just great reading this. Sometimes I was close to tears from sensing the passion you guys have for gentoo. And do you know what's the greatest thing about that is? I can relate! I've got only one gentoo box (started Jan 02, 06) and I love it! It may seem somewhat geekish to love a (meta-) distribution the way we love gentoo, but, hey, we do love it. Both portage and the community have become the main reasons for me turning into a linux missionary more than ever.

I'm losing my point. What did I want to say? Basically thanks, kashani, for sharing, and also thanks whoever got this thread mentioned in the gwn.
_________________
Quote:
This time it was right, it would work, and no one would have to get nailed to anything.

Sadly, however, before she could get to a phone to tell anyone about it, a terribly stupid catastrophe occurred, and the idea was lost forever.
Back to top
View user's profile Send private message
nixnut
Bodhisattva
Bodhisattva


Joined: 09 Apr 2004
Posts: 10974
Location: the dutch mountains

PostPosted: Sat Oct 14, 2006 4:23 pm    Post subject: Reply with quote

deanpence wrote:
The really big problem, one which I still haven't completely solved, is that of configuration file management. Running etc-update or dispatch-conf on 1,000 machines is not only impractical; it's impossible. Software solutions to synchronize config files across a network become very attractive.

Have you considered cfengine for those tasks?
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered

talk is cheap. supply exceeds demand
Back to top
View user's profile Send private message
ErZo
n00b
n00b


Joined: 23 Jul 2005
Posts: 6
Location: Sweden.

PostPosted: Wed Oct 25, 2006 8:26 am    Post subject: Reply with quote

Thanks for sharing this! Awesome, really really awesome. I'm glad you're in our community :-)

Neat :) What more can I say?
Back to top
View user's profile Send private message
symbiat
n00b
n00b


Joined: 20 Aug 2003
Posts: 36
Location: New York

PostPosted: Tue Nov 07, 2006 3:38 pm    Post subject: Re: Goodbye my Gentoo servers, I hardly knew you Reply with quote

kashani wrote:
VI: Goodbye Gentoo servers
As of last week we are now owned by Yahoo! which uses Redhat. We're starting the process of moving to their infrastructure and it looks to be straightforward. I'm sad to see our Gentoo servers go, but trying to convince Yahoo! to run a distro they have no experience with just for us isn't going to happen.
I did however get geek points from a number of Yahoo engineers who thought it was very cool that we used Gentoo to build our system. Of course I never said things like, "Gentoo rulez" when I describe our infrastructure and I did say "We were relying on a number of bleeding edge packages so we picked a distro that catered to that. There are some quirks, but if you're prepared to deal with that it can be great."

I had a similar experience working for an ISP where I eventually standardized on Gentoo for almost all Linux boxes - its a lot less hassle as far as maintenance goes than RedHat. I just started a new developer job about 3 weeks ago (so Im no longer doing sys. admin. work). This is after having worked for the ISP for 4.5 years as sys. admin. Im not assuming my replacement will want to continue working with Gentoo (though, our network guy, who's a FreeBSD nut, loves our Gentoo web servers).

BTW, instead of saying "Gentoo Rulez" you could have pointed out a few high profile projects that *do* use Gentoo (e.g. Asterisk).
Back to top
View user's profile Send private message
symbiat
n00b
n00b


Joined: 20 Aug 2003
Posts: 36
Location: New York

PostPosted: Tue Nov 07, 2006 3:55 pm    Post subject: Reply with quote

deanpence wrote:
The really big problem, one which I still haven't completely solved, is that of configuration file management. Running etc-update or dispatch-conf on 1,000 machines is not only impractical; it's impossible. Software solutions to synchronize config files across a network become very attractive.

Have you looked at cfengine or (even better maybe) puppet? Also check out open-management.com for some great open source management tools.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum