gentoo for production , experiences?

Message

dambacher · Post by **dambacher** » Sun Sep 28, 2003 10:01 pm

Hi gentoo'ers

Now I've some experience with gentoo, I'd like to use gento on production systems (server AND desktop).

Who has experience and some advice for me?

How do I handle updataes carefully ? Is a privately managed gentoo mirror the solution or is there a cleaner way?

Thanks
Ulf

Sanity1971 · Post by **Sanity1971** » Sun Sep 28, 2003 10:30 pm

Well I have one heavy webserver running on it now. A dual P3. To give you an idea of how heavy. Even trimmed access_log fills up 2gb in a few hours. Sure nothing google would lose sleep over but still a decent amount of traffic.

Anyway. I maintain several other servers based on different distro's and even OSes. Gentoo is by far the easier to maintain PROVIDED you do a test install on a eh test machine first. Recent mysql problem for instance was more then just upgrading mysql, you also had to do everything that used the mysql libraries. Obvious perhaps but nice to not have to figure this out on a production machine. On the other hand the two openssh patches were a breeze. emerge -u openssh is so much easier then compiling by hand.

So Ebuilds are not always perfect. Test then deploy.

RPM from redhat are a little bit safer but then that is what you pay for. For the fact that someone is doing the testing for you.

So, yes gentoo can work on a production machine. Its advantages are that you can keep the machine really really thight, no costs (important if you are on shoestring budget) , close to the edge (important if your costumers want the latest features of php)

I am going to switch completly to gentoo in the future. I always found myself installing the critical packages from source anyway. At least now I don't have to worry about breaking RPM.

ON THE OTHER HAND

Gentoo comes with no guarantee. It is by fans for fans more or less and lets face it there are fans who you wouldn't allow to be in charge of the coffee machine let alone your source of income. So buyer beware. If you have "rescued" mangled boxes before then portage can't do anything you can't fix. You can fubar a machine with any distro. Sometimes all it takes is a cd labebeld Microsoft

So far the server. For the desktop well. I currently am testing 2.6 on a desktop so who cares about breaking anything. I am testing all the different window managers. Haven't found the magic one yet btw, and so far nothing broke apart from some missing fonts in openbox.
Still I find it hard to classify a desktop as a production machine. Once setup after all you don't touch it anymore. Sure when you upgrade you run the risk of breaking something. But this is true anytime. One of the things I hated about rpm's was in fact that they always broke everything. Even simple things like my menus

So far gentoo seems to be a lot nicer. Then again I upgrade in my own time. So any time spend fixing things is not lost production time. Your situation might be different.

So the only real adive I got for you is A test B test and C see any time spend testing and fixing the tests as learning time.

dambacher · Post by **dambacher** » Sun Sep 28, 2003 10:46 pm

Hi Sanity1971

Thanks for your reply. Regarding your Advices, I just finished A and now I'm continuing with B

I already have 3 testing maschines online with some linux-unexperienced users trying out the desktop and it seems to work. My server testbed looks very good, too. It can do everything my suse server does, but faster and with smaller footprint. And it more friendly upgradable.

Do you sync your servers directly to gentoo.org or do you maintain a local copy of the tree?
How do you do your uptdate? first try it on your testbed and then update the server to the same versions?

Greetings
Ulf

abt72 · Post by **abt72** » Tue Sep 30, 2003 9:22 am

hi,

we are using gentoo for our oracle-application server, for our gateways, the fileserver and our http-server.
other stuff ( = databases) is still on solaris.

webserver, fileserver and gateways -> no problem
oracle ias9: installation of ias9 didn't work, so we installed them on SuSE 7.2 (recommended besides red hat by oracle) and copied them on our gentoo-systems.

Everything is fine, no probs when emerging or updating something.

greetings,
abt

pple · Post by **pple** » Tue Sep 30, 2003 10:56 pm

These are only school servers, 200k users daily, but gentoo has worked REMARKABLY well. Because it's only user service, I assume.
I have no idea on db-services or bigger userbase, lord forgive for http stuff, but I have to admit it has been a joyride after using winoxer boxes AND redhat.
Goddamn, it's damned how very hard a Linux (The Greatest Distributor For That Matter) distributor, can make all the configing. I'm all for choosing Redhat as your choice of distributor, but it's not for smaller companies.
And fuck my comment for anything over 10k users, you can have configs as you want. I'm just saying. Redhat is not worth it on lower userbase. And windows is never worth it. And this was from someone that had to do a lot of that MINOR handling of users.

entropicrune · Post by **entropicrune** » Sun Oct 12, 2003 3:52 pm

How do you test ebuilds if all you have is a co-located production box?

I recently crashed my co-located production box by attempting emerge world.

The client I use to access the production box can't be used for testing updates because I am not allowed to install gentoo on it, and because I don't have much bandwith in that location anyway.

I would not mind switching users to the test system, so long as a hardware reset would reboot into the stable system.

It would appear I need to do something like this...

Code: Select all

mkdir /trial
mkdir /trail/tmp
# copy everything into /trial, except /trial, /boot, /tmp
# (Should /mnt and /proc be copied??)
cp -R /bin /trial
cp -R /dev /trial
cp -R /etc /trial
cp -R /home /trial
cp -R /lib /trial
cp -R /mnt /trial
cp -R /opt /trial
cp -R /proc /trial
cp -R /root /trial
cp -R /sbin /trial
cp -R /service /trial
cp -R /usr /trial
cp -R /var /trial
# change file hierarchy root
chroot /trial /bin/bash
env-update && source /etc/profile
# update: but only the file hierarchy originally rooted at /trial
emerge rsync
emerge -u world
etc-update
# recompile the new kernel
cd /usr/src/linux
# Clueless noob IQ test failure: At this point need to somehow
# magically switch to new kernel and daemons without rebooting...
# because if the update failed, we want a reboot to return us to
# the known good kernel and daemons. Will need to start a new ssh
# session.
#
# Presuming everything worked, replace the originals with updates.
# First reboot. Then...
rm -R /bin
rm -R /dev
rm -R /etc
rm -R /home
rm -R /lib
rm -R /mnt
rm -R /opt
rm -R /proc
rm -R /root
rm -R /sbin
rm -R /service
rm -R /usr
rm -R /var
cp -R /trial/bin /bin
cp -R /trial/dev /dev
cp -R /trial/etc /etc
cp -R /trial/home /home
cp -R /trial/lib /lib
cp -R /trial/mnt /mnt
cp -R /trial/opt /opt
cp -R /trial/proc /proc
cp -R /trial/root /root
cp -R /trial/sbin /sbin
cp -R /trial/service /service
cp -R /trial/usr /usr
cp -R /trial/var /var
# Replace the old kernel with the tested one (I use grub)
mount /dev/hda1 /boot
cp /usr/src/linux/arch/i386/boot/bzImage /boot
umount /boot
reboot

Is that sensible, or is there a better way to accomplish this? And how about the missing bit in the middle?

Thanks.

kopte · Post by **kopte** » Fri Sep 03, 2004 7:59 am

Hi,

if you don't want to upgrade several packages on your production system. And this packages are removed from portage, than you are in big trouble. You can not even reemerge that particular version as the ebuild is deleted by "emerge sync".

why is that done?
-space+security i think

Space is no matter if you only save the version you have actually installed. If it is because of security reasons, than save the closed "stable" ebuild to the installed version.

Otherwise it will be very hard to use gentoo on machines which are not updated so often.

Greetings
Kopte

sf_alpha · Post by **sf_alpha** » Fri Sep 03, 2004 1:13 pm

I'm using Gentoo for some production systems too.
Serving abount 200k pageview everyday.

For a webserver, Apache 2.0 works nicely with php with default prefork mpm. Some modules extension ca make apache unstable. But I think If use for serving apache+cgi+perl+php+mysql it works nicely.

You can use alternative older Apache 1.3 which might work more stable but not too fast. Apache 1.3 extra modules mostly work stable too but not for 2.0.

Threading mpm_worker for apache 2.0 is possible but with no php, as known as php_thread_safe is not as stable as we can use on high-load production server and most php_extensions is considered not thread_safe stable. If you use it as download_sever, mpm_worker works well with simple php (Use less php extensions as possible) It saves much of your RAM. (With NPTL is extremely fast)

Problem with threaded apache + php also happened with Redhat

But for now, I never got problem with mysql. !

yodi · Post by **yodi** » Sat Sep 04, 2004 11:48 am

I recently set up a web server for the purpose of a little extra cash, with the experiance I had of runnng gentoo on my desktop computer I thought it would be easy to do the same on a server.
Anyway works fine but a updating world led to a few pages not work due to an upgrade of php and also a few problems with apache loading the new php module.

Apart from the trouble with portage and the ease of witch you can if not careful replace a stable application with an unstable one, gentoo works amazing as a server. I don't think I would run any other distro as a server as I feel it is a great base ground to supply the services my customers need.

After the trouble I ran into with the updates it would be great to have some means of which to test the build before you deploy that to your server. I was thinking of running a vmware setup the same as my server but that would be as costly as buying another test server, well nearly.

Has anyone made a good way to sandbox ebuilds yet?

o5gmmob8 · Post by **o5gmmob8** » Mon Sep 06, 2004 1:39 am

Hi guys,

I maintain a server for school here, only a few hundred users that access apache, mysql, and ftp. I agree, gentoo can be unforgiving at times if you run the latest stuff without testing it first.

I found that running 2 gentoos, one in use and the other in a chrooted environment, works pretty well - If I have problems with software in the chrooted gentoo, then I won't install those packages in the current system or vice versa. This basically means that I switch back and forth to different installs if a problem occurs in one. While this certainly works, this is not as good as running the server 24X7 and the clients will complain about something being down for 5 minutes.

I like the Windows XP feature that doesn't really work, the roll back feature. Could that be applied to gentoo? Basically, some limited number of binary packages are built and kept in a safe place on the system in the event of a newer package having problems. GCC would be a good place to start since gentoo relies on it so heavily.

Walter

kopte · Post by **kopte** » Mon Sep 06, 2004 8:45 am

Anyway works fine but a updating world led to a few pages not work

This is the reason why i want to stick to a special version of a program.

But an emerge sync will delete old versions from portage. Is there a way to prefent this?

@walterw
if you don't do

Code: Select all

emerge clean

you can switch to the older gcc version with

Code: Select all

gcc-config

for every other package just unemerge it.

Klavs · Post by **Klavs** » Mon Sep 06, 2004 9:04 am

You simply use quickpkg to create binary package of a given installed program, or remember to add "buildpkg" to FEATURES in /etc/make.conf to make portage automatically build binary packages for you - so you can reinstall in a jiffy, if an upgrade fails.

kopte · Post by **kopte** » Mon Sep 06, 2004 10:39 am

@klavs

thanks it works.
quickpackage <installed but outdated package>
emerge -avk world

Greetings
Kopte

o5gmmob8 · Post by **o5gmmob8** » Mon Sep 06, 2004 1:05 pm

Hi,

I knew there was a switch for that, I will have to turn it on so I can rollback to older software just in case.

Walter

klync · Post by **klync** » Tue Feb 01, 2005 4:55 am

I know there are a few brave souls who run gentoo on high-end production machines. Seems most, if not all of them, can afford to have 2x their production environment, as well as the time to set up some thorough environment replication (more than just quickpkg for ++ servers! this might involve rsync, cron, hardware, use and cflags for the various systems...) and testing.

You can make a mighty machine fairly easy using gentoo. But mainly, the admin trouble is not failed ebuilds, overwriting files (yay etc-update!), or cross-appliation compatibility. It boils down to the lack of guarantees for application consistency across successful ebuilds. Some examples: I was locked out from new ssh sessions after successfully updating "system", and had to get access to the closet where the machine is stored to tell the daemon to reload. One could say I should've seen it coming, and I could've reloaded ssh from my original terminal session, if I had thought it was necessary. But, worse, the cyrus-sasl and cyrus-imapd ebuilds have had upstream compile-time defaults changed, resulting in unreadable db files (sasl went from berkdb to gdbm, iirc, and imapd's seen.db files recently switched from using berkdb to "skiplist" format). These cost me some down time, and it would've taken some thorough testing to find before deploying.

acasto · Post by **acasto** » Tue Feb 01, 2005 6:26 am

I agree with klync. I have four systems at work running gentoo. A file server, web server, mail server, and an ssh-serial console. It's all older hardware, a p-III 800, two p-II 350, and an old p-I 166 laptop for the serial console. All run steady with no instability. The only problems are it's easy to get careless with etc-update, and like klync said, unexpected ebuild changes. One issue I had was with qpopper. It used to be where you would compile it, then set it to run in either xinetd or standalone. Then one day it just quit working, this was a week or so though after the update because the service wasn't re-started right aways. It turns out, a use flag was added to where you had to set the mode at compile time, however it defaulted to standalone. So now I'm just sure to use the -v switch with emerge and carefully study the ebuilds, and pay extra attention to etc-update.

Oh yeah, it may be a good idea to set you up an ssh-serial console.

- Adam

gentoo for production , experiences?

gentoo for production , experiences?

gentoo as production-server

Problem with outdated packages on production systems

gentoo test server

gentoo test server

[Another] Mixed Review