Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Gentoo as a Production OS
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
sf_alpha
Tux's lil' helper
Tux's lil' helper


Joined: 19 Sep 2002
Posts: 136
Location: Bangkok, TH

PostPosted: Wed Dec 05, 2007 3:18 am    Post subject: Reply with quote

I also use Gentoo on production system (I have around 10-20 servers).

Gentoo offer nice customization, I mean not only the USE Flags or CFLAGS. I found that initscripts system is also powerful and ebuild system is great.

Making ebuild or my own initscripts is easy. I can integrate my own software, patches and other stuffs inside overlay.

If your site have a same hardware such as same series of DELL as I have. Use of one server as package builder inside chroot, only add FEATURES="buildpkg" in make.conf, You should use same make.conf and anything in /etc/portage for every servers to do this.
Share /usr/portage among all servers using NFS, you get portage, package, distfiles in place. 'emerge -k whatever' is much like yum or apt-get.

One thing I need to mention you. You may create nice customized Stage3 from that chroot with only one tar command then deploy the hundreds of server from the install CD with the GRUB on it. Now you have you own MyGRP (My Gentoo Reference Platform).

People usually hate Gentoo because of compiling time, but it actually because they even not know what the emerge command can do

(Change: remove buildsyspkg)
_________________
Gentoo Mirrors in Thailand (and AP)
http://gentoo.in.th


Last edited by sf_alpha on Thu Dec 06, 2007 11:05 am; edited 1 time in total
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2752
Location: The Peanut Gallery

PostPosted: Wed Dec 05, 2007 2:53 pm    Post subject: Reply with quote

Great summary, sf_alpha, and nice outline of how to do it: thanks! /me links here from tips
Back to top
View user's profile Send private message
Kasumi_Ninja
Veteran
Veteran


Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands

PostPosted: Wed Dec 05, 2007 3:50 pm    Post subject: Reply with quote

sf_alpha wrote:
I also use Gentoo on production system (I have around 10-20 servers).

Gentoo offer nice customization, I mean not only the USE Flags or CFLAGS. I found that initscripts system is also powerful and ebuild system is great.

Making ebuild or my own initscripts is easy. I can integrate my own software, patches and other stuffs inside overlay.

If your site have a same hardware such as same series of DELL as I have. Use of one server as package builder inside chroot, only add FEATURES="buildpkg buildsyspkg" in make.conf, You should use same make.conf and anything in /etc/portage for every servers to do this.
Share /usr/portage among all servers using NFS, you get portage, package, distfiles in place. 'emerge -k whatever' is much like yum or apt-get.

One thing I need to mention you. You may create nice customized Stage3 from that chroot with only one tar command then deploy the hundreds of server from the install CD with the GRUB on it. Now you have you own MyGRP (My Gentoo Reference Platform).

People usually hate Gentoo because of compiling time, but it actually because they even not know what the emerge command can do


Nice summary! I wonder though why you use both buildpkg and buildsyspkg?

Code:
              buildpkg
                     Binary  packages  will  be  created for all packages that are merged. Also see quickpkg(1) and emerge(1) --buildpkg and
                     --buildpkgonly options.

              buildsyspkg
                     Build binary packages for just packages in the system set.


P.S.
Do you have a link on howto create your own installcd?
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Wed Dec 05, 2007 4:42 pm    Post subject: Reply with quote

Aniruddha wrote:
Suicidal wrote:
Gentoo is fine as a production server, based on you build everything on a image host.

I had at one time ~20 gentoo servers.

I had one that I did all of the building on, I ran it like so:

Code:
emerge system

emerge world

emerge -e world


All with bulidpkg in the use flags.

after I thought I had a stable release I synced the packages to my test box (vm) and:

Code:
emerge -k system && emerge -k world && emerge -ek world


If there was an issue I figured out what not to do and tested again.

It was really not too difficult, and it sure beat Redhats or any other binary distros upgrade mechanism.


How did you sync the binaries with the other 20 servers?


I created a custom bash script to do the following:

sync /etc/make.conf
sync /etc/portage

That way the clients would always have the same make.conf and /etc/portage/foo.bar

I found this. article on the wiki which shows you how to control what directories portage excludes from a emerge --sync

I then croned the script to run every night.

Luckily in my environment all of the servers were identical, but even if they weren't I would think that:

Code:
CFLAGS="-march=pentium3 -mtune=nocona"


would be reasonable ${CFLAGS} that should give decent optimization and compatibility.

I am kicking myself in the rear for not saving that script.

/correction (previous post) I had buildpkg in $FEATURES not $USE
Back to top
View user's profile Send private message
Kasumi_Ninja
Veteran
Veteran


Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands

PostPosted: Wed Dec 05, 2007 4:52 pm    Post subject: Reply with quote

Suicidal wrote:
I found this. article on the wiki which shows you how to control what directories portage excludes from a emerge --sync

Thanks a lot! This wiki article contains lots of useful info.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered
Back to top
View user's profile Send private message
tylerwylie
Guru
Guru


Joined: 19 Sep 2004
Posts: 456
Location: /US/Illinois

PostPosted: Wed Dec 05, 2007 7:24 pm    Post subject: Reply with quote

I deal with Gentoo on production systems all the time, in virtual machines as well as physical. Works pretty well I must say but we are in the process of replacing them.
_________________
Bastiat wrote:
“The state is that great fiction by which everyone tries to live at the expense of everyone else.”
Back to top
View user's profile Send private message
Kasumi_Ninja
Veteran
Veteran


Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands

PostPosted: Wed Dec 05, 2007 7:40 pm    Post subject: Reply with quote

tylerwylie wrote:
I deal with Gentoo on production systems all the time, in virtual machines as well as physical. Works pretty well I must say but we are in the process of replacing them.


Replacing the computers or Gentoo?
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered
Back to top
View user's profile Send private message
tylerwylie
Guru
Guru


Joined: 19 Sep 2004
Posts: 456
Location: /US/Illinois

PostPosted: Wed Dec 05, 2007 7:47 pm    Post subject: Reply with quote

Aniruddha wrote:
tylerwylie wrote:
I deal with Gentoo on production systems all the time, in virtual machines as well as physical. Works pretty well I must say but we are in the process of replacing them.


Replacing the computers or Gentoo?
Both in some cases.
_________________
Bastiat wrote:
“The state is that great fiction by which everyone tries to live at the expense of everyone else.”
Back to top
View user's profile Send private message
Kasumi_Ninja
Veteran
Veteran


Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands

PostPosted: Wed Dec 05, 2007 8:07 pm    Post subject: Reply with quote

tylerwylie wrote:
Aniruddha wrote:
tylerwylie wrote:
I deal with Gentoo on production systems all the time, in virtual machines as well as physical. Works pretty well I must say but we are in the process of replacing them.


Replacing the computers or Gentoo?
Both in some cases.


To which OS are you switching?
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered
Back to top
View user's profile Send private message
sf_alpha
Tux's lil' helper
Tux's lil' helper


Joined: 19 Sep 2002
Posts: 136
Location: Bangkok, TH

PostPosted: Thu Dec 06, 2007 10:59 am    Post subject: Reply with quote

I think only FEATURES="buildpkg" is enough.

But I not sure is the system packages are built or not with only buildpkg so I added buildsyspkg also.
_________________
Gentoo Mirrors in Thailand (and AP)
http://gentoo.in.th
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Fri Dec 07, 2007 1:46 am    Post subject: Reply with quote

sf_alpha wrote:
I think only FEATURES="buildpkg" is enough.

But I not sure is the system packages are built or not with only buildpkg so I added buildsyspkg also.


/etc/make.conf.example wrote:
# 'buildpkg' causes binary packages to be created of all packages that
# are being merged.

# 'buildsyspkg' only build binary packages for system packages.


It appears that buildsyspkg is probably a good feature to implement in case someone
does something epically stupid like unmerging gcc or bash.

buildpkg will build packages for everything.
Back to top
View user's profile Send private message
anomalizer
n00b
n00b


Joined: 15 Jul 2005
Posts: 16

PostPosted: Fri Dec 07, 2007 5:24 pm    Post subject: Reply with quote

After years of using gentoo, I'm still not sure how to have a perfectly reproducible setup. Assume I have 3 boxes configured to be exactly the same (getting ot that stage is a big deal in itself), adding a 4th box after six months that is a replica of the first 3 boxes seems near impossible.

Note that I am neither saying the software is unstable or the package management is flaky; just that it is very tough to try and replicate an existing setup. That IMNSHO makes it a bad choice for production environments.
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Sat Dec 08, 2007 1:51 am    Post subject: Reply with quote

anomalizer wrote:
After years of using gentoo, I'm still not sure how to have a perfectly reproducible setup. Assume I have 3 boxes configured to be exactly the same (getting ot that stage is a big deal in itself), adding a 4th box after six months that is a replica of the first 3 boxes seems near impossible.

Note that I am neither saying the software is unstable or the package management is flaky; just that it is very tough to try and replicate an existing setup. That IMNSHO makes it a bad choice for production environments.


What I used to do was build in a chroot like the gentoo documentation;
except that the chroot was a folder on one of my servers.

I would omit any part such as setting the hostname, or editing configs
I just used the chroot for emerging packages.

After I had the chroot updated I would emerge -e world

I then unmounted /dev /proc from the chroot and then tarred the entire installation.

cd / && tar -cvjp ./ -f $(date +%Y%m%d)-baseline.tar.bz2

I used that archive instead of the default stage.

For future installs I just update the existing chroot and then tar it up like I did before.

1.You can never have an exactly reproducible system if make.conf /etc/portage
and /etc/make.profile are not consistent among all systems which is why in my
previous post they were all synced with the build host before even portage.

2. If I needed to change a USE flag or add a package it was done on the build host
and then synced to clients.
Back to top
View user's profile Send private message
Kasumi_Ninja
Veteran
Veteran


Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands

PostPosted: Sat Dec 08, 2007 9:28 am    Post subject: Reply with quote

Suicidal wrote:
anomalizer wrote:
After years of using gentoo, I'm still not sure how to have a perfectly reproducible setup. Assume I have 3 boxes configured to be exactly the same (getting ot that stage is a big deal in itself), adding a 4th box after six months that is a replica of the first 3 boxes seems near impossible.

Note that I am neither saying the software is unstable or the package management is flaky; just that it is very tough to try and replicate an existing setup. That IMNSHO makes it a bad choice for production environments.


What I used to do was build in a chroot like the gentoo documentation;
except that the chroot was a folder on one of my servers.

I would omit any part such as setting the hostname, or editing configs
I just used the chroot for emerging packages.

After I had the chroot updated I would emerge -e world

I then unmounted /dev /proc from the chroot and then tarred the entire installation.

cd / && tar -cvjp ./ -f $(date +%Y%m%d)-baseline.tar.bz2

I used that archive instead of the default stage.

For future installs I just update the existing chroot and then tar it up like I did before.

1.You can never have an exactly reproducible system if make.conf /etc/portage
and /etc/make.profile are not consistent among all systems which is why in my
previous post they were all synced with the build host before even portage.

2. If I needed to change a USE flag or add a package it was done on the build host
and then synced to clients.


Lol, great post. The problem with Gentoo is not if you can do it but how you can do it. I learned some valuable lessons from this thread :)
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered
Back to top
View user's profile Send private message
crackytron
n00b
n00b


Joined: 16 Nov 2007
Posts: 24

PostPosted: Tue Dec 18, 2007 11:48 am    Post subject: Reply with quote

This thread is a really interesting read :)

This is what I've decided to do:

RHEL Server on Rackspace that is managed and auto-updates.

3x Gentoo Virtual Servers installed on this:

- Virtual Server one: "Present"
This would be the production system, it would be booted all of the time and would contain the most current version of sites, etc.

- Virtual Server two: "Past"
This Virtual server would not be running most of the time, it would contain the last most stable configuration and would only be updated when I was SURE that "Present" was 100% stable. It would also be a fallback if "Present" broke for some reason. It would be like a redundant system.

- Virtual Server three "Future"
This is where I'd apply updates, develop sites, test ideas. When I was 95% certain this was stable, I would have mirrored onto "Present" and the copy of "Present" cloned onto "Past".
It would run most of the time but have a very small % of the server's resources. Sites on here would be ideally deployed via an SVN based deployment system (I have one that sort of works anyway).

This would mean that there was stability, capacity for testing, and a good scope of fallback should shit hit the fan.
Hackers would only be able to gain access to "Present", and they wouldn't be able to touch the actual RHEL OS. If "Present" was comprimised, I could copy "Past" onto it, do further testing of "Future" then copy future onto it.

The problems:

- I have no idea how to do this.

- I think it is a good idea but I'm not 100% sure. Is it overengineered?

- We'd need 3X the hard disk space.

- It would take a lot of work to set up.

I've heard Virtuozzo does something like this - whats the general consensus on virtual servers for a configuration like mine. Is there much performance hit to doing what I do? Also it is expensive and if there are any free alternatives that'd be awesome.


Lastly, I don't know about server-hardware much at all. If anyone could recommend whether I should order an Xeon, Opteron, SCSI vs SATA, how much RAM, sensible RAID? - I don't want to overspend and bankrupt our small company but I don't want to underspend. FYI we're going to be going with Rackspace as their support is excellent.


PS - Apologies if I'm asking too much. I know I should get proper training but I'm working for a small company and we cannot afford things like that :E

PPS - Also would it be dumb to use a server like this as an SVN repos?
_________________
welp
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2752
Location: The Peanut Gallery

PostPosted: Tue Dec 18, 2007 3:56 pm    Post subject: Reply with quote

crackytron wrote:
The problems:

- I have no idea how to do this.

Lol that was funny! You sounded so confident til then ;p

Quote:
- I think it is a good idea but I'm not 100% sure. Is it overengineered?

- We'd need 3X the hard disk space.

- It would take a lot of work to set up.

Sounds fine to me; disk space is not the constraint nowadays (CPU and bandwidth are) and the setup should be reproducible.
I suggest you log into irc.freenode.org and check out #gentoo-server and also #gentoo-chat as there are some knowledgeable people there.

Quote:
PPS - Also would it be dumb to use a server like this as an SVN repos?

Not imo. Any service can be run in a VM. (svn uses more resources than cvs though.)
Back to top
View user's profile Send private message
crackytron
n00b
n00b


Joined: 16 Nov 2007
Posts: 24

PostPosted: Tue Dec 18, 2007 4:56 pm    Post subject: Reply with quote

Thanks steveL!

I think what I'll do is I'll try setting up OpenVZ (the open version of Virtuozzo AFAIK) on a test server or something. Once thats done I can fuck around to my hearts content.

With OpenVZ/Virtuozzo do all the servers have to be running and sharing CPU or can you say like, limit two of them to 0% CPU, etc?

I'm kinda a developer with linux knowledge(ish) thats acting as a sysadmin. I'm not complaining as I enjoy it but it does mean I lack experience.

Here's what I was speccing out for the server with rackspace:

Quote:
Okay, I've looked around and read things and this seems like a reasonable spec for a new server:

Operating System

Red Hat Enterprise Linux
- with Managed Updates, if this is available. It would save me a lot of time and would allow me to focus on maintaining the "virtual" server.

CPU

1x Dual Core AMD Opteron 2214HE
- However, if this is too expensive it is possible to get an AMD Opteron 1212 but it would be nice if the server packed some power.


Memory

1GB of DDR2-5300
- This is fast memory, and can most likely be upgraded if we need more, but Tec4 doesn't need more than 1GB at the moment.


Hard Disk

2x 73GB SCSI drives in RAID
- RAID means that we have the same capacity as one drive but the other acts as a failsafe in case of malfunction
OR
1x 250GB SATA drive
- This would give us a lot more capacity. Our other server has a 70GB IDE drive which I'm surprised still works, and we've nearly filled up.

Bear in mind, Hard drive space is the biggest thing hit by the triple server idea I suggested. This would mean for the first configuration we'd have a mere 24GB for sites. I'd suggest asking about getting 2x 250GB SATA drives in RAID.

Bandwidth

150GB
- I'm not sure how much we get/use currently but 150GB should suffice. Any statistics on current usage would be welcome.


Backups

25GB
- However, I expect we can upgrade this as we put more onto the server. This would be in addition to the "Past" server I mentioned before.

Extras

Redundant Power and Cooling - I'm not sure how much these cost (I would imagine fairly expensive), but it would be up to you guys as to whether you want this to ensure the reliability of the server.
Dedicated Firewall - I don't think we've made any enemies but you never know ;]

_________________
welp
Back to top
View user's profile Send private message
welp
Retired Dev
Retired Dev


Joined: 24 Sep 2005
Posts: 103
Location: Ipswich, UK

PostPosted: Tue Dec 18, 2007 7:07 pm    Post subject: Reply with quote

What what what?! What's my nick doing in your sig? :o
_________________
Gentoo/BSD, Gentoo/Alt AT and Bugday lead
AMD64, Xfce, Sunrise, www-servers, net-irc, lang-misc, Artwork
If you find a bug, submit it! Bugzilla
Back to top
View user's profile Send private message
crackytron
n00b
n00b


Joined: 16 Nov 2007
Posts: 24

PostPosted: Tue Dec 18, 2007 9:17 pm    Post subject: Reply with quote

welp wrote:
What what what?! What's my nick doing in your sig? :o


best post in the thread.

holy cow the saucepan is overflowing brbrb
_________________
welp
Back to top
View user's profile Send private message
georgia_tech_swagger
n00b
n00b


Joined: 14 Nov 2007
Posts: 41

PostPosted: Sat Jan 26, 2008 10:05 pm    Post subject: Anybody else out there with production Gentoo servers? Reply with quote

I have a half dozen production colo servers plane flights away running Gentoo. I now have a very deep and very real fear that there will be a fork where most Devs go, or Gentoo might become defunct. If that happens, I will have to eat thousands of dollars in plane flights, man hours, and downtime flying on planes to datacenters to change servers to Debian. If this happens, I will never come back to WHERE EVER this community ends up.

Aren't there a couple of major managed hosting companies in Europe who use Gentoo? If I'm their CTO, I'm *enraged* right now.
Back to top
View user's profile Send private message
blu3bird
Retired Dev
Retired Dev


Joined: 04 Oct 2003
Posts: 610
Location: Munich, Germany

PostPosted: Sat Jan 26, 2008 10:38 pm    Post subject: Reply with quote

domainFACTORY
_________________
Black Holes are created when God divides by zero!
Back to top
View user's profile Send private message
AllenJB
Veteran
Veteran


Joined: 02 Sep 2005
Posts: 1283
Location: Ashford, Kent

PostPosted: Sat Jan 26, 2008 10:53 pm    Post subject: Reply with quote

There will be no fork. At least, not one that most of the devs will follow.
_________________
http://gentoo-wiki.com :: http://lug.org.uk :: http://www.linux.org/groups/ :: User Blogs
Back to top
View user's profile Send private message
Seek
n00b
n00b


Joined: 22 Jul 2007
Posts: 46
Location: Austria

PostPosted: Sat Jan 26, 2008 11:51 pm    Post subject: Reply with quote

Daniel Robbins won't do a fork.
Read his latest Blog for further information: http://blog.funtoo.org/2008/01/what-to-do.html
and the Gentoo foundation still exists: http://www.gentoo.org/news/20080118-foundation-status.xml

After those events happened, Gentoo is just more 'awake' and hopefully getting better than before.
I think you don't need to worry about your Servers.
Back to top
View user's profile Send private message
i92guboj
Moderator
Moderator


Joined: 30 Nov 2004
Posts: 10089
Location: Córdoba (Spain)

PostPosted: Sun Jan 27, 2008 12:17 am    Post subject: Reply with quote

Don't worry.

People telling you the Earth is flat will not make it flat just using words.

Gentoo is as active as it always was. I have been here for years and can't notice any regresion. In which regards functionality all is better and better everyday, at least that is my view of the things.
_________________
Gentoo Handbook | My website
Back to top
View user's profile Send private message
c0d3g33k
n00b
n00b


Joined: 26 Nov 2003
Posts: 43
Location: S.E. Connecticut

PostPosted: Sun Jan 27, 2008 12:40 am    Post subject: Reply with quote

Unless you paid for a support contract or have some other sort of explicit agreement that leads you to believe Gentoo will continue in perpetuity to support your business and save you trouble and money, you have no basis for any expectation in that regard. FLOSS is a wonderful, wonderful thing, but it doesn't guarantee anything except the ability to carry on with all that you were given.

You have the source code available to you, the tools, portage etc. There is nothing at all preventing you from continuing to run Gentoo, making the updates and changes you want and keeping your servers going. The Gentoo developers, the Gentoo community, the upstream developers of the packages you want owe you nothing. The only thing you stand to lose if the unthinkable happens is loss of free support and development. It's the latter you seem most concerned about losing, and it's the latter you have the least claim to receive.

The builder of my house doesn't owe me indefinite upgrades and repairs just because he built it, even when I pay (dearly) for it, let along getting it for nothing. I am responsible for maintaining it myself. I can do that by paying for workmen to do the things I need, or convince members of my community to help me (most likely in exchange for time spent helping them). With Gentoo you can hire someone to do the work for you, or do it yourself.

That's more than you get from most commercial software.

TANSTAAFL.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum