Installing Gentoo on a very big server.... 16 CPUs, 64GB RAM

Message

Execute · Post by **Execute** » Tue Dec 07, 2004 10:43 pm

Hi,

This is just a short installation report....

at work we evaluated the possibilities for secure internet access from a separate LAN. Our solution was to use a terminal server in our DMZ and a terminal client on the workstations.

In order to save license cost, we choosed gentoo linux as OS for the server and CygWin as terminal client. The server choosed was an IBM x445 wilth the following hardware:

16 x P4 XEON MP 2.0 GHz
64 GB RAM
IBM ServeRaid 4MX
2 x 36.4 GB System Disk (Raid1)
14 x 36.4 GB User Space (home1 and home2) in 2 Raid 5 configuration (2x6 + 2 spare)

The installation:

Getting all exited about the hardware I started the installation by booting the gentoo live CD.
After booting, Linux reported around 850 MB RAM and 3!!! CPUs....
... that was disappointing....
after several boot options and restarts I started to install gentoo with 3 CPUs and 850 MB of ram....
After bootstraping the system and configuring the kernel for x440 NUMA support with 64GB of ram, gentoo reported 16CPUs and 64GB of RAM. Now installing gentoo turns out to be fun.

Compiling a 2.6.x Kernel with the option -j works and takes around 45 seconds!!!

The installation went fine, compiling OpenOffice and KDE still takes a while, but in the end we ended with a system that serves as XDMCP server in our network and around 50 users are using the system at a time.

This means that we have 50 times a full featured KDE session up and running. The overall system load is between 3.5 and 6.5, which is acceptable with a 16 CPU server. Network load turns out to be around 100 MBit (with a GBit connection)

A bottleneck is the system partition. If multiple users are starting applications, the RAID1 does not deliver enough througput. We thought about a RAM disk of about 20GB and loading KDE completely into the RAM disk. This is not yet done, but will for sure solve the IO Problem. RAM is not an issue. 50 Users are using around 10GB of RAM only. That leaves 54 GB for the SysAdmin

The first big update:

The first big update was a disaster.

I did an emerge sync and then an emerge -Du world and nothing worked after that.
Our system is used via XDMCP and uses the GDM for this. During the update GDM changed the configuration files and all sessions where gone. All users where forced to use a gnome failsafe session during logon. What a disaster for 900 users....

Can you imagine the amount of calls I got ?????

I which the portage maintainers would put a warning on the packages during an emerge saying: WARNING: Configuration file change.... emerge will brake current configuration. No automatik migration possible!!!

After all, installation was like for a workstation. No special problems (except the need for a special Kernel config). GDM config has a problem specifying more that 100 XDMCP sessions allowed. The graphical frontend only allows 100 as the maximum number. We want to allow a maximum of 500 concurrent sessions. That requires manual config file changes, but this is acceptable.

Special tools that could be usefull for a system like this:

- a CPU time quota manager (look in /etc/security/limits.conf)
- a logon time quota manager
- a session grabber to watch a user session
- a more multi user aware emerge...

The most important thing is a CPU time quota manager. The sysadmin should have enough power to perform his tasks

If you have any questions, just mail me

ciao

Execute

ewan.paton · Post by **ewan.paton** » Wed Dec 08, 2004 12:50 am

i guess youve learnt emerge -uD system is a bad idea on any production system, i would buy a p4 test server and make sure any app that gets installed is thourghly tested bedfore its installed and probably have a couple of backup partitions with known stable configurations

Sujao · Post by **Sujao** » Wed Dec 08, 2004 9:04 am

Hey execute you dont happen to have another system of this kind to spare, do you? I would be satisfied with only 8 cpu's and 8gb ram. no problem

z3ro · Post by **z3ro** » Wed Dec 08, 2004 9:29 am

Post some pictures. PICTURES DAMN YOU

col · Post by **col** » Wed Dec 08, 2004 9:47 am

I manage many gentoo servers and I have found over the years that it is important to have a test server. Once per month I emerge sync, emerge -u world. The if the test server shows problems I hold back the updates on the other servers. Also good for testing out new kernels. I also use the server for evaluating other software etc. I must say that emerge -u world and etc-update has screwed thing up more times than I can remember whether it being the programmers fault or mine.

Execute · Post by **Execute** » Wed Dec 08, 2004 11:39 am

col wrote:I manage many gentoo servers and I have found over the years that it is important to have a test server.

This is what we do now. The test system needs some different options in the kernel, but thats OK. For sure its cheaper than buying another server like this...

ciao

Execute

Execute · Post by **Execute** » Wed Dec 08, 2004 11:47 am

Z3RO 0 wrote:Post some pictures. PICTURES DAMN YOU

Hehe,

take the ones from the IBM site. Look for the x445. Our system consists of two of those on top of each other and an EXP300 (SCSI enclosure with 14 bays)

This takes 11 Units in a 19" Rack. (2x4 for the servers and 3 for the EXP300)

If you take the visual tour on the ibm site and click on view4, this is how it looks like. You just need to substitude the RXE-100 with the Harddrive Rack.

ciao

Execute

BTW: Check the Fujitsu-Siemens Primergy RX800 I think.... looks similar????

Execute · Post by **Execute** » Wed Dec 08, 2004 11:51 am

Sujao wrote:Hey execute you dont happen to have another system of this kind to spare, do you? I would be satisfied with only 8 cpu's and 8gb ram. no problem

Wait... I think I got one in my drawer.....

... mhh.... maybee here.....

... no, its gone. Sorry

ciao

Execute

tnt · Post by **tnt** » Wed Apr 12, 2006 5:35 pm

Is this monstrum still up and running?

dmartinsca · Post by **dmartinsca** » Thu Apr 13, 2006 1:40 am

wow... where is the drooling emoticon when you need it?

Execute · Post by **Execute** » Thu Apr 20, 2006 6:39 pm

tnt wrote:Is this monstrum still up and running?

Yep. It's still up and running. On a normal working day there are about 100-150 KDE Sessions open. The load is lower than we expected and the memory is much more than enough.

Before I changed the Department in this organization, I tried to create a BIG ramdisk in order to load all the KDE stuff from the ram, but it crashed while formatting a 32GB Ramdisk.... now My Co-Workers can worry about this problem

.... I'm just a normal user of that nice system now.

ciao

Execute

Grooby · Post by **Grooby** » Thu Apr 20, 2006 8:36 pm

w/ such wonderful and powerful machine,
why not setup XEN so you have a production and test on the same box?!?!

vipernicus · Post by **vipernicus** » Thu Apr 20, 2006 8:43 pm

Quick! Ask it what the meaning of life is! We don't have much time!

jmbsvicetto · Post by **jmbsvicetto** » Thu Apr 20, 2006 10:14 pm

dmartinsca wrote:wow... where is the drooling emoticon when you need it?

++1

Execute.
If you ever find your spare system in the drawer, under the bed or in a pocket, let us know!

Xzet · Post by **Xzet** » Sat Apr 22, 2006 9:31 pm

show us pics plz

Execute · Post by **Execute** » Sat Apr 29, 2006 11:05 am

Xzet wrote:show us pics plz

Sorry, you need to live with the pics from the IBM site. It's inside a military installation and they shoot me if I take pictures...

ciao

Execute

azp · Post by **azp** » Sat Apr 29, 2006 12:04 pm

Execute wrote:Sorry, you need to live with the pics from the IBM site. It's inside a military installation and they shoot me if I take pictures...

haha tough luck =)

syg00 · Post by **syg00** » Sat Apr 29, 2006 12:34 pm

execute, you may want to keep an eye on projects such as cpusets and ckrm. Will certainly give you more control than ulimits. ckrm is still a bit "alpha" at present - looks to have some promise though.

BlueFusion · Post by **BlueFusion** » Sat Apr 29, 2006 1:52 pm

Gentoo inside a military installation. Sounds like they did something right for a change!

Maedhros · Post by **Maedhros** » Sat Apr 29, 2006 2:59 pm

Moved from Installing Gentoo to Gentoo Chat since this isn't a support request.

playfool · Post by **playfool** » Sat Apr 29, 2006 7:44 pm

*drool*

I have nothing intelligent to say, that rig just made me think of nakkid chicks.

chunderbunny · Post by **chunderbunny** » Sun Apr 30, 2006 12:08 am

Meh, we've got one of these. 4000 dual core Opterons and 16TB of RAM. Pity it runs SuSE really...

jmbsvicetto · Post by **jmbsvicetto** » Sun Apr 30, 2006 5:14 pm

chunderbunny, wow!!! *drool* *super-drool* *hiper-drool*

I suppose that it runs SLES? Haven't installed Gentoo on any node?

If your system happens to have a descendent or relative you don't need, send it our way, will you?

Monkeh · Post by **Monkeh** » Sun Apr 30, 2006 7:45 pm

chunderbunny wrote:Meh, we've got one of these. 4000 dual core Opterons and 16TB of RAM. Pity it runs SuSE really...

Fuck you

codergeek42 · Post by **codergeek42** » Sun Apr 30, 2006 7:46 pm

chunderbunny wrote:Meh, we've got one of these. 4000 dual core Opterons and 16TB of RAM. Pity it runs SuSE really...

Grr. Share the hardware, please.