Making your Gentoo VMs Awesome™

[GothAlice]

Howdy!

So I've been supporting Gentoo as a platform-of-choice for both my own VM clusters as well as my client's for quite a few years. Speed and out-of-the-box efficiency (plus some deep knowledge of how Gentoo is put together) were key points. One client I managed to estimate $40K/yr savings by migrating from Ubuntu to Gentoo as every single VM could be reduced a notch in allocation while still performing better…

In my travels I've gathered some nifty bits I've been authorized to share with the world; links to the relevant gists are included below.

• The root of each VM is a git repository, hosted on GitHub. This sounds crazy, but it works.
• VMs are generic; on boot they look at the kernel command line for their designated role, pull the root repo and switch to the branch for that role. They also notify a central management server to say they're alive and ready to roll.
• GitHub pushes commit notifications to a central management server we call MCP (tip of the hat to Tron).
• The MCP then identifies (based on its own internal records) which VMs were effected by the commit and sends an RPC request to each to self-update.
• Self-updating is magic. We also use incron to identify changes elsewhere automatically.
• Monitoring is done through MongoDB running on the MCP for central aggregation of stats and logs; this generates a LOT of data.
• User home folders, for the hosting platform, are pseudo-chroots. Users get their own bin, etc, var, &c. and each home folder is already a Python virtualenv.
• Nginx configurations are automatically pulled in from /home/*/etc/nginx/*.conf and all users have sudo access to /etc/init.d/nginx reload — this is resilient because reload checks the config first.
• All users have their own cron folders and init.d folder: ~/etc/cron/{daily,hourly,weekly,monthly,yearly}/ where they can drop scripts. Access to crontab is restricted to clients whom I know know what they are doing.
• Logging into one of the servers is both informative and attractive.
• On most of the VMs, home folders are each a separate EBS volume; this allows us to allocate explicit IOPs on a per-user basis and move users around (or to isolation) very easily. On one of the clusters home folders are distributed using moosefs to exploit the cheap (and fast) ephemeral storage on each separate VM instance.
• Prelink and distcc are my friends. A genkernel compile from distclean compiles in about 50 seconds. (Not that we use Genkernel kernels, we use the Xen kernels provided by Amazon or Rackspace, I just wanted to see. ;)

Comments and questions would be awesome, also, what forms of automation is everyone else using?

Have a great day,
— Alice.

P.s. edited to add that the auto-update gist will be updated with some of the boot-time automation for self-configuration as I can get the releases OK'd. Also updated to add notes about user home folder storage.

Latest edit: just wanted to note that the RAID auto-configuration script has been added to the "self-updating is magic" gist.

[SlashBeast]

Unless you run nginx in the per-user chroots, allowing them to push own config to nginx is plain wrong, you can adjust the config to hijack other users' files, for example set root to other-user-dir and disable php or anything for that so you can easly copy the files. Also with symlinks you can get any-nginx-user readable file, and the in-nginx symlink checker is vuln to race condition so the very only proper solutions is grsecurity and group for checking if symlink owner match.

Also what you use for the pseudo chroots, a LD_PRELOAD things?

[GothAlice]

[jlpoole]

This is a great post and very inspirational. Thank you for taking the time to share what you did.

Any updates or afterthoughts given that this posting has matured for several years?