Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Making your Gentoo VMs Awesome™
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
GothAlice
n00b
n00b


Joined: 05 Mar 2012
Posts: 4

PostPosted: Wed Feb 13, 2013 7:37 pm    Post subject: Making your Gentoo VMs Awesome™ Reply with quote

Howdy!

So I've been supporting Gentoo as a platform-of-choice for both my own VM clusters as well as my client's for quite a few years. Speed and out-of-the-box efficiency (plus some deep knowledge of how Gentoo is put together) were key points. One client I managed to estimate $40K/yr savings by migrating from Ubuntu to Gentoo as every single VM could be reduced a notch in allocation while still performing better…

In my travels I've gathered some nifty bits I've been authorized to share with the world; links to the relevant gists are included below.
  • The root of each VM is a git repository, hosted on GitHub. This sounds crazy, but it works.
  • VMs are generic; on boot they look at the kernel command line for their designated role, pull the root repo and switch to the branch for that role. They also notify a central management server to say they're alive and ready to roll.
  • GitHub pushes commit notifications to a central management server we call MCP (tip of the hat to Tron).
  • The MCP then identifies (based on its own internal records) which VMs were effected by the commit and sends an RPC request to each to self-update.
  • Self-updating is magic. We also use incron to identify changes elsewhere automatically.
  • Monitoring is done through MongoDB running on the MCP for central aggregation of stats and logs; this generates a LOT of data.
  • User home folders, for the hosting platform, are pseudo-chroots. Users get their own bin, etc, var, &c. and each home folder is already a Python virtualenv.
  • Nginx configurations are automatically pulled in from /home/*/etc/nginx/*.conf and all users have sudo access to /etc/init.d/nginx reload — this is resilient because reload checks the config first.
  • All users have their own cron folders and init.d folder: ~/etc/cron/{daily,hourly,weekly,monthly,yearly}/ where they can drop scripts. Access to crontab is restricted to clients whom I know know what they are doing.
  • Logging into one of the servers is both informative and attractive.
  • On most of the VMs, home folders are each a separate EBS volume; this allows us to allocate explicit IOPs on a per-user basis and move users around (or to isolation) very easily. On one of the clusters home folders are distributed using moosefs to exploit the cheap (and fast) ephemeral storage on each separate VM instance.
  • Prelink and distcc are my friends. A genkernel compile from distclean compiles in about 50 seconds. (Not that we use Genkernel kernels, we use the Xen kernels provided by Amazon or Rackspace, I just wanted to see. ;)
Comments and questions would be awesome, also, what forms of automation is everyone else using?

Have a great day,
— Alice.

P.s. edited to add that the auto-update gist will be updated with some of the boot-time automation for self-configuration as I can get the releases OK'd. Also updated to add notes about user home folder storage.

Latest edit: just wanted to note that the RAID auto-configuration script has been added to the "self-updating is magic" gist.


Last edited by GothAlice on Wed Feb 13, 2013 11:43 pm; edited 1 time in total
Back to top
View user's profile Send private message
SlashBeast
Moderator
Moderator


Joined: 23 May 2006
Posts: 2831

PostPosted: Wed Feb 13, 2013 8:42 pm    Post subject: Reply with quote

Unless you run nginx in the per-user chroots, allowing them to push own config to nginx is plain wrong, you can adjust the config to hijack other users' files, for example set root to other-user-dir and disable php or anything for that so you can easly copy the files. Also with symlinks you can get any-nginx-user readable file, and the in-nginx symlink checker is vuln to race condition so the very only proper solutions is grsecurity and group for checking if symlink owner match.

Also what you use for the pseudo chroots, a LD_PRELOAD things?
_________________
BitBucket -- better-initramfs
Back to top
View user's profile Send private message
GothAlice
n00b
n00b


Joined: 05 Mar 2012
Posts: 4

PostPosted: Wed Feb 13, 2013 8:48 pm    Post subject: Reply with quote

SlashBeast wrote:
Unless you run nginx in the per-user chroots, allowing them to push own config to nginx is plain wrong, you can adjust the config to hijack other users' files, for example set root to other-user-dir and disable php or anything for that so you can easly copy the files. Also with symlinks you can get any-nginx-user readable file, and the in-nginx symlink checker is vuln to race condition so the very only proper solutions is grsecurity and group for checking if symlink owner match.

The few users who use this particular functionality (there aren't many; most of the clients are managed hosting and never see a shell) have been apprised of the potential pitfalls. This isn't your average public hosting service and there are clear penalties for screwing around. (I.e. immediate termination of service and liability.) The worst a user has done so far (in… 6 years?) was forget a semicolon and forget about it, and a quick support ticket from another client needing to reload the config cleared that up quick. (We subsequently added configtest runs to our regular monitoring.)

Quote:
Also what you use for the pseudo chroots, a LD_PRELOAD things?

Simpler than that. Basically just a $PATH that includes ~/bin. Python's virtualenv system makes this work pretty smoothly (i.e. users can install Python packages as if they were global, but are actually local to the user). No LD_PRELOAD magic or anything like that, thus it's very pseudo. ;) What makes it more like a chroot is the fact that there are support scripts to make ~/etc/init.d/*, ~/etc/cron/{daily,etc.}, etc. work, and their own nginx and application configs are set to log to ~/var/log, etc.

Expert users are able to easily use a --prefix=$HOME and have things "just work". (Though if at all possible we get them to file a request for system-wide installation so we can maintain a consistent cluster setup. We have every version of Python from 2.5 and up installed, as an example.)

— Alice.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum