Introduction

This article explains one way to handle management of several different, yet similar sites using the following software:

• Debian Lenny/Squeeze

• Git version control used with OpenSSH

• Puppetmaster running as a Rack application hosted by Apache which has the Phusion Passenger module.

Phusion Passenger setup is already described in this article and applies here for the most part. The few differences are mentioned specifically. The high-level view of this configuration is as follows:

• Puppet configuration is split into global and local segments

• One puppetmaster server is selected as the master, or more correctly the primary host of the global puppet repository

• Other puppetmaster servers (or slaves) pull the global part of their configuration from the global repository

• All puppet configuration and definition files are stored in Git repositories

• All node definitions are stored in local Git repositories

On the filesystem level this configuration looks like this:

• /etc/puppet: puppet configuration files, local puppet modules, node definitions

• /etc/puppetglobal: global puppet modules

The important thing is to keep everything puppetmaster needs to run in /etc/puppet. A number of things can go wrong if you change the defaults, for example:

• mod_passenger not working at all when trying to add command-line parameters for puppetmasterd into config.ru

• puppetmasterd not respecting $modulepath, or any other variables for that matter

These happened on puppetmaster 0.25.4; newer puppetmaster versions may be better. With the above directory layout you have one foolproof last resort: symlink every module directory in /etc/puppetglobal/modules to /etc/puppet/modules. This way the /etc/puppet directory is completely self-contained. All you need to do is add symlinks for any new global modules you create.

The rationale for doing this split is simple:

• Forces one to write reusable puppet code
• Helps avoid having to reinvent the wheel
• Avoids manual rule migration hell
• Allows use of local modules where applicable

It is very important to write any local modules so that they only depend on global modules, not vice versa. Otherwise you'll end up having to migrate your local modules to all servers, meaning they're not really local and should really be in the global puppet configuration instead.

Setting up the environment

Initial setup

First, make sure that Puppetmaster and Passenger are configured properly on all puppetmaster servers, e.g. as described here. Then stop puppetmasterd and puppetd services:

$ /etc/init.d/apache2 stop
$ /etc/init.d/puppet stop

This prevents them messing up the puppet configuration files while you're at work.

Master server configuration

If you don't have puppetmaster in production, all you need to do is:

$ cd /etc/puppet
$ git init

If you have a production puppetmaster already, do something like this instead:

$ mv /etc/puppet /etc/puppetglobal
$ mkdir -p /etc/puppet/manifests/nodes
$ mkdir -p /etc/puppet/modules
$ mv /etc/puppetglobal/puppet.conf /etc/puppet/
$ mv /etc/puppetglobal/manifests/site.pp /etc/puppet/manifests/
$ mv /etc/puppetglobal/manifests/nodes/* /etc/puppet/manifests/nodes/
$ cd /etc/puppet
$ git init
$ cd /etc/puppetglobal
$ git init

After this you can migrate site-specific ("local") modules to /etc/puppet/modules.

Slave server configuration

Make the existing puppet configuration directory a git repo:

$ cd /etc/puppet
$ git init

Then configure puppetmaster normally. Next checkout master's global git repository using SSH:

$ cd /etc
$ git clone ssh://user@puppet.domain.org/etc/puppetglobal puppetglobal
Cloning into puppetglobal...
user@puppet.domain.org's password: 
remote: Counting objects: 1857, done.
remote: Compressing objects: 100% (1567/1567), done.
remote: Total 1857 (delta 726), reused 0 (delta 0)
Receiving objects: 100% (1857/1857), 16.95 MiB | 1.12 MiB/s, done.
Resolving deltas: 100% (726/726), done. 

Make sure the SSH user has appropriate permissions to /etc/puppetglobal on primary puppetmaster server.

Puppet module migration

Although migrating the files is trivial, you may have to manually merge lots of code. However, if module/class names don't clash, you can do the migration incrementally by starting with all modules in /etc/puppet/modules and moving them over to /etc/puppetglobal/modules in time. Also make sure the fileserver references in your modules don't point to a static, fully-qualified fileserver URI - unless you can use one for all puppetmasters. For details, look here.

Basic workflow

When you're writing puppet code, first determine whether it's should be global or local. In a nutshell, anything that contains sensitive and/or site-specific information should be placed into the local puppet modules; the rest should be global. Examples:

• user module that creates users to each servers should probably be local

• authorization module that adds authorized SSH keys (e.g. to allow rsync-based backups) should probably be local

• apache module should probably be global

In case of local puppet code you'll just follow the basic Git workflow. If you're making global code changes, do them at the master server and fetch the changes to the slaves using Git, e.g. with

$ cd /etc/puppetglobal
$ git pull --rebase origin

You should be able to do pushes to the master, too, if you know your way around Git.