This article explains one way to handle management of several different, yet similar sites using the following software:

Phusion Passenger setup is already described in this article and applies here for the most part. The few differences are mentioned specifically. The high-level view of this configuration is as follows:

On the filesystem level this configuration looks like this:

The important thing is to keep everything puppetmaster needs to run in /etc/puppet. A number of things can go wrong if you change the defaults, for example:

These happened on puppetmaster 0.25.4; newer puppetmaster versions may be better. With the above directory layout you have one foolproof last resort: symlink every module directory in /etc/puppetglobal/modules to /etc/puppet/modules. This way the /etc/puppet directory is completely self-contained. All you need to do is add symlinks for any new global modules you create.

The rationale for doing this split is simple:

It is very important to write any local modules so that they only depend on global modules, not vice versa. Otherwise you'll end up having to migrate your local modules to all servers, meaning they're not really local and should really be in the global puppet configuration instead.

Setting up the environment

Initial setup

First, make sure that Puppetmaster and Passenger are configured properly on all puppetmaster servers, e.g. as described here. Then stop puppetmasterd and puppetd services:

$ /etc/init.d/apache2 stop
$ /etc/init.d/puppet stop

This prevents them messing up the puppet configuration files while you're at work.

Master server configuration

If you don't have puppetmaster in production, all you need to do is:

$ cd /etc/puppet
$ git init

If you have a production puppetmaster already, do something like this instead:

$ mv /etc/puppet /etc/puppetglobal
$ mkdir -p /etc/puppet/manifests/nodes
$ mkdir -p /etc/puppet/modules
$ mv /etc/puppetglobal/puppet.conf /etc/puppet/
$ mv /etc/puppetglobal/manifests/site.pp /etc/puppet/manifests/
$ mv /etc/puppetglobal/manifests/nodes/* /etc/puppet/manifests/nodes/
$ cd /etc/puppet
$ git init
$ cd /etc/puppetglobal
$ git init

After this you can migrate site-specific ("local") modules to /etc/puppet/modules.

Slave server configuration

Make the existing puppet configuration directory a git repo:

$ cd /etc/puppet
$ git init

Then configure puppetmaster normally. Next checkout master's global git repository using SSH:

$ cd /etc
$ git clone ssh:// puppetglobal
Cloning into puppetglobal...'s password: 
remote: Counting objects: 1857, done.
remote: Compressing objects: 100% (1567/1567), done.
remote: Total 1857 (delta 726), reused 0 (delta 0)
Receiving objects: 100% (1857/1857), 16.95 MiB | 1.12 MiB/s, done.
Resolving deltas: 100% (726/726), done. 

Make sure the SSH user has appropriate permissions to /etc/puppetglobal on primary puppetmaster server.

Puppet module migration

Although migrating the files is trivial, you may have to manually merge lots of code. However, if module/class names don't clash, you can do the migration incrementally by starting with all modules in /etc/puppet/modules and moving them over to /etc/puppetglobal/modules in time. Also make sure the fileserver references in your modules don't point to a static, fully-qualified fileserver URI - unless you can use one for all puppetmasters. For details, look here.

Basic workflow

When you're writing puppet code, first determine whether it's should be global or local. In a nutshell, anything that contains sensitive and/or site-specific information should be placed into the local puppet modules; the rest should be global. Examples:

In case of local puppet code you'll just follow the basic Git workflow. If you're making global code changes, do them at the master server and fetch the changes to the slaves using Git, e.g. with

$ cd /etc/puppetglobal
$ git pull --rebase origin

You should be able to do pushes to the master, too, if you know your way around Git.