Differences between revisions 8 and 9
Revision 8 as of 2006-03-13 20:10:45
Size: 7715
Comment: Erorr in adduser usage. Tested with adduser in sarge.
Revision 9 as of 2006-03-13 20:35:00
Size: 7724
Comment: Changed a couple of ' to " so we don't loop and get 'Too many arguments'.
Deletions are marked like this. Additions are marked like this.
Line 148: Line 148:
DUPOPTS='--remove-older-than 2W' DUPOPTS="$DUPOPTS --remove-older-than 2W"
Line 150: Line 150:
#DUPOPTS='$DUPOPTS --exclude **/pictures/XXX' #DUPOPTS="$DUPOPTS --exclude **/pictures/XXX"

[http://packages.debian.org/duplicity duplicity] is a very useful tool to make remote unassisted backups. The backups can be incremental, encrypted, and over a variety of transports. Here is the example of setup to backup various directories remotely. Other tools such as [http://packages.debian.org/backupninja backupninja] can be used for the same purpose.


It is convenient to split the backups per directory, such as /etc, /var/lib/dpkg or /var/spool. This script will create small backups, one per directory defined later in BACKDIRS. The remote machine will contain the following directory tree in LPATH:


and the commands we will run from localmachine will look like this:

duplicity --encrypt-key AE45AE12 --sign-key AE45AE12 \
  --remove-older-than 2W /var/spool \

and then to remove the old backups:

duplicity --encrypt-key AE45AE12 --sign-key AE45AE12 \
  --remove-older-than 2W \

You need first to know what directories to backup on your machine. A bare minimum to reconstruct the installation could be /etc and /var/lib/dpkg, then your mileage may vary. Think about twice, double check the size it takes, and the size you have one the server. How do the size vary along time? how long do you want to keep the old backups?

Here we keep 2 weeks of old backups, with a full backup every week and a daily increment.

Create the ssh and gpg keys

$ su
# apt-get install duplicity keychain
# ssh-keygen -t dsa
# gpg --gen-key
# umask 077
# touch /root/backup.sh /root/.duplicity.conf
# chmod u+x /root/backup.sh
# ls -la /root/backup.sh /root/.duplicity.conf
-rwx------  1 root root 0 2006-01-16 06:47 /root/backup.sh
-rw-------  1 root root 0 2006-01-16 06:47 /root/.duplicity.conf

Now you should have:

  • a password protected gpg public/private key pair

localmachine:~# gpg --list-secret-keys
sec   1024D/AE45AE12 2005-08-26 [expires: 2016-08-26]
uid                  Backup signature (localmachine) <root@localmachine>
ssb   2048g/AE45AE12 2005-08-26
  • an ssh private/public key pair

# ls -la /root/.ssh
total 16
drwx------  2 root root   54 2006-01-14 01:30 .
drwxr-xr-x 13 root root 4096 2006-01-16 07:48 ..
-rw-------  1 root root 1264 2006-01-13 21:30 id_dsa
-rw-r--r--  1 root root 1113 2006-01-13 21:30 id_dsa.pub

On the remote machine, the user used to receive the backups could be created like this:

# apt-get install scponly
# adduser --disabled-password --shell /usr/bin/scponly babar
# getent passwd babar

Example of duplicity script file

First, the script file to run the backups: /root/backup.sh

# uncomment for debug
#set -x

source /root/.duplicity.conf

# duplicity command
DUPEXEC="duplicity --encrypt-key $ENCRKEY --sign-key $SIGNKEY $DUPOPTS $*"
# loop on directories
echo -n ---- Incremental backup of $HOSTNAME ---- ;date
for i in $BACKDIRS
  echo Starting backup of directory /$i
  # create directory, then backup, then erase old backups
  # verify backup integrity
  #$DUPEXEC --verify $RPATH/$i /$i
#  if local, fix permissions
if [ -z $HOST ]; then chown -R $NAME.$NAME $LPATH; fi
echo -n ---- Finished backup on $HOSTNAME ---- ;date

Example of settings

Edit and complete the file /root/.duplicity.conf (XXXX must be replaced):

# path to backup to

## 1. remote settings
# remote host
# remote login (user for backup on server)
# send over ssh

## 2. local settings (use another disk!)
# remote host *empty*
# user name to change ownership too to
# RPATH now uses file://

# complete with root gpg signature and encryption key
# yes, we need to store the gpg pass phrase in clear somewhere

# local list of directories to backup
BACKDIRS='etc var/lib/dpkg var/log var/mail usr/local/XXX'

# full '/' backup setting
#DUPOPTS='--exclude /proc --exclude /mnt --exclude /tmp'

# duplicity options (backup.sh also accepts command line arguments)
# cleanup (really needs --force)
DUPOPTS="$DUPOPTS --remove-older-than 2W"
# exclude patterns
#DUPOPTS="$DUPOPTS --exclude **/pictures/XXX"

# load ssh agent info using keychain
[[ -f /root/.keychain/$HOSTNAME-sh ]] && \
       source /root/.keychain/$HOSTNAME-sh

export GNUPGHOME=/root/.gnupg

if [ -z $HOST ]; then
  MKDIR="mkdir -p"
  MKDIR="ssh $NAME@$HOST mkdir -p"

Testing the script

Add the following to your localmachine:/root/.bashrc

keychain --nogui --clear id_dsa
. ~/.keychain/$HOSTNAME-sh

and login as root again. You should be prompted for the ssh passphrase.

The new ssh key needs to be installed on the server (i.e. add the content of localmachine:/root/.ssh/id_dsa.pub into remotemachine:/home/babar/.ssh/authorized_keys (-rw-------) ), and the directory LPATH created.

Then try running the script:

# /root/backup.sh
---- Incremental backup of localmachine ---- Mon Jan 16 06:49:25 GMT 2006
Starting backup of directory /home/barfoo
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1137394351.70 (Mon Jan 16 06:52:31 2006)
EndTime 1137394838.14 (Mon Jan 16 07:00:38 2006)
ElapsedTime 486.44 (8 minutes 6.44 seconds)
SourceFiles 26330
SourceFileSize 677134571 (646 MB)
NewFiles 4185
NewFileSize 125189830 (119 MB)
DeletedFiles 0
ChangedFiles 1
ChangedFileSize 2190 (2.14 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4186
RawDeltaSize 99926952 (95.3 MB)
TotalDestinationSizeChange 45211649 (43.1 MB)
Errors 0

No old backup sets found, nothing deleted.
---- Finished backup on localmachine ---- Mon Jan 16 07:00:57 GMT 2006

The set -x line can be uncommented to debug the script.

Saving the keys

Now you have it working, you will need to backup the key, away from the machine to backup and from the remote backup storage. Somewhere safe, such as a usb key:

$ cd /media/usbdisk
$ sudo tar zcvf root-$HOSTNAME.tar.gz /root

You may also need to store the ssh passphrase close to it.

To make the remote machine safer, you should also set the script of the backup user to scponly.

Making it automagic

You can now run manualy the script with

$ sudo /root/backup.sh

(you will need to add the ssh key to keychain after a reboot).

To recreate a full backup, add the --full flag. To delete the old backups, use --force.

To add it to cron:

$ sudo crontab -e

and put something like:

# m h  dom mon dow command
33 23  *   *   1-6   /root/backup.sh
# full backup every sunday, deleting old ones
33 23  *   *   7     /root/backup.sh --full --force

More infos

For more, man duplicity, or read the [http://www.nongnu.org/duplicity/duplicity.1.html online manpage] at the [http://www.nongnu.org/duplicity/ duplicity webpage]