Differences between revisions 1 and 20 (spanning 19 versions)
Revision 1 as of 2006-01-16 07:41:03
Size: 5233
Editor: PaulBrossier
Comment: created new page to setup duplicity
Revision 20 as of 2019-11-24 06:51:02
Size: 8035
Editor: AlfonsRovira
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
[http://packages.debian.org/duplicity duplicity] is a very useful tool to make remote unassisted backups. The backups can be incremental, encrypted, and over a variety of transports. Here is the example of setup to backup various directories remotely. Other tools such as [http://packages.debian.org/backupninja backupninja] can be used for the same purpose. #language en
~- [[fr/DebianWiki/EditorGuide#traduction|Traduction(s)]] : English - [[fr/Duplicity|Francais]] - - [[ca/Duplicity|Català]]-~

[[http://packages.debian.org/duplicity|duplicity]] is a very useful tool to make remote unassisted backups. The backups can be incremental, encrypted, and over a variety of transports. Here is the example of setup to backup various directories remotely. Other tools such as [[http://packages.debian.org/backupninja|backupninja]] can be used for the same purpose.

<<TableOfContents(1)>>

It is convenient to split the backups per directory, such as {{{/etc}}}, {{{/var/lib/dpkg}}} or {{{/var/spool}}}. This script will create small backups, one per directory defined later in BACKDIRS. The remote machine will contain the following directory tree in LPATH:

{{{
localmachine/etc/
localmachine/var/lib/dpkg
localmachine/var/spool
...
}}}

and the commands we will run from localmachine will look like this:
{{{
duplicity --encrypt-key AE45AE12 --sign-key AE45AE12 \
  remove-older-than 2W /var/spool \
  scp://babar@remotehostname//home/babar/duplic/localmachine/var/spool
}}}
and then to remove the old backups:
{{{
duplicity --encrypt-key AE45AE12 --sign-key AE45AE12 \
  remove-older-than 2W \
  scp://babar@remotehostname//home/babar/duplic/localmachine/var/spool
}}}

You need first to know what directories to backup on your machine. A bare minimum to reconstruct the installation could be {{{/etc}}} and {{{/var/lib/dpkg}}}, then your mileage may vary. Think about twice, double check the size it takes, and the size you have one the server. How do the size vary along time? how long do you want to keep the old backups?

Here we keep 2 weeks of old backups, with a full backup every week and a daily increment.

= Create the ssh and gpg keys =
Line 13: Line 46:
# chmod u+x /root/backup.sh
Line 18: Line 52:
= Example of duplicity script file /root/backup.sh = Now you should have:

 * a password protected gpg public/private key pair
{{{
localmachine:~# gpg --list-secret-keys
/root/.gnupg/secring.gpg
------------------------
sec 1024D/AE45AE12 2005-08-26 [expires: 2016-08-26]
uid Backup signature (localmachine) <root@localmachine>
ssb 2048g/AE45AE12 2005-08-26
}}}
 * an ssh private/public key pair
{{{
# ls -la /root/.ssh
total 16
drwx------ 2 root root 54 2006-01-14 01:30 .
drwxr-xr-x 13 root root 4096 2006-01-16 07:48 ..
-rw------- 1 root root 1264 2006-01-13 21:30 id_dsa
-rw-r--r-- 1 root root 1113 2006-01-13 21:30 id_dsa.pub
}}}

On the remote machine, the user used to receive the backups could be created like this:
{{{
# apt-get install scponly
# adduser --disabled-password --shell /usr/bin/scponly babar
# getent passwd babar
babar:x:1002:1002:,,,:/home/babar:/usr/bin/scponly
}}}

= Example of duplicity script file =

First, the script file to run the backups: /root/backup.sh
Line 25: Line 90:
source /root/.duplicity.conf . /root/.duplicity.conf
Line 30: Line 95:
echo -n ---- Incremental backup of $HOSTNAME ---- ;date echo -n "---- Incremental backup of $HOSTNAME ---- "; date
Line 35: Line 100:
  $MKDIR $LPATH/$i && $DUPEXEC /$i $RPATH/$i && $DUPEXEC $RPATH/$i   $MKDIR $LPATH/$i && $DUPEXEC /$i $RPATH/$i && $DUPEXEC $DUPOPTS_CLEANUP $RPATH/$i
Line 41: Line 106:
echo -n ---- Finished backup on $HOSTNAME ---- ;date
}}}

= Example of settings file /root/.duplicity.conf =
echo -n "---- Finished backup on $HOSTNAME ---- "; date
}}}

= Example of settings =

Edit and complete the
file /root/.duplicity.conf (XXXX must be replaced):
Line 50: Line 117:
LPATH=/home/babar/precious/$HOSTNAME LPATH=/home/babar/XXX/$HOSTNAME
Line 54: Line 121:
HOST=remotehostname HOST=remotehostnameXXX
Line 56: Line 123:
NAME=babar NAME=babarXXX
Line 60: Line 127:
## 2. local settings ## 2. local settings (use another disk!)
Line 75: Line 142:
BACKDIRS='etc var/lib/dpkg var/log var/mail usr/loca/more/XXX
opt/another/dirXXXXXXX
'
BACKDIRS='etc var/lib/dpkg var/log var/mail usr/local/XXX'
Line 85: Line 151:
DUPOPTS='--remove-older-than 2W' DUPOPTS_CLEANUP="remove-older-than 2W"
Line 87: Line 153:
DUPOPTS='$DUPOPTS --exclude **/pictures/XXX' #DUPOPTS="$DUPOPTS --exclude **/pictures/XXX"
Line 98: Line 164:
  MKDIR="ssh $HOST mkdir -p"   MKDIR="ssh $NAME@$HOST mkdir -p"
Line 102: Line 168:
= Testing the whole thing = = Testing the script =
Line 111: Line 177:
and login as root again. You should be prompted for the ssh passphrase. 

Once you have also added the content of {{{localmachine:/root/.ssh/id_dsa.pub}}} into {{{remotemachine:/home/babar/.ssh/authorized_keys (-rw-------)}}}, try running the script:
and login as root again. You should be prompted for the ssh passphrase.

The new ssh key needs to be installed on the server (i.e. add the content of {{{localmachine:/root/.ssh/id_dsa.pub}}} into {{{remotemachine:/home/babar/.ssh/authorized_keys (-rw-------)}}} ), and the directory LPATH created.

Then try
running the script:
Line 153: Line 221:
To make the remote machine safer, you could also set the script of the backup user to scponly. You may also need to store the ssh passphrase close to it.

To make the remote machine safer, you should also set the script of the backup user to scponly.
Line 157: Line 227:
You can now run manualy the script with  You can now run manualy the script with
Line 174: Line 244:
33 23 * * 7 /root/backup.sh --full --force 33 23 * * 7   /root/backup.sh --full --force
Line 179: Line 249:
For more, {{{man duplicity}}}, or read the [http://www.nongnu.org/duplicity/duplicity.1.html online manpage] at the [http://www.nongnu.org/duplicity/ duplicity webpage] For more, {{{man duplicity}}}, or read the [[http://www.nongnu.org/duplicity/duplicity.1.html|online manpage]] at the [[http://www.nongnu.org/duplicity/|duplicity webpage]]

See also the [[http://packages.debian.org/duply|duply]] package maybe.

----

CategorySoftware | CategorySystemAdministration | CategoryBackup

Traduction(s) : English - Francais - - Català

duplicity is a very useful tool to make remote unassisted backups. The backups can be incremental, encrypted, and over a variety of transports. Here is the example of setup to backup various directories remotely. Other tools such as backupninja can be used for the same purpose.

It is convenient to split the backups per directory, such as /etc, /var/lib/dpkg or /var/spool. This script will create small backups, one per directory defined later in BACKDIRS. The remote machine will contain the following directory tree in LPATH:

localmachine/etc/
localmachine/var/lib/dpkg
localmachine/var/spool
...

and the commands we will run from localmachine will look like this:

duplicity --encrypt-key AE45AE12 --sign-key AE45AE12 \
  remove-older-than 2W /var/spool \
  scp://babar@remotehostname//home/babar/duplic/localmachine/var/spool

and then to remove the old backups:

duplicity --encrypt-key AE45AE12 --sign-key AE45AE12 \
  remove-older-than 2W \
  scp://babar@remotehostname//home/babar/duplic/localmachine/var/spool

You need first to know what directories to backup on your machine. A bare minimum to reconstruct the installation could be /etc and /var/lib/dpkg, then your mileage may vary. Think about twice, double check the size it takes, and the size you have one the server. How do the size vary along time? how long do you want to keep the old backups?

Here we keep 2 weeks of old backups, with a full backup every week and a daily increment.

Create the ssh and gpg keys

$ su
# apt-get install duplicity keychain
 [...]
# ssh-keygen -t dsa
 [...]
# gpg --gen-key
 [...]
# umask 077
# touch /root/backup.sh /root/.duplicity.conf
# chmod u+x /root/backup.sh
# ls -la /root/backup.sh /root/.duplicity.conf
-rwx------  1 root root 0 2006-01-16 06:47 /root/backup.sh
-rw-------  1 root root 0 2006-01-16 06:47 /root/.duplicity.conf

Now you should have:

  • a password protected gpg public/private key pair

localmachine:~# gpg --list-secret-keys
/root/.gnupg/secring.gpg
------------------------
sec   1024D/AE45AE12 2005-08-26 [expires: 2016-08-26]
uid                  Backup signature (localmachine) <root@localmachine>
ssb   2048g/AE45AE12 2005-08-26
  • an ssh private/public key pair

# ls -la /root/.ssh
total 16
drwx------  2 root root   54 2006-01-14 01:30 .
drwxr-xr-x 13 root root 4096 2006-01-16 07:48 ..
-rw-------  1 root root 1264 2006-01-13 21:30 id_dsa
-rw-r--r--  1 root root 1113 2006-01-13 21:30 id_dsa.pub

On the remote machine, the user used to receive the backups could be created like this:

# apt-get install scponly
# adduser --disabled-password --shell /usr/bin/scponly babar
# getent passwd babar
babar:x:1002:1002:,,,:/home/babar:/usr/bin/scponly

Example of duplicity script file

First, the script file to run the backups: /root/backup.sh

# uncomment for debug
#set -x

. /root/.duplicity.conf

# duplicity command
DUPEXEC="duplicity --encrypt-key $ENCRKEY --sign-key $SIGNKEY $DUPOPTS $*"
# loop on directories
echo -n "---- Incremental backup of $HOSTNAME ---- "; date
for i in $BACKDIRS
do
  echo Starting backup of directory /$i
  # create directory, then backup, then erase old backups
  $MKDIR $LPATH/$i && $DUPEXEC /$i $RPATH/$i && $DUPEXEC $DUPOPTS_CLEANUP $RPATH/$i
  # verify backup integrity
  #$DUPEXEC --verify $RPATH/$i /$i
done
#  if local, fix permissions
if [ -z $HOST ]; then chown -R $NAME.$NAME $LPATH; fi
echo -n "---- Finished backup on $HOSTNAME ---- "; date

Example of settings

Edit and complete the file /root/.duplicity.conf (XXXX must be replaced):

# path to backup to
LPATH=/home/babar/XXX/$HOSTNAME

## 1. remote settings
# remote host
HOST=remotehostnameXXX
# remote login (user for backup on server)
NAME=babarXXX
# send over ssh
RPATH=scp://$NAME@$HOST/$LPATH

## 2. local settings (use another disk!)
# remote host *empty*
#HOST=
# user name to change ownership too to
#NAME=babar
# RPATH now uses file://
#RPATH=file://$LPATH

# complete with root gpg signature and encryption key
SIGNKEY=XXXXXXXX
ENCRKEY=$SIGNKEY
# yes, we need to store the gpg pass phrase in clear somewhere
export PASSPHRASE='XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

# local list of directories to backup
BACKDIRS='etc var/lib/dpkg var/log var/mail usr/local/XXX'

# full '/' backup setting
#BACKDIRS=/
#DUPOPTS='--exclude /proc --exclude /mnt --exclude /tmp'

# duplicity options (backup.sh also accepts command line arguments)
DUPOPTS=
# cleanup (really needs --force)
DUPOPTS_CLEANUP="remove-older-than 2W"
# exclude patterns
#DUPOPTS="$DUPOPTS --exclude **/pictures/XXX"

# load ssh agent info using keychain
[[ -f /root/.keychain/$HOSTNAME-sh ]] && \
       source /root/.keychain/$HOSTNAME-sh

export GNUPGHOME=/root/.gnupg

if [ -z $HOST ]; then
  MKDIR="mkdir -p"
else
  MKDIR="ssh $NAME@$HOST mkdir -p"
fi

Testing the script

Add the following to your localmachine:/root/.bashrc

keychain --nogui --clear id_dsa
. ~/.keychain/$HOSTNAME-sh

and login as root again. You should be prompted for the ssh passphrase.

The new ssh key needs to be installed on the server (i.e. add the content of localmachine:/root/.ssh/id_dsa.pub into remotemachine:/home/babar/.ssh/authorized_keys (-rw-------) ), and the directory LPATH created.

Then try running the script:

# /root/backup.sh
---- Incremental backup of localmachine ---- Mon Jan 16 06:49:25 GMT 2006
Starting backup of directory /home/barfoo
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1137394351.70 (Mon Jan 16 06:52:31 2006)
EndTime 1137394838.14 (Mon Jan 16 07:00:38 2006)
ElapsedTime 486.44 (8 minutes 6.44 seconds)
SourceFiles 26330
SourceFileSize 677134571 (646 MB)
NewFiles 4185
NewFileSize 125189830 (119 MB)
DeletedFiles 0
ChangedFiles 1
ChangedFileSize 2190 (2.14 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4186
RawDeltaSize 99926952 (95.3 MB)
TotalDestinationSizeChange 45211649 (43.1 MB)
Errors 0
-------------------------------------------------

No old backup sets found, nothing deleted.
---- Finished backup on localmachine ---- Mon Jan 16 07:00:57 GMT 2006

The set -x line can be uncommented to debug the script.

Saving the keys

Now you have it working, you will need to backup the key, away from the machine to backup and from the remote backup storage. Somewhere safe, such as a usb key:

$ cd /media/usbdisk
$ sudo tar zcvf root-$HOSTNAME.tar.gz /root

You may also need to store the ssh passphrase close to it.

To make the remote machine safer, you should also set the script of the backup user to scponly.

Making it automagic

You can now run manualy the script with

$ sudo /root/backup.sh

(you will need to add the ssh key to keychain after a reboot).

To recreate a full backup, add the --full flag. To delete the old backups, use --force.

To add it to cron:

$ sudo crontab -e

and put something like:

# m h  dom mon dow command
33 23  *   *   1-6   /root/backup.sh
# full backup every sunday, deleting old ones
33 23  *   *   7     /root/backup.sh --full --force

More infos

For more, man duplicity, or read the online manpage at the duplicity webpage

See also the duply package maybe.


CategorySoftware | CategorySystemAdministration | ?CategoryBackup