Anok Discourse

Encrypted backup of site and MySQL with Duplicity on Ubuntu

Web Development
1

I wanted to make a quick backup of the blog, the files in /var/www/html and the MySQL database dump to another server. This is the process I followed on Ubuntu but works on any system.

Install Duplicity.

I found on my cloud providers image I had to also install python-paramiko (YMMV). Obviously other OS have different package managers, or you can install from source

sudo apt-get install duplicity python-paramiko

Secure the transport  between servers

Now generate a key pair, pressing enter at the prompt asking for a password. You want a password-less SSH key.

ssh-keygen -t rsa

Copy the key across to the server you will be backing up to (obviously use your own account name, not mine :)

ssh-copy-id user@thebackupserver.org

Check that you can now login to the backup server without a password, and create your backup directory while there, then exit.

ssh user@thebackupserver.org
mkdir ~/backups/domain.org
exit

Setup the file encryption

For the encryption we are using GPG, and the following commands will store our keys in /root/.gnupg/

The defaults are fine, with your email address. You want a password for your backups here. Where the ssh key pair protects the connection between your server and the backup server, this password will encrypt your data, and you will need it to decrypt your backups.

gpg --gen-key
gpg (GnuPG) 2.2.12; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Your Name
Email address: user@example.com
You selected this USER-ID:
    "Your Name "

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O

You will be setting up a passphrase to use with GPG, and then it will generate some randimosity.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 15B2496D4CBB2495 marked as ultimately trusted
gpg: directory '/root/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/15B2496D4CBB249E815ABB2496DA51423AC095CB.rev'
public and secret key created and signed.
pub   rsa3072 2019-05-21 [SC] [expires: 2021-05-20]
      15B2496D4CBB249E815ABB2496DA51423AC095CB
uid                      Your Name 
sub   rsa3072 2019-05-21 [E] [expires: 2021-05-20]

You can then view your keys with

gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2021-05-20
/root/.gnupg/pubring.kbx
------------------------
pub   rsa3072 2019-05-21 [SC] [expires: 2021-05-20]
      15B2496D4CBB249E815ABB2496DA51423AC095CB
uid           [ultimate] Your Name 
sub   rsa3072 2019-05-21 [E] [expires: 2021-05-20]

We will put this passphrase for encrypting in a safe file.

echo 'PASSPHRASE="thepassphraseyoumade"' > .passphrase
chmod 700 .passphrase

Setup MySQL

We want to run MySQL commands without entering password, so we store the user and password in the ~/.my.cnf configuration file.

[mysql]
user=user
password=password

Set the permissions so only owner can read or write, and check that it works

chmod 0600 .my.cnf
mysql -u user

The Script

Create a script to run daily

vim /etc/cron.daily/duplicity.inc

#!/bin/sh
DATABASENAME="your_database_name"
BACKUPSERVER="user@backupserver"
DOMAINNAME="nomasters.org"mysqldump --single-transaction --routines --events --triggers --add-drop-table --extended-insert -h 127.0.0.1  $DATABASENAME | gzip -9 > /var/www/backups/$DATABASENAME_$(date +"%Y_week_%U").sql.gz
test -x $(which duplicity) || exit 0
. /root/.passphrase
export PASSPHRASE
$(which duplicity) --encrypt-key 15B2496D4CBB249E815ABB2496DA51423AC095CB /var/www sftp://$BACKUPSERVER/backups/$DOMAINNAME

chmod 755 duplicity.inc

Create a script to run weekly

vim /etc/cron.daily/duplicity.full

#!/bin/sh
BACKUPSERVER="user@backupserver"
DOMAINNAME="nomasters.org"
test -x $(which duplicity) || exit 0
. /root/.passphrase
export PASSPHRASE
#clean out old backup files. keep 2 full backups and their incrementals
$(which duplicity) remove-all-but-n-full 2 --force sftp://$BACKUPSERVER/backups/$DOMAINNAME

chmod 755 duplicity.full

Viola! To restore fully (You will need to unpack your db and import to MySQL however):

PASSPHRASE="XXXXXXXX" duplicity sftp://user@backupserver/backups/$DOMAINNAME /var/www

And it will suck all those files back onto your server ...or you can just grab a file that you may need with

PASSPHRASE="XXXXXXXX" duplicity --file-to-restore /path/to/file sftp://user@backupserver/backups/$DOMAINNAME /path/to/restore/file
3

You can try also rclone, it is very powerful. Has built-in zero-knowledge encryption, even can mount remote encrypted disks.

1 Like
4

They are very different both with pros and cons. rclone is a synchronisation tool not really for backups.

For sure use it if you are only backing up a small amount of data and it works for you, but from my understanding duplicity is orders of magnitude faster with large backups, including saving a lot of space still allowing snapshot restores. rclone just would not be able to do it without massive time, and massive storage for those files. This is because rclone syncs files while duplicity does incremental backups.

Just something to keep in mind I guess.

5

Sure, rclone even does not have the rotation I remember.
I use rclone to mirror a large amount of data (1TB+) between clouds, of course such volume is not easy and cheap to version and treat as backup.

1 Like