{"id":6529,"date":"2023-09-11T18:09:10","date_gmt":"2023-09-11T18:09:10","guid":{"rendered":"https:\/\/www.webhi.com\/how-to\/?p=6529"},"modified":"2023-09-11T18:09:12","modified_gmt":"2023-09-11T18:09:12","slug":"setup-use-duplicity-automatic-backups","status":"publish","type":"post","link":"https:\/\/www.webhi.com\/how-to\/setup-use-duplicity-automatic-backups\/","title":{"rendered":"How to install and use Duplicity to Automate Backups"},"content":{"rendered":"\n
\"Duplicity<\/figure>\n\n\n\n

Duplicity is a powerful open source backup tool that allows you to perform encrypted and incremental backups. It supports a variety of backends for storing backup data including local or remote file systems, FTP, SSH, WebDAV and cloud storage services. Duplicity uses GnuPG for encryption and signing of backup archives.<\/p>\n\n\n\n

In this comprehensive guide, we will cover how to install and configure Duplicity and then use it to setup automated backups on Linux.<\/p>\n\n\n\n

Installing Duplicity<\/h2>\n\n\n\n

Duplicity is available in the default repositories of most Linux distributions.<\/p>\n\n\n\n

On Debian\/Ubuntu<\/h3>\n\n\n\n
$ sudo apt update\n$ sudo apt install duplicity<\/code><\/pre>\n\n\n\n
On CentOS\/RHEL<\/h3>\n\n\n\n
$ sudo yum install epel-release\n$ sudo yum update\n$ sudo yum install duplicity<\/code><\/pre>\n\n\n\n
On Arch Linux<\/h3>\n\n\n\n
$ sudo pacman -S duplicity<\/code><\/pre>\n\n\n\n
On Fedora<\/h3>\n\n\n\n
$ sudo dnf install duplicity<\/code><\/pre>\n\n\n\n
For other Linux distributions, consult your package manager.<\/p>\n\n\n\n
Once installed, verify that Duplicity is available by checking the version:<\/p>\n\n\n\n
$ duplicity --version<\/code><\/pre>\n\n\n\n
Generating GPG Keys<\/h2>\n\n\n\n
Duplicity uses GnuPG keys to encrypt and\/or sign backup archives. We need to generate a keypair for this purpose.<\/p>\n\n\n\n
Import the GPG tools package if not already installed:<\/p>\n\n\n\n
$ sudo apt install gnupg<\/code><\/pre>\n\n\n\n
Generate a new keypair with RSA encryption.<\/p>\n\n\n\n
$ gpg --gen-key<\/code><\/pre>\n\n\n\n
Choose the key type as “RSA and RSA” with size 4096 bits. Set an expiry period if desired.<\/p>\n\n\n\n
Provide your details for user ID such as name, email etc. Add a secure passphrase for the keys. This will generate a public\/private keypair for you.<\/p>\n\n\n\n
List the keys to find the ID:<\/p>\n\n\n\n
$ gpg --list-keys<\/code><\/pre>\n\n\n\n
Export the public key for backup. Substitute the ID appropriately:<\/p>\n\n\n\n
$ gpg -a --export 1234ABCD > public.gpg<\/code><\/pre>\n\n\n\n
The public key needs to be transferred to any remote backend you intend to use such as SSH or cloud storage. The private key should be kept securely on the local system perform the backups.<\/p>\n\n\n\n
Configuring Duplicity<\/h2>\n\n\n\n
Duplicity supports a variety of storage backends such as local, SSH, FTP, WebDAV etc. We will look at configuring for some common backends:<\/p>\n\n\n\n
Local Filesystem<\/h3>\n\n\n\n
To backup to a local directory, set the backup destination URL like:<\/p>\n\n\n\n
file:\/\/\/home\/user\/backups<\/code><\/pre>\n\n\n\n
SSH<\/h3>\n\n\n\n
To backup to a remote system over SSH:<\/p>\n\n\n\n
ssh:\/\/user@host\/\/path\/to\/backup<\/code><\/pre>\n\n\n\n
This assumes you have SSH access setup between the systems.<\/p>\n\n\n\n
Amazon S3<\/h3>\n\n\n\n
To backup to an Amazon S3 bucket:<\/p>\n\n\n\n
s3:\/\/s3-bucket-name[\/prefix] <\/code><\/pre>\n\n\n\n
The S3 credentials can be supplied via a ~\/.boto config file or environment variables.<\/p>\n\n\n\n
See the Duplicity S3 Backend<\/a> documentation for details.<\/p>\n\n\n\n
Google Cloud Storage<\/h3>\n\n\n\n
For backing up to Google Cloud Storage:<\/p>\n\n\n\n
gs:\/\/cloud-storage-bucket[\/prefix]<\/code><\/pre>\n\n\n\n
Authentication can be done by various means including service account files, ADC JSON files or environmental variables.<\/p>\n\n\n\n
Refer the GCS Backend<\/a> section in the docs for more details.<\/p>\n\n\n\n
Swift<\/h3>\n\n\n\n
To backup to an OpenStack Swift container:<\/p>\n\n\n\n
swift:\/\/container_name[\/prefix]<\/code><\/pre>\n\n\n\n
Authentication is through environment variables. See the Swift Backend<\/a> documentation.<\/p>\n\n\n\n
WebDAV<\/h3>\n\n\n\n
To use a WebDAV server as backend:<\/p>\n\n\n\n
webdav[s]:\/\/hostname[:port]\/path<\/code><\/pre>\n\n\n\n
Duplicity will prompt for username\/password as required.<\/p>\n\n\n\n
See WebDAV Backend<\/a> for details.<\/p>\n\n\n\n
In this manner, you can configure backups to a variety of storage endpoints. Now we are ready to create our first backup.<\/p>\n\n\n\n
Creating Backups<\/h2>\n\n\n\n
With the backend URL configured, we can now make a full backup.<\/p>\n\n\n\n
Set the following environment variable to avoid interactive prompts:<\/p>\n\n\n\n
export PASSPHRASE=your_passphrase <\/code><\/pre>\n\n\n\n
Then run a full backup:<\/p>\n\n\n\n
$ duplicity \/path\/to\/source dir file:\/\/\/path\/to\/destination<\/code><\/pre>\n\n\n\n
This will recursively backup the \/path\/to\/source<\/code> directory to local destination \/path\/to\/destination<\/code>.<\/p>\n\n\n\n
To backup to a remote location over SSH:<\/p>\n\n\n\n
$ duplicity \/path\/to\/source ssh:\/\/user@host\/\/backup\/path<\/code><\/pre>\n\n\n\n
For backing up to cloud storage:<\/p>\n\n\n\n
$ duplicity \/local\/source s3:\/\/s3-bucket[\/prefix]<\/code><\/pre>\n\n\n\n
Duplicity will prompt for any required credentials like SSH password or AWS secret keys. The backup will be encrypted and stored in the destination.<\/p>\n\n\n\n
To backup certain folders only, specify them as include paths. For example:<\/p>\n\n\n\n
$ duplicity include \/path\/to\/folders1 include \/path\/to\/folders2 \/path\/to\/source file:\/\/\/\/path\/to\/destination<\/code><\/pre>\n\n\n\n
This will only backup the specified folders from the source.<\/p>\n\n\n\n
After the initial full backup, consecutive backups will be incremental. This saves time and storage space. Duplicity uses librsync<\/a> to efficiently determine changed content.<\/p>\n\n\n\n
To force a full backup instead of incremental, use the --full-if-older-than<\/code> option:<\/p>\n\n\n\n
$ duplicity --full-if-older-than 60D \/path\/to\/source ssh:\/\/user@host\/\/path\/to\/backup<\/code><\/pre>\n\n\n\n
This will do a full backup if last full backup is older than 60 days.<\/p>\n\n\n\n
Backup Scheduling with Cron<\/h2>\n\n\n\n
We can automate the Duplicity backups using Cron jobs.<\/p>\n\n\n\n
Open the crontab for editing:<\/p>\n\n\n\n
$ crontab -e<\/code><\/pre>\n\n\n\n
Add a cron schedule like:<\/p>\n\n\n\n
0 1 * * * \/usr\/bin\/duplicity \/path\/to\/source ssh:\/\/user@host\/\/backup\/path<\/code><\/pre>\n\n\n\n
This will run a backup job everyday at 1 AM.<\/p>\n\n\n\n
For weekly backups:<\/p>\n\n\n\n
0 1 * * 0 \/usr\/bin\/duplicity \/path\/to\/source ssh:\/\/user@host\/\/backup\/path<\/code><\/pre>\n\n\n\n
This will trigger the backup every Sunday at 1 AM.<\/p>\n\n\n\n
Similarly you can schedule monthly, yearly backups etc.<\/p>\n\n\n\n
For more granular control, you can trigger separate full and incremental jobs:<\/p>\n\n\n\n
0 1 * * * \/usr\/bin\/duplicity --full-if-older-than 30D \/path\/to\/source ssh:\/\/user@host\/\/path\/to\/full\/backup\n0 *\/4 * * * \/usr\/bin\/duplicity \/path\/to\/source ssh:\/\/user@host\/\/path\/to\/incr\/backup <\/code><\/pre>\n\n\n\n
This will run full backup on 1st of month and incremental every 4 hours.<\/p>\n\n\n\n
Remember to redirect output<\/a> if the jobs produce a lot of output to avoid cron email spam.<\/p>\n\n\n\n
Restoring Backups<\/h2>\n\n\n\n
To restore the latest backup version:<\/p>\n\n\n\n
$ duplicity restore ssh:\/\/user@host\/\/backup\/path \/local\/restore\/path<\/code><\/pre>\n\n\n\n
This will restore the backup available in the remote location to specified local path.<\/p>\n\n\n\n
To restore an earlier version from a specific date:<\/p>\n\n\n\n
$ duplicity restore --time 2020-01-01T12:30:00 ssh:\/\/user@host\/\/backup\/path \/local\/restore\/path<\/code><\/pre>\n\n\n\n
List all stored backup versions:<\/p>\n\n\n\n
$ duplicity collection-status ssh:\/\/user@host\/\/backup\/path<\/code><\/pre>\n\n\n\n
Delete old backups:<\/p>\n\n\n\n
$ duplicity remove-older-than 6M --force ssh:\/\/user@host\/\/backup\/path<\/code><\/pre>\n\n\n\n
This will delete all backup versions older than 6 months.<\/p>\n\n\n\n
In this manner you can manage your backed up archives. Restoring selected versions whenever required.<\/p>\n\n\n\n
Duplicity on Mac with Homebrew<\/h2>\n\n\n\n
On MacOS, Duplicity can be installed via Homebrew<\/a>:<\/p>\n\n\n\n
$ brew install duplicity<\/code><\/pre>\n\n\n\n
Usage remains same as on Linux:<\/p>\n\n\n\n
$ duplicity \/path\/to\/source \/path\/to\/destination<\/code><\/pre>\n\n\n\n
Schedule cron backup similarly using the native crontab:<\/p>\n\n\n\n
$ crontab -e<\/code><\/pre>\n\n\n\n
Duplicity on Windows<\/h2>\n\n\n\n
Duplicity can be installed on Windows using the Cygwin<\/a> Linux environment.<\/p>\n\n\n\n
First install Cygwin<\/a> with the rsync and python packages.<\/p>\n\n\n\n
Then install Duplicity via pip:<\/p>\n\n\n\n
C:\\> pip install duplicity<\/code><\/pre>\n\n\n\n
Now you can use Duplicity to backup files locally or to remote Windows shares:<\/p>\n\n\n\n
C:\\> duplicity C:\\Users\\user\\Documents E:\\Backups<\/code><\/pre>\n\n\n\n
Automate the scheduled backups using the Task Scheduler.<\/p>\n\n\n\n
A Windows native port of Duplicity called cwDup<\/a> is also available though with reduced functionality.<\/p>\n\n\n\n
Duplicity Best Practices<\/h2>\n\n\n\n
Here are some best practices to follow when using Duplicity:<\/p>\n\n\n\n