Backup Encryption: Chunking & Automation

TL;DR

Encrypt your backups in chunks for better security and faster restores. This guide shows you how to split files, encrypt each chunk individually, and automate the process using common tools.

1. Why Chunk & Encrypt?

Traditional backup encryption protects the entire file. Chunking breaks your data into smaller pieces before encryption. This offers several benefits:

• Faster Restores: You can restore specific chunks without decrypting the whole backup.
• Reduced Impact of Compromise: If one chunk is compromised, only that part of your data is affected.
• Parallel Processing: Encryption and decryption can happen in parallel, speeding up the process.

2. Tools You’ll Need

We’ll use readily available tools:

• Split/Concatenate: split (Linux/macOS) or 7-Zip (Windows).
• Encryption: GnuPG (GPG) is a strong, free option.
• Automation: Bash scripting (Linux/macOS) or PowerShell scripting (Windows).

3. Chunking Your Files

First, split your files into manageable chunks. A good size is 10-50MB depending on your hardware and network.

Linux/macOS

split -b 20M my_large_file chunk_prefix_

This command splits my_large_file into chunks of 20MB each, named chunk_prefix_aa, chunk_prefix_ab, and so on.

Windows (using 7-Zip)

Right-click the file in 7-Zip. Select ‘Add to archive…’. Set ‘Split to volumes, bytes’ to your desired chunk size (e.g., 20480000 for 20MB). Choose a suitable output format like .zip or .7z.

4. Encrypting Each Chunk

Encrypt each chunk individually using GPG.

Linux/macOS

for file in chunk_prefix_*; do
gpg --encrypt --recipient "your_email@example.com" "$file"
done

Replace your_email@example.com with your GPG key’s email address.

Windows (using GPG)

You can use a similar loop in PowerShell:

Get-ChildItem chunk_prefix_* | ForEach-Object { gpg --encrypt --recipient "your_email@example.com" $_.FullName }

Again, replace your_email@example.com with your GPG key’s email address.

5. Automating the Process (Bash Example)

Create a Bash script to automate chunking and encryption:

#!/bin/bash

FILE_TO_BACKUP="my_large_file"
CHUNK_SIZE="20M"
OUTPUT_PREFIX="backup_chunk_"
RECIPIENT="your_email@example.com"

# Chunk the file
split -b "$CHUNK_SIZE" "$FILE_TO_BACKUP" "$OUTPUT_PREFIX"

# Encrypt each chunk
for file in $OUTPUT_PREFIX*;
do
gpg --encrypt --recipient "$RECIPIENT" "$file"
done

echo "Backup completed!"

Save this script (e.g., backup.sh), make it executable with chmod +x backup.sh, and run it with ./backup.sh.

6. Decryption & Reassembly

To restore, decrypt the chunks first:

Linux/macOS

for file in *.gpg; do
gpg --decrypt "$file" > "${file%.gpg}"
done

This decrypts all .gpg files and removes the extension.

Windows (using GPG)

Get-ChildItem *.gpg | ForEach-Object { gpg --decrypt $_.FullName -o ($_.BaseName) }

Then, concatenate the decrypted chunks back into a single file:

Linux/macOS

cat chunk_prefix_* > restored_file

Windows (using 7-Zip)

Use 7-Zip to join the files. Select all the decrypted chunks, right-click, and choose ‘Add to archive…’. Ensure no compression is used.

7. Scheduling Backups

Use tools like cron (Linux/macOS) or Task Scheduler (Windows) to schedule your backup script to run automatically at regular intervals.