Encrypting Large Files in Python

TL;DR

Use a library like cryptography with AES encryption in CBC mode for strong security and good performance when encrypting large files (around 1GB). Break the file into chunks to manage memory usage. Use a randomly generated key and store it securely – never hardcode it!

Encrypting Large Files Programmatically

This guide shows you how to encrypt a ~1GB file using Python. We’ll focus on AES encryption, which is widely considered secure.

Step 1: Install the cryptography Library

First, you need to install the cryptography library. Open your terminal or command prompt and run:

pip install cryptography

Step 2: Import Necessary Modules

In your Python script, import the required modules from the cryptography library.

from cryptography.fernet import Fernet
from cryptography.hazmat.primitives.ciphers import AES, modes
import os

Step 3: Generate a Key

Generate a strong, random encryption key. Important: Store this key securely! Losing the key means you can’t decrypt your data.

key = Fernet.generate_key()
with open('secret.key', 'wb') as key_file:
    key_file.write(key)

Step 4: Load the Key

If you already have a key, load it from its file.

with open('secret.key', 'rb') as key_file:
    key = key_file.read()

Step 5: Encryption Function

Create a function to encrypt the file in chunks. This is important for large files to avoid loading everything into memory at once.

def encrypt_file(input_filepath, output_filepath):
    f = Fernet(key)
    filesize = os.path.getsize(input_filepath)
    chunksize = 64 * 1024  # 64KB chunks

    with open(input_filepath, 'rb') as infile:
        with open(output_filepath, 'wb') as outfile:
            while True:
                chunk = infile.read(chunksize)
                if not chunk:
                    break
                encrypted_data = f.encrypt(chunk)
                outfile.write(encrypted_data)

Step 6: Decryption Function (for completeness)

Create a function to decrypt the file, mirroring the encryption process.

def decrypt_file(input_filepath, output_filepath):
    f = Fernet(key)
    filesize = os.path.getsize(input_filepath)
    chunksize = 64 * 1024  # 64KB chunks

    with open(input_filepath, 'rb') as infile:
        with open(output_filepath, 'wb') as outfile:
            while True:
                chunk = infile.read(chunksize)
                if not chunk:
                    break
                decrypted_data = f.decrypt(chunk)
                outfile.write(decrypted_data)

Step 7: Example Usage

Call the encryption function with your input and output filepaths.

input_file = 'large_file.dat'
output_file = 'encrypted_file.dat'
encrypt_file(input_file, output_file)

Step 8: Security Considerations

• Key Management: The most important part! Securely store the key. Consider using a dedicated key management system if dealing with sensitive data.
• Initialization Vector (IV): Fernet handles this automatically, which is good for simplicity and security.
• Error Handling: Add error handling to your code to gracefully manage potential issues like file not found or incorrect keys.
• cyber security best practices: Regularly update the cryptography library to benefit from the latest security patches.