Anyone who has handled backups knows the drill: data volumes are ballooning, backup windows are shrinking, and storage costs are rising. A single full backup can easily run into the hundreds of gigabytes; running one daily would quickly exhaust disk space. Even more frustrating is the repeated transmission and storage of redundant data during the backup process, wasting bandwidth, disk space, and time.

There are two core strategies to solve this: deduplication and compression. The former eliminates redundancy, while the latter reduces size. Combining the two can boost backup efficiency several-fold.

Data Deduplication: Storing Only What Changes

The logic behind deduplication is simple: store only one copy of a piece of data, no matter how many times it appears. However, implementation occurs at two distinct levels.

File-level deduplication is the most basic form. If two files share the exact same hash value, only one copy is kept. While simple to implement, this method fails to handle scenarios involving minor internal changes—altering a single character causes the entire file to be treated as new and stored again.

Block-level deduplication is the truly efficient approach. It breaks files into data blocks—either fixed or variable in size—and calculates hashes for each block. Only blocks with changed content are stored again; the rest simply reference existing copies. Modern backup tools like Restic and Borg utilize block-level deduplication; if you back up a 10GB virtual machine image but only 100MB of content has changed, the actual additional storage required might be less than 100MB.

Another advantage of block-level deduplication is its ability to deduplicate across files. If multiple virtual machine images or database exports share system files or common data, those blocks are stored only upon their first appearance. In scenarios involving multiple similar servers, deduplication ratios can reach 5:1 or even 10:1.

Data Compression: Trading CPU for Space and Bandwidth

Deduplication solves the problem of redundant storage, while compression addresses the need for a smaller footprint. When combined, the effects are multiplicative.

Choosing a compression algorithm requires balancing compression ratio, speed, and CPU usage. Gzip is a popular, general-purpose choice offering good compatibility and a respectable compression ratio. Bzip2 achieves higher compression ratios but is frustratingly slow, making it unsuitable for frequent backups. Lz4 offers extremely high speed but a modest compression ratio, making it ideal for scenarios with strict backup window requirements.

Zstandard (zstd) is currently the most recommended option. It compresses 3–5 times faster than gzip and decompresses more than 10 times faster, all while achieving a slightly better compression ratio. Crucially, zstd supports adjustable compression levels—ranging from `-3` (prioritizing speed) to `-19` (prioritizing compression ratio)—allowing for flexibility based on your backup window.

Enabling compression in backup tools usually requires just a single parameter. Restic uses zstd compression by default, while Borg supports various algorithms such as lz4, zstd, and zlib. If you are manually archiving data with `tar`, you can use a command like this:

tar -cf - /data | zstd -T4 -3 > backup.tar.zst

Here, `-T4` enables parallel compression using four threads, and `-3` selects the speed-oriented compression level. In most scenarios, this combination can compress tens of gigabytes of data within minutes, reducing the size to 30%–50% of the original.

Practical Application: Combining Deduplication and Compression

Taking Restic as an example, a single command handles both deduplication and compression:

restic -r sftp:user@backup-server:/backup backup /data --compression max

Restic automatically chunks, deduplicates, and compresses the data before transferring it to the remote repository. During subsequent backups, only new or modified chunks are uploaded; existing chunks are referenced directly from the local cache. This results in extremely fast incremental backups with minimal bandwidth consumption.

If using Borg, the command is equally concise:

borg create --compression zstd,10 /backup::archive-{now} /data

The `--compression zstd,10` flag specifies the zstd algorithm at compression level 10. Borg’s block-level deduplication mechanism automatically handles redundant data, often resulting in a backup repository size that is only a fraction of the original data.

Hardware Fundamentals: Why Server Performance Determines Backup Limits

Both deduplication and compression are CPU-intensive operations that also place high demands on disk I/O. Block-level deduplication requires frequent reading of data chunks and hash calculations, while compression consumes significant CPU cycles. If server CPU performance is insufficient, the compression process slows down the entire backup workflow; if disk IOPS are inadequate, reading data blocks becomes a bottleneck.

Jtti’s cloud server solutions provide the robust hardware foundation required for backup scenarios. All models come standard with enterprise-grade NVMe SSDs, offering 4K random read/write IOPS that far exceed those of standard SATA SSDs, thereby accelerating read speeds and hash calculations during block-level deduplication. High-performance CPUs enable effective parallelization of multi-threaded zstd compression, shortening the backup window. Dedicated bandwidth ensures that backup data uploads to remote storage are not interrupted by timeouts caused by "noisy neighbors" competing for bandwidth. For cross-region backup needs, Jtti’s Hong Kong and US nodes connect via premium CN2 GIA lines, ensuring a smooth and stable experience when managing tasks and transferring data from within China.

A "same-price renewal" policy guarantees that the renewal cost matches the initial purchase price, allowing businesses running long-term backup operations to accurately forecast costs.

Deduplication and compression are not merely "nice-to-have" features; they are critical to the long-term viability of a backup solution. Without deduplication, backup storage requirements grow linearly; without compression, backup windows become excessively long. Combining these technologies with a high-performance server ensures that backups can be reliably executed, stored, and restored.