Backup & Recovery Architecture

Source document for the portfolio Backup section. Sanitized — no credentials, identifiers, or internal-only operational notes. Hostnames use lab.example.

---

Summary

Four independent layers protect a two-server homelab and one workstation: file-level encrypted backups with a central dashboard, weekly bare-metal system images, continuous parity across the bulk storage array, and off-site replication to a geographically separate site over a private mesh. Each layer covers a different failure mode, and no layer depends on another to function.

The design principle throughout: independence over convenience. A backup system whose components depend on each other fails as one unit.

---

Layer 1 — Borg Backup Server (daily, file-level)

A self-hosted Borg orchestration server with a web dashboard, agent management, scheduling, and alerting. Three clients enrolled.

All repositories use repokey-blake2 encryption with lz4 compression and a 7 daily / 4 weekly / 6 monthly retention policy.

How a backup runs

1. The server's scheduler dispatches a job for an enrolled client.
2. The client's agent — a continuously running native systemd service — polls

the server over HTTPS and picks up the job.

1. The agent runs borg create locally and pushes the resulting archive

over SSH to a per-client account on the backup server.

1. The agent reports completion or failure back over HTTPS.
2. The dashboard updates and a Discord notification fires through Apprise.

Append-only agents

Each client's SSH account on the server is restricted to append-only Borg operations. Clients can write new archives; they cannot delete or prune existing ones.

This matters because the clients are the things most likely to be compromised. Ransomware that encrypts a client and then reaches for its backups finds a credential that can only add data. Pruning and retention are enforced server-side, where the attacker isn't.

Server state

The entire server — MariaDB (clients, repositories, schedules), ClickHouse (file catalog), the Borg repository home, SSH host keys, and server config — lives under a single bind-mounted directory on the storage array. The container is fully reconstructable from that directory alone.

The practical payoff: if the host dies, the enclosure moves to any Linux machine, the compose stack is deployed, and the dashboard comes back with every client, repository, and schedule intact.

---

Layer 2 — ReaR (weekly, bare-metal)

Relax-and-Recover produces a bootable rescue ISO plus a full system tarball for each server, weekly.

ReaR has exactly one dependency: the backup array being writable. It shares no code, no database, and no scheduling with the Borg layer. If the Borg server is removed entirely, ReaR keeps working — and the reverse is also true.

Both rescue ISOs are also kept on a Ventoy USB, so a server with a dead OS drive can be booted and recovered without needing a working machine to first retrieve the ISO from.

Two-stage recovery

Bare-metal restore and file restore solve different problems and are used together. ReaR returns the machine to its weekly image — bootable, with partitions, bootloader, packages, and configuration in place. That image can be up to seven days stale, so the Borg layer then restores the most recent daily file state on top.

---

Layer 3 — SnapRAID (parity, storage array)

The bulk storage pool is a MergerFS union across three data disks with a fourth disk holding SnapRAID parity.

This is not a backup and isn't treated as one — parity is computed on a schedule, so a file created and deleted between syncs was never protected. What it does cover is drive failure: any single disk in the pool can die and be rebuilt from parity, without restoring several terabytes across the network.

Because MergerFS keeps whole files on individual disks rather than striping, losing a disk beyond parity coverage loses only the files that lived on it — not the entire array.

Parity sizing constraint

Parity must be greater than or equal to the largest data drive. All four disks are the same size, so the array sits at its capacity limit: adding a larger data disk requires role-swapping parity onto the new disk and rebuilding parity first. This is a design constraint of the architecture, not a defect, and it shapes the upgrade path.

---

Monitoring and alerting

Three signals, deliberately not routed through a single service:

The Healthchecks instance exists for a specific reason. The Borg server reports its own failures, but a job that never starts produces no failure to report. Each ReaR unit pings a check at start and again at exit with its status code, so a run that silently stops firing is caught by the absence of a ping rather than the presence of an error.

Health pings are explicitly non-fatal in the systemd units. An earlier configuration let a failed ping abort the backup before it ran, which meant a monitoring outage produced backup outages — the monitoring became a dependency of the thing it monitored.

---

What is deliberately not backed up

Naming exclusions is part of the design; an unbounded backup is one nobody verifies.

• Game installs. Re-downloadable from their platforms. Saves and configs in

the user home are backed up.

• Regenerable Docker volumes. Local LLM model data runs to tens of

gigabytes and is restored with a single pull command.

• Search index translogs. Excluded because they rotate mid-snapshot and

produce inconsistent captures.

---

Recovery paths

The last two are the ones worth noting. The backup server is orchestration and convenience; the archives underneath it are a documented open format that can be read with the standard Borg CLI and nothing else. No proprietary layer sits between the data and recovery.

---

Layer 4 — Off-site replication

Borg pushes a second copy of the archives over a Tailscale link to a machine at a geographically separate location, backed by two 6 TB drives in a RAID 1 mirror — the same enclosure topology as the primary array.

Two properties make this tier worth having rather than just another copy:

The transport is the mesh, not the internet. The remote host has no published service, no forwarded port, and no public DNS record. It is reachable only as a peer on the private mesh, so the off-site copy adds no attack surface at either end. There is nothing to find by scanning.

It is the same format, not a different product. The remote target holds standard Borg repositories, identical to the primary. Recovery from off-site uses exactly the same commands as recovery from the local array — no separate tooling, no vendor-specific restore path, and no second procedure that only gets exercised during an actual disaster.

Combined with the RAID 1 mirror on the remote side, a full copy survives the loss of a single drive in either location, or the loss of an entire site.

---

What this covers

The design goal was that no single failure — including compromise of a client, loss of the backup host, or loss of the building — leaves the environment unrecoverable.