home-serv — Infrastructure Reference

Primary storage and services host. Runs the container fleet, the AI/ML stack, the bulk storage array, and serves as the backup target for the environment.

Host

Kernel tuning

Set permanently via /etc/sysctl.d/:

• vm.swappiness=10
• vm.max_map_count=262144

Docker daemon

/etc/docker/daemon.json:

{
  "dns": ["1.1.1.1", "8.8.8.8"],
  "bip": "172.19.0.1/24",
  "default-address-pools": [
    { "base": "172.20.0.0/16", "size": 24 },
    { "base": "172.21.0.0/16", "size": 24 },
    { "base": "172.22.0.0/16", "size": 24 }
  ]
}

One external Docker network is created manually outside any stack:

docker network create ai-net

ai-net carries qdrant, librechat-rag-api, and librechat.

Storage architecture

Backup array

Contents:

• /mnt/backup/bbs/ — all Borg Backup Server state in a single bind mount: MariaDB, ClickHouse, Borg repositories, SSH host keys, server config. The container is fully reconstructable from this directory alone.
• /mnt/backup/rear/{home-serv,sec-serv}/ — bootable rescue ISO plus system tarball per server. Independent of BBS and survives its teardown. home-serv writes locally with BACKUP=NETFS; sec-serv pushes here over rsync-via-SSH as root.

Bulk storage pool — MergerFS + SnapRAID

Three data disks and one parity disk, all 6 TB Seagate IronWolf, ext4. Pool mounts at /mnt/media, ~17 TB usable.

Device letters are not stable across reboots. Always resolve by UUID or label.

MergerFS fstab entry:

/mnt/disk1:/mnt/disk2:/mnt/disk3 /mnt/media mergerfs \
  defaults,allow_other,use_ino,category.create=mfs,minfreespace=50G,\
  x-systemd.requires-mounts-for=/mnt/disk1,\
  x-systemd.requires-mounts-for=/mnt/disk2,\
  x-systemd.requires-mounts-for=/mnt/disk3 0 0

x-systemd.requires-mounts-for guarantees every branch is mounted before MergerFS starts. Without it, a slow-enumerating USB drive produces a pool silently missing a disk.

All four pool entries carry nofail. Every disk on this host is USB-attached — without nofail, a single failed enumeration drops a headless machine to an emergency console.

Branch management. MergerFS 2.33.3. Branches change live through the xattr control interface; mount -o remount does not work for branch changes and fails with fuse: mountpoint is not empty.

sudo getfattr -n user.mergerfs.branches /mnt/media/.mergerfs
sudo setfattr -n user.mergerfs.branches -v '+>/mnt/diskN=RW' /mnt/media/.mergerfs

SnapRAID. Config at /etc/snapraid.conf. Parity file at /mnt/parity1/snapraid.parity; data disks d1/d2/d3; content file mirrored to all three data drives. Single-drive failure is recoverable with snapraid fix. Two simultaneous drive failures exceed single-parity coverage.

Parity constraint. Parity must be greater than or equal to the largest data drive. All four disks are 6 TB, so the array sits at its limit — adding a data disk larger than 6 TB requires role-swapping parity onto the new disk and rebuilding parity first.

USB attachment

Both /mnt/backup RAID 1 members and all four pool drives sit behind ASMedia ASM1153E SATA bridges (174c:55aa) on Genesys Logic USB 3.1 hubs. There are no internally-attached spinning disks on this host. The four pool drives share a single hub at 4-1.4.

Stability mitigations live in the GRUB cmdline rather than a kernel package, so they survive kernel upgrades automatically:

usbcore.autosuspend=-1 usb-storage.quirks=174c:55aa:u

usb-storage.quirks matches by VID:PID, so any new drive on the same bridge chipset inherits the fix without a cmdline change.

Verify topology after adding a drive:

for d in sdc sdd sde sdf; do
  echo -n "$d: "
  readlink -f /sys/block/$d | grep -oE 'usb[0-9]+(/[0-9.-]+)+' | head -1
done

Network

Traffic flow

flowchart LR
    User[Browser or mobile]
    CF[Cloudflare edge]
    CFD[cloudflared daemon]
    SVC[home-serv services]

    User -->|HTTPS 443| CF
    CF -->|outbound QUIC tunnel| CFD
    CFD --> SVC

External access is inbound-free. Cloudflared holds an outbound QUIC tunnel to the Cloudflare edge; no ports are forwarded at the perimeter.

Mesh

home-serv is a member of a four-machine Tailscale mesh alongside sec-serv, jeremy-pc, and jeremy-lt. Other LAN hosts: 10.0.4.110 (sec-serv), 10.0.4.201 (cyberlab).

Exposure tiers

Services

Notes, knowledge and reading

Files and documents

AI and ML

LibreChat stack — /home/jeremy/docker/stacks/librechat/

Configured through librechat.yaml v1.3.9 with agents enabled, MCP domain allowlisting, and SearXNG-backed web search. Two MCP tools are exposed: search_knowledge_base against Qdrant, and web_search against the local SearXNG instance. Image generation routes through a Stable Diffusion agent to SD.Next.

RAG knowledge base — /home/jeremy/docker/stacks/rag-ingest/

SD.Next — native systemd, install at /home/jeremy/docker/stacks/sdnext/. IPEX backend on the Arc A770.

Security

Backup

BBS is the primary backup system, providing a web UI, agent management, and Discord notifications through Apprise. Three clients are enrolled: home-serv, sec-serv, and jeremy-pc. All state lives on the /mnt/backup/bbs/ bind mount.

ReaR produces weekly bare-metal recovery images — a bootable rescue ISO plus a system tarball — written to /mnt/backup/rear/home-serv/. It runs entirely independently of BBS.

Healthchecks tracks run status for the ReaR jobs on both servers.

Finance, utilities and dashboards

Cloudflare tunnel routes

Catch-all rule: http_status:404.

ntfy.lab.example requires disableChunkedEncoding: true.

Host port map

Compose projects

Stacks live under /home/jeremy/docker/stacks/, managed through Dockge.

actual/               anki/                 anythingllm/
bbs/                  beszel/               blinko/
bookstack/            changedetection/      convertx/
dockge/               freshrss/             games/
gitea/                healthchecks/         inkspire/
karakeep/             librechat/            nextcloud-aio/
nexus-pulse/          ntfy/                 ollama-intel/
omnitools/            paperless-ngx/        qdrant/
rag-ingest/           reactive-resume/      sdnext/
searxng/              silverbullet/         tamari/
viperdash/            watchtower/

sdnext/ and rag-ingest/ hold installs and scripts for services that run as native systemd units rather than compose stacks.

Native systemd services

Scheduled tasks

rear-backup.timer sets Persistent=true, so a reboot inside the Sunday window triggers a catch-up run.