On this page
Most homelab data loss doesn't start with a crash. It starts weeks earlier, quietly: a pool that went DEGRADED and kept serving files, a drive that began reallocating sectors, a UPS that has been running on battery since the storm. None of it shows up on a CPU or disk-usage graph, and all of it is cheap to watch once you know which numbers matter.
The short version
- ZFS: alert on any pool state other than ONLINE, on rising error counters, and on scrubs that stop completing.
- SMART: watch reallocated, pending and uncorrectable sectors on HDDs, and endurance and media errors on NVMe. Rising counts matter more than totals.
- UPS: alert on battery, low battery, and on the UPS going unreachable, because that's when shutdown scripts fail.
- Keep raw-disk access out of anything network-facing: read SMART with a root helper, not a privileged agent.
- A failing-disk alert should never clear on its own. Dismiss it when the drive is replaced.
Why usage graphs aren't enough
CPU, memory and disk usage tell you how busy a machine is today. They say nothing about whether its storage is about to fail. ZFS will keep serving data from a degraded mirror, a drive with a growing list of bad sectors still reads fine most of the time, and a server on UPS battery looks exactly like a server on mains power, right up until the battery runs out. The failures that cost you data are the ones that look healthy from the outside.
What to watch on a ZFS pool
Pool health
A healthy pool reports ONLINE. Anything else (DEGRADED, FAULTED, UNAVAIL, and so on) means a device is missing or failing, and your redundancy is already spent. This is the alert that matters most, because the next failure in a degraded vdev is the one that loses data. Check it by hand with:
zpool status -xError counters
zpool status shows read, write and checksum error counts per device. A pool can stay ONLINE while these climb: ZFS repairs what it can from redundancy and carries on. A counter that goes up is worth an alert even when the pool looks fine, because it's often the first sign of a failing drive, a bad cable or a flaky controller.
Scrubs
A scrub reads every block and checks it against its checksum, which is how ZFS finds silent corruption before you need the data. Monthly is a common rhythm for a home pool. Many Debian-based systems, Proxmox included, ship a scheduled scrub with the ZFS packages, but don't assume it's running: alert when no scrub has completed in your chosen window.
What to watch with SMART
SMART is the drive's own health report. You can read it with smartctl -a /dev/sda (from smartmontools). Most of what it prints is noise; a handful of numbers are worth alerting on.
| Drive type | Watch | Why |
|---|---|---|
| HDD | Reallocated sector count | Sectors the drive has already given up on |
| HDD | Current pending sector count | Sectors it couldn't read and may give up on |
| HDD | Offline uncorrectable | Sectors that failed even on retry |
| NVMe | Percentage used | How much of the rated write endurance is spent |
| NVMe | Media errors | Data integrity errors the controller couldn't correct |
| Any | Overall verdict | PASSED or FAILED: the drive's own last word |
The trend is what matters. A drive that has had 8 reallocated sectors for three years is a different story from one that went from 0 to 8 this week. And don't wait for the overall verdict: by the time a drive reports FAILED, it has usually been telling you for a while.
SMART is an early warning, not a guarantee
What to watch on a UPS
Network UPS Tools (NUT) talks to most UPS models and reports their status with short codes: OL (on line power), OB (on battery) and LB (low battery). Check yours with:
upsc myups@localhost- On battery tells you the power is out, often before anything else does. Allow a short grace period so a brief flicker doesn't page you.
- Low battery means shutdown is minutes away. It should alert at once, with no grace period.
- Unreachable is the sneaky one: if the monitoring host can't talk to the UPS, your automatic shutdown can't either.
Setting it up with Ensju
Ensju's server agent (0.6.0 and newer) has three optional collectors, one each for ZFS, SMART and UPS. All three are off by default, because each reads something the agent has no business touching unless you asked. Tick them in the Reinstall / Update Agent dialog on a server's page, or add them to the install command:
curl -sSL https://ensju.com/agent/install.sh | sudo ENSJU_TOKEN=ensrv_xxxxxxxx \
ENSJU_ZFS=true ENSJU_SMART=true ENSJU_UPS=true sh- ZFS needs
zpoolon the path, and no extra privilege. - SMART needs smartmontools 7.0 or newer. The installer sets up a separate root-owned helper on a timer that runs
smartctland writes a report file; the agent itself stays unprivileged and only reads that file. Asmartctlthat hangs on a dying drive delays the helper, never your server's check-in. - UPS needs a running NUT server with
upscon the path. To read a UPS attached to another machine, addENSJU_UPS_SERVER=host.
The collectors run on Linux and FreeBSD, which covers Proxmox, plain Debian or Ubuntu hosts, and OPNsense or pfSense boxes. Re-running the installer keeps your existing settings, so adding a collector later is just running the command again with the new flag.
What alerts, and when
| Condition | Fires when | Clears when |
|---|---|---|
| Pool degraded | Pool state isn't ONLINE | The pool is ONLINE again |
| Pool errors | A read, write or checksum counter goes up | No new errors for 24 hours |
| Scrub overdue | No completed scrub in 45 days (adjustable) | A scrub completes |
| Disk failing | FAILED verdict, a rising sector count, NVMe endurance at 90%, or a media error | Never on its own: dismiss it |
| UPS on battery | 2 check-ins on battery (adjustable), or at once if also low | Back on line power |
| UPS low battery | The UPS reports LB | It stops reporting it |
| UPS unreachable | Three failed reads in a row | A clean read |
Two of these are deliberate. A failing-disk alert never clears by itself, because a drive that started reallocating sectors isn't healthy again just because this hour's numbers held steady; it stays open until you dismiss it or pull the drive. And the scrub clock starts when Ensju first sees a pool, so switching ZFS monitoring on for an old server doesn't page you tomorrow about a scrub schedule you never had.
Alerts go to the same channels as the server's offline alerts. All of this is on every plan: Free covers 1 server agent, and Homelab ($10/mo) covers 5.
The rest of the storage picture
Healthy disks still fill up, and healthy pools still lose data when the backup job quietly stopped running in March. Two more checks finish the picture:
- A disk-full forecast that alerts on "fills in 6 days" instead of a percentage, so a slow leak gets caught early and a big, stable disk doesn't nag you.
- A heartbeat on every backup job, so a backup that silently stops running pages you.
If you're setting up a lab from scratch, the homelab monitoring guide covers the rest of what's worth watching.
Frequently asked questions
- How often should I scrub a ZFS pool?
- Monthly is a common rhythm for a home pool on consumer drives. What matters more is that scrubs actually run and finish: a scrub schedule that silently stopped months ago is worse than no schedule, because you think you're covered. Alert when no scrub has completed in your chosen window.
- Which SMART attributes actually predict a failing drive?
- On spinning disks, watch reallocated sectors, pending sectors and offline uncorrectable sectors. A count that is rising matters more than its absolute value. On NVMe drives, watch the endurance used percentage and media errors. The overall SMART verdict (PASSED or FAILED) is the last line of defense, not the first.
- Does SMART monitoring need root access?
- Reading SMART data needs raw device access, which normally means root. Ensju keeps that out of the monitoring agent: a separate root-owned helper runs smartctl on a timer and writes a report file, and the unprivileged agent only reads the file.
- Can I monitor a UPS that's plugged into a different machine?
- Yes, if that machine runs a NUT server. Point the agent at it with ENSJU_UPS_SERVER=host (or host:port) and it queries that NUT server instead of a local one.
- Does this work on OPNsense or pfSense?
- Yes. The ZFS, SMART and UPS collectors run on Linux and FreeBSD, and OPNsense and pfSense are FreeBSD-based. They are ignored on macOS and Windows.