If ZFS Scrubs Hurt Production, Your Pool Design Is Broken

When a monthly ZFS scrub doubles application latency, operators often delay it, then skip it entirely — until a disk fails, resilver crawls for a week, and a latent checksum error on a second disk turns a routine replacement into a restore-from-backup incident. A scrub is ZFS's lowest-priority, read-only workload; if that workload alone degrades production, the pool never had performance headroom. Since OpenZFS 0.8, scrubs run in two phases: a metadata traversal that builds sorted block ranges, followed by large sequential reads. On fragmented, small-block pools, that first phase remains an unavoidable random-read storm no sorting can fix.

The real cause is sizing. Pools are typically planned around usable terabytes and price alone, run at 85-95% full as a norm, RAIDZ delivers roughly one disk's worth of random IOPS per VDEV, and maintenance workloads — scrubs, resilvers, snapshot cleanup — never get a capacity budget. The result is a pool that barely serves the application and can't afford its own upkeep.

The fix doesn't require doubling spindle count. OpenZFS allocation classes let you add a special VDEV — typically a mirrored set of enterprise SSDs or NVMe drives — that absorbs all pool metadata and, via special_small_blocks, small file data too. This moves the scrub's random-read burden off spinning disks entirely, leaving HDDs to do what they're good at: sequential reads. zdb -bb can measure current metadata footprint, typically under 1% of allocated space on 128K-recordsize pools, so a modest mirrored NVMe set is a small fraction of the hardware budget. Because it's a top-level VDEV it must be mirrored, and since allocation classes apply only to new writes, zfs rewrite (OpenZFS 2.4) can accelerate the payoff on existing pools.