ZFS Stripe on Proxmox: Why One SSD Failure Can Destroy Your Pool

What he built, and why it bit him​

Lewis says the two SATA SSDs had served him for years and were still basically healthy. He combined them into one striped ZFS pool inside his new Proxmox server. The pool holds ISOs, virtual machines, containers and a few short-term working copies of files.

Reads and writes hit both drives at once. He describes the result as much faster than a single old SSD, and the capacity is one big volume. He offers no benchmarks, so treat the speedup as his impression, not a measurement.

The trouble is what ended up on the pool. It was meant for disposable data. It now holds a large ISO repository and some prebuilt containers he would rather not lose. He sums up the trade as speed and size in exchange for roughly double the risk of data loss.

He adds two more complaints:

• Troubleshooting: there are two physical disks and two sets of SMART data behind what looks like one pool. When an error appears, he has to work out whether it comes from one drive, both drives, something upstream or software.
• Backups: his safety net is intermittent copies on a mechanical hard drive.

Why one failed disk takes the whole pool​

OpenZFS documentation backs up his central warning. ZFS dynamically stripes data across all top-level vdevs in a pool. A single disk used as a top-level vdev provides no redundancy, and its loss loses the whole pool.

The documentation also makes a terminology point. There is no "stripe" vdev type in ZFS. Striping is what the pool does across its top-level vdevs. A "striped ZFS pool" is therefore simply a pool with two single-disk top-level vdevs.

The rule applies to every layout, not just two-disk stripes. Redundancy in ZFS lives inside a top-level vdev, not across them. If a top-level vdev is lost, the pool is lost. Klara Systems makes the same point: if a disk dies in a striped pool, ZFS will refuse to import it because a top-level vdev is missing.

Compare the failure behaviour:

Usable space will be somewhat lower than nominal in every case because of formatting and pool overhead.

Is "double the risk" accurate?​

It is a reasonable rule of thumb, not a measured figure. Two drives give two chances to fail, and either failure kills the pool. Nobody has published failure rates for his specific drives, and age alone doesn't mean a failure is imminent. His reasoning that NAND wears over time is sound. His report includes no SMART data, endurance ratings or wear figures. The practical lesson is that the blast radius doubled, and that matters more than the exact probability.

What a better layout looks like​

Lewis's advice, and where I'd refine it:

• Disposable data: a stripe is a legitimate choice for scratch VMs and temporary files. He notes that RAID 0 and a ZFS stripe are conceptually equivalent, with RAID 0 usually relying on a hardware or other RAID layer and ZFS doing it in software.
• Data you care about: use a mirror, which resists one failure at half the capacity, and keep backups.
• Lowest maintenance: buy one larger, modern SSD.

A mirror is not a backup. It protects against one disk dying. It does not protect against deleted files, corruption written to both sides, ransomware or a lost server. His mechanical-drive copies are the right kind of mitigation, though how often they run and whether he verifies them matters a lot.

His closing suggestion is to stripe only if you have four drives and can add parity. That is a rule of thumb, not a hard threshold. OpenZFS lists RAIDZ1 as needing a minimum of 2 disks, with 3 or more recommended, and RAIDZ2 as needing at least 3, with 5 or more recommended. A pool of mirrors is another way to use four disks. The right layout depends on how much capacity you need and how many failures you want to survive.

Can you convert an existing pool?​

Lewis says ZFS can "unstripe" a drive only if one drive can hold everything. He says that stops being possible once you pass 50% of combined capacity. The OpenZFS documentation I reviewed doesn't state a 50% rule, so treat it as his experience, not a universal limit. What the documentation does describe is narrower:

• Removal: removing a top-level vdev requires matching ashift, no raidz, and encryption keys loaded. The OpenZFS layout guide says the removal evacuates allocated data to the remaining vdevs. That only works if they have room.
• Making a mirror: zpool attach can convert a single disk into a mirror. zpool attach adds a device to a mirror, or to an existing raidz group. The OpenZFS manual page adds that for a plain device, the new device is mirrored with it and resilvers. It contributes redundancy once the resilver completes.
• A common mistake: zpool add does something different. It appends a new top-level vdev and widens the stripe, with no redundancy gain. A GitHub issue shows a user making that exact mix-up and ending up with a more complicated pool.

In practice, one route is to back up everything, destroy the pool, and rebuild it as a mirror or as separate pools. Another is to remove one top-level vdev with zpool remove if the data fits on the other disk, then zpool attach a second disk to the survivor. The second route needs spare space and a good backup first. Always run zpool add -n or zpool remove -n first to preview the result. Check the exact layout with zpool status before attempting either.

Monitoring a striped pool​

If you run one, here is a minimal routine:

1. Run zpool status regularly and watch for read, write and checksum errors. The OpenZFS documentation says it shows the health and layout of all vdevs.
2. Use zpool iostat to see I/O statistics per vdev.
3. Track SMART data for each physical disk, since the pool will not summarize both for you.
4. Test restores from your backup, not just the backup jobs.

Verdict​

Lewis's headline holds up. Reusing old SSDs was a good idea. Striping them under data he didn't want to lose was the mistake. Matching the layout to the value of the data, plus a separate backup, solves most of this. If you have only two spare drives and something you can't afford to lose, mirror them or leave them separate.