Architecture and operating model
SLOG, L2ARC and special vdevs are not three interchangeable SSD tiers. A SLOG moves the ZFS Intent Log for synchronous-write durability to a separate device; it affects synchronous write latency and is not a general write cache. Power-loss protection, low latency and suitable endurance are primary selection criteria.
L2ARC is a secondary cache for read data leaving ARC. It can help when repeatedly accessed random data does not fit in RAM; an SSD might add little for one-pass sequential reads or workloads with few reusable misses. L2ARC headers consume RAM, and cold-start or warm-up behavior must be measured.
A special vdev stores metadata and, when configured, small blocks separately from normal data vdevs. It is an integral allocation class: losing it can lose the pool. Never add one unmirrored SSD without redundancy matching the main pool, independent backup and tested recovery.
- 1Collect workload and ARC/ZIL telemetry
- 2Classify sync-write, reread or metadata bottleneck
- 3Design the matching class with redundancy
- 4Test pilot, failure and recovery
Design parameters
- Synchronous write ratio
- Measure real fsync/O_SYNC traffic and p95/p99 latency before sizing SLOG.
- ARC hits and working set
- Base L2ARC decisions on ARC hit/miss, reuse and working-set size together.
- Small-block threshold
- Include blocks directed to the special class in capacity and redundancy calculations.
- Failure domain
- SLOG loss can affect pending synchronous work; special-vdev loss can affect pool integrity.
Worked example
A target of 10,000 synchronous 8 KiB IOPS over NFS represents about 78 MiB/s, but the design requirement is low p99 latency and burst durability. A five-second worst-case TXG window suggests only 390 MiB of data; device latency, power-loss protection, queue depth and endurance still require production-like testing.
If ARC hit ratio is 96% and remaining misses are mostly one-pass backup reads, L2ARC adds limited value. With high metadata latency and millions of small files, a mirrored special-vdev pilot may be more useful after measuring metadata size and the capacity effect of the small-block policy.
Example commands: replace lab values and confirm permissions and software versions before use.
zpool status -v
zpool iostat -v 1
arcstat 1
# Inspect pool and dataset properties on Linux/OpenZFS
zdb -C poolname
zfs get primarycache,secondarycache,recordsize,special_small_blocks poolname/datasetTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Throughput unchanged after adding SLOG | The workload may be asynchronous or limited by data-vdev throughput. | Compare synchronous-write ratio, zilstat and iostat latency. |
| ARC pressure rises after adding L2ARC | An oversized L2ARC may consume RAM for headers or cache data with little reuse. | Use arc_summary or arcstat to inspect headers, hits and evictions. |
| Special class fills rapidly | The small-block threshold may be broad or metadata estimate incomplete. | Inspect allocation-class capacity and dataset recordsize/special_small_blocks. |
Acceptance checks
- Classify workload into synchronous/asynchronous writes and reusable reads.
- Record baseline ARC hit ratio and p95/p99 device latency.
- Verify power-loss protection and endurance for a SLOG candidate.
- Design the special class with redundancy suitable for the main pool.
- Test device loss and import/reboot behavior in the pilot.
- Keep independent backup and test restore; snapshots and RAID are not substitutes.
Related concepts
Filesystem and storage layers
An application directory, mount point, logical volume and physical device belong to different layers. Inode exhaustion, read-only mounts and filesystem errors can stop writes even when space appears available. A snapshot often shares the same storage failure domain and is not an independent backup. Mount options and expansion methods depend on the filesystem. Verify mounts after reboot: writing to an unmounted directory can fill the wrong device.
IOPS and block size
IOPS measures operations per second, whereas MB/s measures transferred data. Identical IOPS values produce very different bandwidth at different block sizes. For example, 10,000 operations/s at 4 KiB is approximately 39.1 MiB/s. Read/write mix, random versus sequential access and protection mechanisms affect the result. Measure actual block distributions and concurrency rather than transferring a benchmark result directly to production. High disk IOPS does not prove good application response; latency must be examined alongside it.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Write endurance and wear
SSD selection requires more than capacity and initial performance. Consider TBW, DWPD, warranty duration, power-loss protection and workload write patterns together. DWPD calculations must use the capacity and conditions defined by the manufacturer. Database logs, small random writes and temporary work files affect wear differently. Monitor SMART or vendor health data, considering temperature, media errors and remaining endurance together. Write amplification means application-written bytes may differ from the amount written to NAND.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Durability and power loss
Survival of committed data depends on write guarantees across the database, OS, filesystem, controller and disks. Do not disable safety settings for speed without understanding caches and flush behavior. Power-loss-protected storage helps but does not prove correctness of the entire chain. Run failure and recovery tests in a controlled lab, not production. Validate application records and supported consistency checks rather than merely opening a sample file.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.