Architecture and operating model
`zfs receive -s` preserves partial receive state after interruption and exposes an opaque `receive_resume_token` on the target. The source can continue the missing stream with `zfs send -t <token>`. A token is neither a snapshot name nor a reusable retry key. It binds a particular source history, target partial state and send semantics, so record it with both datasets.
Before resuming, confirm that source snapshots still exist, the target has not received another stream or local writes, and the token is present. `zstream token` can decode diagnostic details on supported releases, but never edit the token. A raw encrypted, replicated or property-preserving original stream must keep compatible semantics. Deleting a required source snapshot can invalidate the token.
Read the token from the target, hold it in a protected variable, generate `zfs send -t` on the source and connect it to `zfs receive -s` on the same target. Track every command in an SSH pipeline; checking only the final exit code can hide a send failure. `set -o pipefail`, throughput metrics and timestamps on both systems make the retry auditable. On completion, the token should clear and the expected snapshot should appear.
Do not immediately discard partial state when resume fails. Capture the error, source snapshots, target token, pool health and free space. If you intentionally abandon it, `zfs receive -A <dataset>` removes the partial state and makes the old token unusable. Before a new full stream, evaluate RPO impact, transfer time, common-snapshot retention and dependent clones.
- 1Interrupted receive and partial target state
- 2Saved receive_resume_token
- 3Source zfs send -t stream
- 4Snapshot and token validation plus RPO record
Design parameters
- Snapshot continuity
- Retain the source snapshot and common incremental base required by the token.
- Send semantics
- Keep raw encryption, recursive, property and replication flags compatible with the first stream.
- Target space
- Budget remaining data, metadata, reservations and copy-on-write overhead.
- Pipeline failure
- Prove SSH, send and receive exit codes with pipefail.
- Abandon decision
- Record token, RPO, full-send duration and clone dependencies before `receive -A`.
Worked example
If 3 TiB of a 4 TiB initial stream completed, resume can focus on the remainder, but the token is not a byte counter. Combine `zfs send -nPv -t`, network metrics and target space.
At 400 Mbit/s, a 600 GiB remainder has an ideal transfer time near 12,000 seconds, or 3 hours 20 minutes. Recalculate the window from measured throughput after SSH, disk and protocol overhead.
Example commands: replace lab values and confirm permissions and software versions before use.
set -o pipefail
token=$(ssh backup zfs get -H -o value receive_resume_token tank/replica/app)
zfs send -nPv -t "$token"
zfs send -t "$token" | ssh backup zfs receive -s tank/replica/app
ssh backup zfs get receive_resume_token tank/replica/app
ssh backup zfs list -t snapshot tank/replica/app
Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| invalid argument during resume | The source history or send semantics may no longer match the token. | Decode diagnostics with `zstream token` and compare snapshots and original flags. |
| Target reports no space | Partial data, reservations or metadata may have exhausted the pool. | Check pool space and dataset reservations before changing the token state. |
| Command ends but snapshot is absent | Only the last pipeline status may have been checked, or receive used another dataset. | Enable pipefail and verify both logs, destination and snapshot list. |
Acceptance checks
- Record the target dataset token.
- Confirm required source snapshots and common base still exist.
- Match initial send and receive flags.
- Run resume with pipefail and retain both endpoint logs.
- Verify token clearance and target snapshot creation.
- Record RPO, transferred bytes and measured duration.
Related concepts
Replication scope
Replication transfers data or state changes to another copy. Synchronous transfer can introduce latency and connectivity dependence; asynchronous transfer can lag behind. A current replica is not necessarily protected from incorrect changes: deletion and corruption can propagate too. Compare lag with the application’s last committed operation, not merely connection status. Define which copy receives write authority after the source is lost and how the old node is reconciled when it returns. Report replication lag and the actual recoverable point separately.
RPO and the actual loss window
RPO is the acceptable duration of data loss after an incident. A backup schedule alone does not prove it: failed jobs or delayed replication to another site can enlarge the window. Compare the incident time with the latest usable, consistent recovery point. If an incident occurs at 14:00 and the verified copy is from 13:20, the observed loss window is 40 minutes. Set targets per application; a file archive and a database receiving continuous orders may have different requirements.
Application consistency
A bootable copy does not prove application-data consistency. Operating-system caches, database logs and write ordering across disks or services affect the result. A crash-consistent copy resembles recovery after an unexpected shutdown; an application-consistent copy follows supported application preparation and write coordination. Validate transaction integrity, relationships between records and application behavior after recovery, rather than only counting files. Confirm backup integration, application version and reported errors before assuming that consistency was achieved.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Retention and capacity
Retention defines which recovery points are kept and for how long. Daily, weekly and monthly points do not represent identical change patterns; full-copy creation and chain dependencies affect physical capacity. Retention decisions combine business requirements, applicable obligations and technical capacity. Longer retention does not automatically provide better recovery: the right point must be discoverable and readable. When changing a policy, test whether existing points are deleted immediately or handled differently by the product.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.