Architecture and operating model
A Ceph replicated pool places `size` full copies of every object across CRUSH failure domains. An erasure-coded pool splits data into k data and m coding shards; any k surviving shards can reconstruct it. Three-way replication consumes roughly three times raw capacity, while 4+2 EC has a theoretical 1.5-times overhead before placement and operating reserves.
The theoretical ratio is not enough. Small random writes can add read-modify-write work, coding CPU and shard traffic with EC. Replication often provides simpler latency behavior for metadata or small I/O. EC can suit large, sequential, capacity-oriented objects. Measure the actual workload on the intended disk, network and Ceph release.
The CRUSH rule must place shards across the chosen host or rack failure domain. Without enough independent domains for k+m shards, the design cannot deliver its expected tolerance or capacity. Uneven domain weights can strand capacity. Full and backfill-full ratios reserve room for re-protection after failure, so normal-state utilization alone is unsafe.
Pilot replication and the candidate EC profile on identical hardware. Measure usable capacity, 4 KiB and 1 MiB I/O, P95/P99 latency, recovery time and network use. Exercise an OSD and, where possible, a host failure. Document min_size and recovery controls. Keep backups separate: neither replication nor EC protects against every deletion or cluster-wide logical corruption.
- 1Workload and protection goal
- 2CRUSH failure-domain placement
- 3Replication or k+m writes
- 4Failure, recovery and capacity validation
Design parameters
- Protection profile
- Choose replication size or EC k+m from the shard and domain losses to tolerate.
- Failure domain
- Validate host or rack count and weights against CRUSH placement.
- Free capacity
- Calculate backfill room after a domain failure, not only normal utilization.
- I/O shape
- Test small random writes, large sequential objects and read/write ratios separately.
- Recovery budget
- Bound recovery network, CPU, client latency and time to re-protection together.
Platform implementation
Service names are reference points. Scope, defaults, region availability and operating requirements differ; they are not interchangeable guarantees.
| Criterion | Replication | Erasure coding |
|---|---|---|
| Raw overhead example | About 3.0× for size 3 | Theoretical 1.5× for 4+2 |
| Write path | Full copies | Split, encode and write shards |
| Latency profile | Can be predictable for small I/O | Can add CPU and network for small writes |
| Domain requirement | Placement targets equal to size | Independent targets for k+m shards |
| Typical candidate | Metadata and latency-sensitive data | Large, capacity-oriented objects |
Worked example
With 600 TB raw, three-way replication yields a theoretical 200 TB, while 4+2 EC yields 400 TB. Do not set a 400 TB operating target before applying an 85% full threshold, uneven rack weights and post-failure backfill reserve. Run the failure-domain calculation with actual CRUSH weights.
A 4+2 profile creates six shards. If each shard needs a separate host, at least six suitable hosts are required. Several hosts inside one rack do not provide rack-loss tolerance. When rack is the failure domain, six suitable, reasonably balanced racks may be required. Evaluate the selected domain, not only the OSD count.
Example commands: replace lab values and confirm permissions and software versions before use.
ceph osd erasure-code-profile ls
ceph osd erasure-code-profile get ec42
ceph osd pool get data_ec size
ceph osd pool get data_ec min_size
ceph osd df tree
ceph pg stat
ceph osd pool autoscale-statusTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Usable capacity is lower than expected | Uneven CRUSH weights or domain limits may strand raw capacity. | Compare OSD and rack weights with the failure-domain calculator. |
| EC has high small-write P99 | Read-modify-write, coding CPU or network fan-out may dominate. | Compare I/O size, CPU, network and OSD latency with replication. |
| Backfill stops after failure | OSDs may reach backfill-full or no placement target may remain. | Inspect fullness, PG state, CRUSH targets and backfill-full ratio. |
Acceptance checks
- Classify data and metadata workloads separately.
- State the loss tolerance for replication size or EC k+m.
- Validate CRUSH domain counts and weights.
- Calculate post-failure backfill reserve.
- Run P95/P99 and recovery pilots on identical hardware.
- Test restore from a backup outside cluster protection.
Related concepts
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Replication scope
Replication transfers data or state changes to another copy. Synchronous transfer can introduce latency and connectivity dependence; asynchronous transfer can lag behind. A current replica is not necessarily protected from incorrect changes: deletion and corruption can propagate too. Compare lag with the application’s last committed operation, not merely connection status. Define which copy receives write authority after the source is lost and how the old node is reconciled when it returns. Report replication lag and the actual recoverable point separately.
Durability and power loss
Survival of committed data depends on write guarantees across the database, OS, filesystem, controller and disks. Do not disable safety settings for speed without understanding caches and flush behavior. Power-loss-protected storage helps but does not prove correctness of the entire chain. Run failure and recovery tests in a controlled lab, not production. Validate application records and supported consistency checks rather than merely opening a sample file.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Failover and failback
Failover moves service to another component; failback returns it to the preferred location. Their risks and sequencing may differ. DNS updates, routing, sessions, replication lag and application dependencies determine user interruption. Define failover triggers, false-alarm behavior and approval for returning service. Simply shutting down a VM does not test every failure type. Measure network, storage and management-plane failures separately.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.