Architecture and operating model
Multipart upload divides a large object into parts that upload independently and are recorded by part number and ETag. The object becomes visible after CompleteMultipartUpload submits the ordered part list. An initiated but incomplete upload may be absent from normal object listings while continuing to consume storage and cost.
After a network interruption, retrying with the same upload ID and part number can replace a part; opening a new upload ID leaves the old parts orphaned. Do not assume every ETag is plain MD5: encryption, checksum selection and S3-compatible implementation behavior vary. Compare ListParts with the client's local manifest before completion.
A cleanup rule distinguishes active and abandoned uploads mostly by age. For long archive transfers, keep the lifecycle threshold above the real maximum duration. Abort requires authorization and can race with in-flight parts, so verify the result again with ListMultipartUploads.
- 1CreateMultipartUpload and upload ID
- 2Upload parts and checksums
- 3Validate manifest with ListParts
- 4Complete or controlled Abort
Design parameters
- Part size and count
- Balance retry cost, concurrency, product limits and memory.
- Checksum/ETag
- Select a checksum supported by client and target; do not infer ETag semantics.
- Concurrency
- Tune for link capacity, source read rate, CPU and target throttling.
- Orphan threshold
- Keep lifecycle age above the longest valid transfer plus maintenance margin.
Worked example
An 800 GiB file split into 128 MiB parts creates 6,400 parts. Eight workers with a 256 MiB buffer each need about 2 GiB of client memory. A 1 Gb/s link has a theoretical minimum near 1.8 hours before overhead and retries, so a one-day abort threshold may still be risky on a slow planned path.
If ListMultipartUploads returns 37 records while the application has four active jobs, correlate the other 33 by age, owner and last part time. Abort one abandoned upload as a pilot, verify storage and listing results, then apply a lifecycle rule.
Example commands: replace lab values and confirm permissions and software versions before use.
aws s3api list-multipart-uploads --bucket lab-bucket --prefix archive/
aws s3api list-parts --bucket lab-bucket --key archive/db.dump --upload-id UPLOAD_ID
aws s3api abort-multipart-upload --bucket lab-bucket --key archive/db.dump --upload-id UPLOAD_ID
aws s3api list-multipart-uploads --bucket lab-bucket --prefix archive/Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| InvalidPart or InvalidPartOrder | The completion manifest may contain a missing, changed or unordered part. | Compare ListParts with local part number, size and ETag/checksum records. |
| Storage grows while object count is flat | Incomplete multipart uploads may be accumulating. | Group ListMultipartUploads by age, prefix and initiating identity. |
| Parts remain after Abort | A part may have raced with Abort or listing may lag. | Stop new uploads, relist the same upload ID and repeat Abort if necessary. |
Acceptance checks
- Map active upload IDs to application jobs.
- Validate part size and maximum-part calculations.
- Compare ListParts with the local manifest before completion.
- Test checksum behavior on the target product version.
- Pilot Abort and capacity recovery on one orphan.
- Set lifecycle age from the longest valid transfer.
Related concepts
Application consistency
A bootable copy does not prove application-data consistency. Operating-system caches, database logs and write ordering across disks or services affect the result. A crash-consistent copy resembles recovery after an unexpected shutdown; an application-consistent copy follows supported application preparation and write coordination. Validate transaction integrity, relationships between records and application behavior after recovery, rather than only counting files. Confirm backup integration, application version and reported errors before assuming that consistency was achieved.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Retention and capacity
Retention defines which recovery points are kept and for how long. Daily, weekly and monthly points do not represent identical change patterns; full-copy creation and chain dependencies affect physical capacity. Retention decisions combine business requirements, applicable obligations and technical capacity. Longer retention does not automatically provide better recovery: the right point must be discoverable and readable. When changing a policy, test whether existing points are deleted immediately or handled differently by the product.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.