Architecture and operating model
Journald can keep volatile data under /run/log/journal or persistent data under /var/log/journal. Storage=auto uses persistent disk only when the persistent directory exists. One server can therefore retain history across reboot while another installation of the same distribution does not. Read effective configuration and directory state together.
When RateLimitIntervalSec and RateLimitBurst are exceeded, messages are suppressed per service and a summary is logged. SystemMaxUse, RuntimeMaxUse, KeepFree and filesystem pressure can also remove archived files. An application that never emitted an event, journald suppression and retention cleanup are different root causes.
journalctl --verify checks file integrity but cannot prove that the application generated every expected event. Combine application counters, a test event ID and journal cursors. Central forwarding is not automatically an independent backup of the local journal; queueing and loss during destination outage require separate design.
- 1Application stdout/syslog
- 2journald intake and rate limit
- 3Runtime or persistent journal
- 4journalctl/forwarder
Design parameters
- Storage
- Map persistent, volatile or auto behavior to reboot requirements and disk budget.
- Size limit
- Choose SystemMaxUse and SystemKeepFree alongside actual /var capacity and other services.
- Rate limit
- Measure normal peaks; fix application log storms instead of simply disabling all limits.
- Time and cursor
- Use boot ID, monotonic time and cursors to distinguish reboot or clock correction from loss.
Worked example
A service may emit 200 events/s normally and 8,000 events/s during failure, or 240,000 records in 30 seconds. Distribution limits may suppress part of that burst. Obtain a unique application counter first, then correlate journal suppression and disk usage over the same interval.
In a pilot, write event-id=doz-journal-001 with logger, record a cursor, flush journald and perform a controlled reboot. Locate the event around that cursor after startup. A production reboot requires separate change approval.
Example commands: replace lab values and confirm permissions and software versions before use.
systemd-analyze cat-config systemd/journald.conf
journalctl --list-boots
journalctl --disk-usage
journalctl --verify
logger -t doz-journal 'event-id=doz-journal-001'
journalctl -t doz-journal -o short-iso-preciseTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Old logs vanish after reboot | Storage may be volatile or /var/log/journal may not exist. | Inspect effective journald configuration, directory state and boot list. |
| Seconds disappear under load | Rate limiting or application stdout pressure may be involved. | Compare suppression messages, service rate and application counters over one window. |
| Journal retains fewer days | SystemMaxUse, KeepFree or an external vacuum may remove archives. | Inspect disk usage, filesystem free space, timers and configuration drop-ins. |
Acceptance checks
- Record effective Storage and drop-in order.
- Trace one event ID from application to journal.
- Produce controlled rate-limit suppression in a pilot.
- Measure archive deletion under disk pressure.
- Validate persistence and boot boundaries after restart.
- Test local retention during forwarder outage.
Related concepts
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behaviour. A log describes an event, a metric shows behaviour over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Filesystem and storage layers
An application directory, mount point, logical volume and physical device belong to different layers. Inode exhaustion, read-only mounts and filesystem errors can stop writes even when space appears available. A snapshot often shares the same storage failure domain and is not an independent backup. Mount options and expansion methods depend on the filesystem. Verify mounts after reboot: writing to an unmounted directory can fill the wrong device.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Retention and capacity
Retention defines which recovery points are kept and for how long. Daily, weekly and monthly points do not represent identical change patterns; full-copy creation and chain dependencies affect physical capacity. Retention decisions combine business requirements, applicable obligations and technical capacity. Longer retention does not automatically provide better recovery: the right point must be discoverable and readable. When changing a policy, test whether existing points are deleted immediately or handled differently by the product.
Service lifecycle and dependencies
A running process does not prove that users receive correct service. Review startup order, network or database dependencies, environment variables, file access and health checks together. Restart loops can conceal the root cause. Compare exit codes, recent logs and resource limits, and identify whether systemd or container management triggered the restart. After a controlled restart, validate sessions, writes and dependent services.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.