What does this guide solve?
Good monitoring is more than a CPU graph; alerts should connect capacity, errors, latency, service health and user impact.
Host metrics
Beyond CPU utilization, load average, steal time, context switches and run queue can reveal contention.
For memory, monitor cache, swap, OOM events and application working set.
Disk and filesystem
Monitor latency, IOPS, throughput, queue depth, capacity and inode use together.
Use growth trends and exhaustion forecasts, not only a 100% threshold.
Network and services
Errors/drops, latency, connection counts and retransmits can explain application issues.
Add systemd service, HTTP health, database connectivity and critical-port checks.
Prometheus vs Zabbix
Prometheus is strong for label-oriented time-series and cloud-native systems.
Zabbix is strong for integrated agent/SNMP/template monitoring in traditional infrastructure; choose by operations model.
Frequently Asked Questions
Does high load always mean insufficient CPU?
No. I/O waits and runnable process counts can also increase load average.
What disk threshold should trigger alerts?
Use growth trends and forecasted exhaustion in addition to fixed thresholds.
Should monitoring run separately?
Independent monitoring helps preserve visibility when a monitored host fails.
Sources and technical references
Evaluate Your Linux Server Project
We can review your Windows/Linux roles, application dependencies and migration targets.
Architecture and operating model
Monitoring should combine resource consumption with user impact. Prometheus scraping/metrics and Zabbix agent, SNMP and other checks offer different operating models. High-cardinality labels such as user identifiers can inflate data. Manage collection gaps and incorrect labels alongside alert thresholds.
- 1Metrics/log sources
- 2Collection and labels
- 3Time series/alerts
- 4Service response
Design parameters
- Cardinality and series
- Unbounded user/request-ID labels increase series count; define a label budget.
- Scrape and service health
- Exporter access does not prove application success; combine metrics, synthetic transactions and logs.
- Service and identity
- List the runtime user, groups, working directory and required permissions. Success as root does not prove operation under a restricted identity.
- Storage and persistence
- Check mounts, filesystem, inodes and persistent data together. Restart and recreation are different tests.
- Network and security layer
- Distinguish listening interfaces, firewall, TLS and SELinux/AppArmor policies. A block in one layer is not solved by altering another.
- Packages and operation
- Document versions, repositories, updates and service dependencies. Plan reboots and application validation after maintenance.
Worked example
Monitor latency, error rate, volume and saturation for an application. A live process can still fail HTTP health checks. Stop an exporter in a lab to test missing-data alarms. Define ownership and response instead of generating redundant alerts for the same event.
Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Metrics arrive but outage is missed | Only host health may be monitored or alert conditions may be wrong. | Test alert generation and notification with a controlled application failure. |
| Service is active but unresponsive | Process health, listening ports and application health are different checks. | Compare local health checks with remote requests, then examine logs and dependencies. |
| Files cannot be written | Permissions, ACLs, read-only mounts, inodes or capacity may be involved. | Validate directory traversal and mount state as the service identity; do not open permissions globally. |
| Configuration disappears after reboot | Temporary mounts, runtime settings or incorrect configuration sources may be involved. | Check persistent settings and startup order, then repeat a clean reboot test. |
Acceptance checks
- Test missing-data and user-impact alarms separately.
- Run the service under its designated restricted identity and test its workflow.
- Validate mounts, networking, time and application status after reboot.
- Record configuration, data and keys as separate recovery items.
- Run equivalent health checks before and after updates.
- Restore to a separate target to verify platform dependencies.
Related concepts
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Queues and concurrency
A queue holds work arriving faster than a resource can process it. More concurrency can improve utilization up to a point, then increase waiting time. In a stable system Little’s law relates L = λ × W: average work in the system equals throughput multiplied by average total time. Units must agree. At 2,000 operations/s and 5 ms total time, roughly 10 operations are present concurrently. This is a planning relationship; queue limits, bursts and highly variable service times still require measurement.
Resource contention
Contention occurs when workloads sharing CPU, memory, storage or a link make each other wait. Apparent spare total capacity can hide a hot core or single-queue bottleneck. Correlate backup jobs, antivirus scans, index maintenance and user traffic on a common timeline. Confirm the bottleneck before adding resources. Run a workload alone and with its usual competitors to separate shared-resource effects, and report peak-hour latency alongside average utilization.
Service lifecycle and dependencies
A running process does not prove that users receive correct service. Review startup order, network or database dependencies, environment variables, file access and health checks together. Restart loops can conceal the root cause. Compare exit codes, recent logs and resource limits, and identify whether systemd or container management triggered the restart. After a controlled restart, validate sessions, writes and dependent services.