Architecture and operating model
eBPF attaches verified programs to tracepoints, kprobes, fentry, networking and other kernel hooks for low-level observation. The verifier explores register and memory access paths so accepted programs can run in a constrained context. Acceptance does not mean a tracing query has zero overhead or that collected data is non-sensitive.
bpftrace compiles concise diagnostic queries into BPF programs. Stable tracepoints usually survive kernel changes better than kprobes tied to internal symbols. Attaching wildcards to thousands of functions, calling `printf` for every event or using high-cardinality map keys can consume substantial CPU and memory. Prefer aggregation, sampling and a time limit.
State one production hypothesis, such as whether P99 file-open latency comes from NFS waits. List the target tracepoint, validate fields through BTF or tracefs and run first on a low-traffic canary. Capture baseline CPU, load, dropped events and target latency. Enforce automatic stop with `timeout` and bound map output.
Privileges can involve CAP_BPF, CAP_PERFMON, CAP_SYS_ADMIN or a privileged service, depending on deployment. Package persistent tools through controlled units and audit. PIDs, paths, addresses and arguments may be sensitive; define purpose, masking and retention. After diagnosis, prove that no probe or pinned map remains.
- 1Measurable hypothesis
- 2Bounded probe and verifier
- 3In-kernel map aggregation
- 4Timed output, comparison and cleanup
Design parameters
- Probe stability
- Prefer defined tracepoints or fentry and use version-sensitive kprobes only when needed.
- Event rate
- Estimate events per second; aggregate in maps and sample instead of printing each event.
- Map budget
- Bound key cardinality × value size × per-CPU copies.
- Run duration
- Add timeout, canary scope and automatic cleanup.
- Data handling
- Record masking, access and retention for paths, PIDs, addresses and arguments.
Worked example
At 80,000 syscalls per second, emitting one userspace line per event can be costly. Aggregating counts by process in a kernel map and printing every five seconds reduces 400,000 lines to a small summary. Still compare CPU and lost events against baseline; low output alone does not prove low overhead.
If a latency histogram has 900 calls below 1 ms, 90 at 1–10 ms and 10 at 10–100 ms, one percent sits in the last band. That approximates a P99 boundary but is not an exact percentile without finer buckets. Correlate it with application and block-I/O or NFS evidence.
Example commands: replace lab values and confirm permissions and software versions before use.
sudo bpftrace -l 'tracepoint:syscalls:sys_enter_openat'
sudo timeout 30 bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); } interval:s:5 { print(@); clear(@); }'
sudo bpftool prog show
sudo bpftool map show
sudo bpftool link showTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Verifier rejects the program | Field access, pointer bounds, stack or helper use may not fit the program type. | Capture the verifier log and kernel/BTF version; do not try to disable verification. |
| CPU rises during bpftrace | Event rate, wildcard probes, printf or key cardinality may be too high. | Compare hits, lost events, map size and process CPU with baseline. |
| Probe name is missing | Kernel configuration, BTF, module or symbol can differ on this release. | Check `bpftrace -l`, tracefs events and BTF availability. |
| Resources remain after exit | A pinned map, program or service attachment may still exist. | Use `bpftool prog/map/link show` to identify ownership and pin paths. |
Acceptance checks
- Write the diagnostic hypothesis and success metric.
- List and validate probes and fields on the target kernel.
- Bound event rate and map cardinality.
- Measure CPU and lost events with timeout on a canary.
- Apply masking and retention to sensitive fields.
- Verify cleanup of links, programs and pinned maps.
Related concepts
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Service lifecycle and dependencies
A running process does not prove that users receive correct service. Review startup order, network or database dependencies, environment variables, file access and health checks together. Restart loops can conceal the root cause. Compare exit codes, recent logs and resource limits, and identify whether systemd or container management triggered the restart. After a controlled restart, validate sessions, writes and dependent services.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Least privilege and separation of duties
Least privilege limits the permissions needed for a task to specific resources and time periods. Giving everyone the same administrator role complicates access reviews and investigations. Separate daily-use identities from privileged accounts, and restrict interactive use and unnecessary network access for service identities. An access matrix should show which operation an identity may perform on each resource. Test whether an application keeps working after a permission is removed: unnecessary privileges sometimes become visible only during controlled reduction.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.