Architecture and operating model
On a NUMA host, each CPU socket reaches its local memory controller with lower latency than remote nodes. If a KVM guest runs vCPUs on one node while allocating memory on another, remote traffic and interconnect pressure rise. vCPU pinning controls processors, `numatune` controls memory nodes, and huge pages control page size and reservation. They must be designed together.
Measure physical topology and workload first. `lscpu -e`, `numactl -H` and libvirt capabilities map CPUs, siblings and NUMA nodes. Do not count SMT siblings as independent physical cores. For a latency-sensitive VM, place vCPUs on cores in one node and reserve separate housekeeping CPUs for emulator and I/O threads. Leave capacity for host services.
HugeTLB pages can reduce TLB pressure and preallocate memory, lowering variability. However, 1 GiB or 2 MiB pages leave the flexible normal-memory pool; a wrong reservation can pressure the host or prevent VM startup. Calculate guest memory, QEMU overhead, vhost/I/O buffers and host reserve separately. Strict binding fails when the target node lacks enough pages.
Capture a baseline, pilot one VM and preserve rollback XML. CPU sets, memory backing and `numatune` can change together, but measure each effect separately. `numastat -p`, `/proc/meminfo`, perf and hypervisor scheduling metrics expose remote memory, page use and CPU wait. Test live migration, hotplug and ballooning requirements against huge-page and pinning constraints.
- 1Physical NUMA and CPU sibling map
- 2vCPU, emulator and I/O pinning
- 3NUMA memory policy and huge pages
- 4Load test, counters and rollback
Design parameters
- CPU topology
- Map sockets, cores, threads and NUMA nodes using physical sibling data.
- Host reserve
- Keep CPUs and memory for QEMU emulator, I/O threads, kernel and management agents.
- Memory budget
- Add QEMU and I/O overhead and host page-cache needs to guest memory.
- Page size
- Compare 2 MiB and 1 GiB support, fragmentation, reservation timing and rollback.
- Operational limits
- Test live migration, NUMA hotplug, ballooning and overcommit expectations.
Worked example
On a two-node host with 128 GiB per node, a 96 GiB VM may reduce remote access by keeping its memory with pinned vCPUs on one node. Strict policy still fails if that leaves no QEMU or host reserve.
A 64 GiB allocation uses 32,768 pages at 2 MiB before overhead. Compare `HugePages_Free`, `HugePages_Rsvd` and per-node hugepage files before and after startup rather than reserving only the theoretical guest total.
Example commands: replace lab values and confirm permissions and software versions before use.
lscpu -e=CPU,NODE,SOCKET,CORE,ONLINE
numactl -H
virsh vcpupin vm01
virsh emulatorpin vm01
virsh numatune vm01
numastat -p $(pgrep -f 'qemu.*vm01')
grep -E 'Huge|MemAvailable' /proc/meminfo
Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| VM cannot start in strict mode | The target node may lack free huge pages or lockable memory. | Check per-node huge pages, the QEMU log and memlock limits. |
| CPU is low but latency is high | vCPUs may be local while memory is remote. | Compare `numastat -p` miss counters with the live domain XML. |
| Host management becomes slow | VM pinning or huge-page reservation may have consumed housekeeping resources. | Inspect housekeeping CPU sets, normal-page `MemAvailable` and PSI metrics. |
Acceptance checks
- Record CPU sibling and NUMA topology.
- Reserve housekeeping cores and host memory.
- Verify vCPU, emulator and I/O pins in live XML.
- Reserve hugepage capacity per NUMA node.
- Measure remote memory and P99 latency under load.
- Test migration, restart and rollback.
Related concepts
NUMA locality
On NUMA systems a processor may access nearby and remote-node memory at different costs. VM vCPU and memory sizes must be assessed against physical node capacity and hypervisor placement. More vCPUs do not always improve performance: reduced locality and scheduling waits can make it worse. Record physical topology, topology exposed to the guest and the workload’s memory behavior together. Compare changes under the same load, using application latency and remote-memory effects rather than CPU percentage alone.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Resource contention
Contention occurs when workloads sharing CPU, memory, storage or a link make each other wait. Apparent spare total capacity can hide a hot core or single-queue bottleneck. Correlate backup jobs, antivirus scans, index maintenance and user traffic on a common timeline. Confirm the bottleneck before adding resources. Run a workload alone and with its usual competitors to separate shared-resource effects, and report peak-hour latency alongside average utilization.
Isolation versus virtualization
Containers and VMs provide different isolation boundaries. Containers share the host kernel; VMs run guest operating systems. Rootless execution, namespaces and capability restrictions can reduce risk, but configuration and host security still matter. Images, running containers and persistent volumes have separate lifecycles. Updating an image does not back up data. Verify process privileges, mounts, network access and persistence after recreation separately.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.