Architecture and operating model
VFIO uses an IOMMU to restrict a PCIe device's DMA to the selected VM address space. Full passthrough assigns one Physical Function to one VM; SR-IOV creates multiple Virtual Functions on a supporting PF and assigns each VF separately. These paths are more direct than paravirtual virtio devices but constrain live migration, snapshots and host-driver use.
The security boundary is the IOMMU group, not the individual device. Functions in a group that cannot be isolated must be assigned together; forcing an ACS override does not improve the hardware's real isolation. Enable IOMMU/VT-d/AMD-Vi in firmware, inspect kernel groups and bind the device to VFIO under change control.
NUMA placement determines proximity between the PCIe root, vCPUs and memory. Remote access can add latency and interconnect traffic. Budget VF count, MSI-X vectors, IOMMU capacity and physical-port bandwidth together. Creating the maximum supported number of VFs does not guarantee line rate to every VM.
- 1Firmware IOMMU and group discovery
- 2Prepare PF/VF and bind driver
- 3Assign hostdev through libvirt
- 4Test load, isolation and rollback
Design parameters
- IOMMU group
- Prove that assigned functions can be isolated from every other device in the group.
- NUMA proximity
- Place VM vCPUs and memory near the device's NUMA node where practical.
- VF and queue budget
- Calculate VFs, RX/TX queues, interrupt vectors and physical link capacity together.
- Lifecycle
- Document live migration, reboot, driver update and rollback constraints.
Worked example
Eight VFs with a 10 Gb/s target on a dual-port 25 Gb/s NIC request 80 Gb/s while physical capacity is 50 Gb/s before overhead. Acceptance must measure concurrent p95 throughput and loss across all eight VFs, not a single-VF peak.
If a device is on NUMA node 1 while 8 vCPUs and 32 GiB are pinned to node 0, remote memory and interrupt cost can rise. Run the same workload in local and remote placement and compare p99 latency, CPU steal and interconnect counters.
Example commands: replace lab values and confirm permissions and software versions before use.
dmesg | grep -Ei 'IOMMU|DMAR|AMD-Vi'
find /sys/kernel/iommu_groups/ -type l | sort
lspci -nnk -s 0000:3b:00.1
echo 8 | sudo tee /sys/class/net/ens3f0/device/sriov_numvfs
virsh nodedev-detach pci_0000_3b_00_1
virsh dumpxml vm01 | sed -n '/<hostdev/,/<\/hostdev>/p'Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| group is not viable | Another function in the IOMMU group may remain attached to a host driver. | Map every function with `find /sys/kernel/iommu_groups` and `lspci -nnk`. |
| VM starts but has no network | VF trust, VLAN, spoof check or switch-port policy may differ. | Inspect PF/VF settings, VM MAC/VLAN and physical-switch counters together. |
| Latency is higher than expected | Remote NUMA placement, interrupt distribution or IOMMU translation pressure may apply. | Compare `numactl`, `/proc/interrupts` and a local-NUMA run. |
Acceptance checks
- Confirm IOMMU is active in firmware and the kernel.
- Inventory the complete IOMMU group of every assigned device.
- Test VFIO binding and return to the host driver.
- Verify device ownership with the VM stopped and running.
- Measure throughput, p99 latency and loss under concurrent VF load.
- Record live-migration and host-maintenance limitations.
Related concepts
Isolation versus virtualization
Containers and VMs provide different isolation boundaries. Containers share the host kernel; VMs run guest operating systems. Rootless execution, namespaces and capability restrictions can reduce risk, but configuration and host security still matter. Images, running containers and persistent volumes have separate lifecycles. Updating an image does not back up data. Verify process privileges, mounts, network access and persistence after recreation separately.
NUMA locality
On NUMA systems a processor may access nearby and remote-node memory at different costs. VM vCPU and memory sizes must be assessed against physical node capacity and hypervisor placement. More vCPUs do not always improve performance: reduced locality and scheduling waits can make it worse. Record physical topology, topology exposed to the guest and the workload’s memory behavior together. Compare changes under the same load, using application latency and remote-memory effects rather than CPU percentage alone.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Version and support lifecycle
Installability does not prove production support. Review the compatibility chain across OS, application, drivers, extensions and management tools. Update plans should record version, support end, restart needs and rollback methods. A test environment that does not reflect production can produce misleading results. Validate service health and existing workflows after a change, not just version numbers. Remember that pinning a version can also prevent future security fixes.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.