Architecture and operating model
GPU capacity has three budgets: model weights, runtime workspace and KV cache for active sequences. Parameter count multiplied by bytes per precision is only the weight floor. Tensor parallelism can divide weights but adds communication, topology constraints and replicated buffers.
Kubernetes device plugins advertise GPUs as extended resources; the scheduler does not inherently understand GPU memory or performance class. Node labels and affinity, taints and tolerations, topology, runtime class and health checks define valid placement. Sharing requires explicit design around supported MIG or time-slicing and isolation properties.
CPU or request-count autoscaling can react too late for LLM queues. Observe queue depth, pending tokens, TTFT and KV utilization. A replica may need minutes to download and load a model, so minimum replicas, warm pools or preloading must follow the SLO.
- 1Request/token demand model
- 2GPU memory and topology profile
- 3Kubernetes placement and autoscaling
- 4SLO, cost and failure validation
Design parameters
- Weight memory
- An 8B model is theoretically about 16 GB at BF16; runtime and KV allocations are additional.
- KV budget
- Measure layers, hidden size, context, dtype and concurrent sequences using engine allocator metrics.
- Topology
- Align NVLink/PCIe, NUMA and NIC locality with worker placement across GPUs.
- Startup time
- Measure image pull, weight download, validation and warmup separately and include them in scale-up policy.
Worked example
On a 24 GB GPU, 15 GB of weights and 2 GB runtime leave 7 GB for KV. If an 8k-context sequence uses 420 MB in the lab, the memory ceiling is about 16 sequences; fragmentation and safety margin suggest a target of 12-13.
At 12 requests/s and five seconds average service time, Little's Law implies 60 active requests. If a replica supports 12 sequences, at least five active replicas are needed; a sixth covers N+1 failure and maintenance.
Example commands: replace lab values and confirm permissions and software versions before use.
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu
kubectl describe pod llm-worker-0
llm-capacity profile --context 1024,4096,8192 --concurrency 1,8,16Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Pod remains Pending | GPU resource, labels/taints or plugin health do not match. | Inspect node allocatable, pod events, device-plugin DaemonSet and runtime class. |
| Scale-up is too slow | Model loading exceeds the alert horizon. | Compare queue growth with cold-start phases and set minimum replicas or a warm pool. |
| One node is slower | GPU model, power/clock, PCIe/NVLink or NUMA placement differs. | Compare node features and identical benchmarks by topology identity. |
Acceptance checks
- Capture real prompt and output token histograms.
- Measure weight, runtime and KV-cache memory separately.
- Connect GPU model/topology labels to scheduling policy.
- Test cold starts and model downloads during disruption.
- Validate autoscaling against queue and token metrics.
- Include N+1 failure, maintenance and capacity margin in cost.
Related concepts
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
NUMA locality
On NUMA systems a processor may access nearby and remote-node memory at different costs. VM vCPU and memory sizes must be assessed against physical node capacity and hypervisor placement. More vCPUs do not always improve performance: reduced locality and scheduling waits can make it worse. Record physical topology, topology exposed to the guest and the workload’s memory behavior together. Compare changes under the same load, using application latency and remote-memory effects rather than CPU percentage alone.
Resource contention
Contention occurs when workloads sharing CPU, memory, storage or a link make each other wait. Apparent spare total capacity can hide a hot core or single-queue bottleneck. Correlate backup jobs, antivirus scans, index maintenance and user traffic on a common timeline. Confirm the bottleneck before adding resources. Run a workload alone and with its usual competitors to separate shared-resource effects, and report peak-hour latency alongside average utilization.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.