Architecture and operating model
An LLM serving layer does more than load weights: it manages request queues, continuous or dynamic batching, KV cache, tokenization, streaming, parallelism, health signals and backpressure. High tokens per second can conflict with low time to first token, so selection must use the real prompt-length distribution.
vLLM focuses on LLM serving with PagedAttention, continuous batching and an OpenAI-compatible API. Triton combines multiple frameworks and model types using ensembles, model instances and dynamic batching. TGI is an established LLM server, but its maintenance and support status must be verified for the chosen version in official documentation.
Selection is not a feature checklist. Evaluate tensor/pipeline parallelism, quantization, LoRA lifecycle, multi-model placement, GPU architecture, observability and operational failure domains. A portable external API plus a versioned test set reduces migration cost.
- 1API and identity gateway
- 2Scheduler, queue and batching
- 3Model workers and GPU memory
- 4Streaming, metrics and result
Design parameters
- Queue policy
- Set maximum queue, timeout, priority and cancellation behavior for product traffic.
- Batch window
- Larger batches may improve throughput; measure the delay that keeps p95 TTFT within budget.
- KV-cache share
- Budget weights, workspace and KV cache separately; long context reduces concurrency.
- API portability
- Wrap identity, streaming, errors and usage accounting in a provider-neutral contract.
Worked example
If one GPU generates 18,000 output tokens/s and the average answer is 450 tokens, the theoretical ceiling is 40 responses/s. At 70% sustainable utilization and six seconds average generation time, Little's Law suggests roughly 168 concurrent jobs, but KV-cache capacity may impose a lower ceiling.
If a 2 ms batch delay raises throughput 28% while p95 TTFT moves from 420 to 510 ms, it may fit an 800 ms SLO. Re-test when short interactive prompts share the server with long batch requests.
Example commands: replace lab values and confirm permissions and software versions before use.
llm-bench run --targets vllm,triton,tgi --trace production-sample.jsonl --concurrency 1,8,32,128
llm-bench report --metrics ttft_p95,tpot_p95,tokens_per_second,error_rateTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| GPU busy but throughput low | Small batches, CPU tokenization or memory bandwidth may dominate. | Correlate queue, batch size, prefill/decode time and GPU SM/memory metrics. |
| TTFT spikes at peak | The queue grows or long prefill work blocks short requests. | Separate input-token distribution and per-request queue/prefill time. |
| Random OOM failures | KV cache, long contexts and concurrency exceed the memory budget. | Measure free memory after weights and KV growth per active sequence. |
Acceptance checks
- Compare engines with the same model, tokenizer and precision.
- Generate load traces from real input/output token distributions.
- Record TTFT, TPOT, token/s, errors and cancellations.
- Test queue behavior during OOM and worker restarts.
- Pin model versions for canary and rapid rollback.
- Include idle GPU, replication and telemetry overhead in cost.
Related concepts
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Queues and concurrency
A queue holds work arriving faster than a resource can process it. More concurrency can improve utilization up to a point, then increase waiting time. In a stable system Little’s law relates L = λ × W: average work in the system equals throughput multiplied by average total time. Units must agree. At 2,000 operations/s and 5 ms total time, roughly 10 operations are present concurrently. This is a planning relationship; queue limits, bursts and highly variable service times still require measurement.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Resource contention
Contention occurs when workloads sharing CPU, memory, storage or a link make each other wait. Apparent spare total capacity can hide a hot core or single-queue bottleneck. Correlate backup jobs, antivirus scans, index maintenance and user traffic on a common timeline. Confirm the bottleneck before adding resources. Run a workload alone and with its usual competitors to separate shared-resource effects, and report peak-hour latency alongside average utilization.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.