Architecture and operating model
Total request duration is insufficient for LLM observability. Measure gateway, retrieval, queue, prefill, time to first token (TTFT), time per output token (TPOT), tools and streaming as separate spans. Input/output tokens, model version and outcome connect cost and capacity to the same trace.
Content telemetry should be disabled or redacted by default. Prompts and responses may carry personal or confidential data; hashes, lengths, classes and sampling are often enough for performance diagnosis. Full content belongs only in approved, access-controlled and time-limited debug windows.
Define SLOs by user path: p95 TTFT and TPOT for chat, completion time and tokens/s for batch, tool success and total steps for agents. Version provider/model dimensions without creating unbounded metric cardinality.
- 1Request and identity context
- 2Retrieval/tool/model spans
- 3Token and latency metrics
- 4SLO, alert and cost feedback
Design parameters
- TTFT decomposition
- Keep gateway, retrieval, queue and prefill as separate spans and histograms.
- TPOT and throughput
- Derive TPOT from streaming intervals and output token/s from workers, correlated with queues.
- Cost accounting
- Calculate input, cached input, output, embedding, rerank and tool cost by request/tenant/use case.
- Content privacy
- Keep prompt/tool arguments off by default; use redaction, sampling and short retention when enabled.
Worked example
If p95 total latency is 9.2 s with retrieval 0.4, queue 3.8, prefill 0.7 and generation 4.1 s, address queue capacity before model speed. Correlate queue time with traffic, KV usage and replicas.
For 120M monthly input, 18M output and 30M embedding tokens, apply separate unit prices. Confirm cached-input discounts with cache-hit metrics; cost per request hides tenants with long contexts.
Example commands: replace lab values and confirm permissions and software versions before use.
otel-check trace --service ai-gateway --required-spans retrieval,model,tool
llm-slo evaluate --window 30m --ttft-p95 900ms --tpot-p95 45ms --error-rate 0.01
llm-cost reconcile --period 2026-10 --group-by tenant,use_case,modelTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Latency rises while GPU looks normal | Retrieval, queue, network or tools may dominate. | Compare span p95 changes by release and traffic class. |
| Cost rises at constant request volume | Context/output tokens or retries increased. | Inspect token histograms, cache hits, tool retries and model routing. |
| Dashboard becomes slow | High-cardinality IDs are metric labels. | Move IDs to traces/logs and bound the metric label set. |
Acceptance checks
- Verify trace propagation from gateway to model.
- Measure queue, TTFT, TPOT and total time with one clock basis.
- Reconcile token counters with a billing sample.
- Keep sensitive content disabled and time-bound.
- Test SLO alerts by model version and traffic class.
- Trace incidents back to release manifests and eval evidence.
Related concepts
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Queues and concurrency
A queue holds work arriving faster than a resource can process it. More concurrency can improve utilization up to a point, then increase waiting time. In a stable system Little’s law relates L = λ × W: average work in the system equals throughput multiplied by average total time. Units must agree. At 2,000 operations/s and 5 ms total time, roughly 10 operations are present concurrently. This is a planning relationship; queue limits, bursts and highly variable service times still require measurement.
Cost boundaries
Cost includes more than purchase price or a monthly resource bill. Licensing scope, storage, egress, backup, support, operating effort and disruption impact are separate items. Compare equal service levels, capacity and time periods; a cheaper option may exclude an obligation. Compare forecast and actual spending by resource or business unit. Assess underuse risk for commitments and uncontrolled growth for flexible models. Verify pricing and licensing conditions against official documents at decision time.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.