Architecture and operating model
LLM evaluation is not one average score. Task success, factuality, grounding, safety, style, latency and cost are measured separately. The dataset represents intents, risk tiers, languages, lengths and difficult cases, while production incidents continuously become regression examples.
Deterministic checks cover schema, citation, prohibited terms and tool parameters. Human review calibrates ambiguous quality and business fit. Model judges scale evaluation but the judge model, rubric and answer order can bias results; monitor agreement on a small human-labeled control set.
Compare releases on paired examples. Inspect losses against the baseline and high-risk slices, not only the candidate's aggregate. Report statistical uncertainty and run-to-run variation. A gate must not trade security or p95 latency away for a quality increase.
- 1Versioned, risk-stratified dataset
- 2Deterministic, human and model judges
- 3Paired regression analysis
- 4Production canary and feedback
Design parameters
- Golden-set record
- Store input, expected behavior, source, risk label and rubric version for every case.
- Paired comparison
- Run identical inputs on candidate and baseline and measure win/loss/tie plus critical regressions.
- Judge calibration
- Track agreement, class-specific errors and instability on human-labeled examples.
- Acceptance budget
- Set separate thresholds for quality, security, p95 TTFT, token cost and errors.
Worked example
If a candidate wins 330, ties 210 and loses 60 of 600 cases, its win rate excluding ties is 84.6%. But if it regresses on 6 of 40 payroll questions, the critical-slice regression is 15% and should block deployment despite the aggregate.
If Cohen's kappa between judge and humans is 0.42, the judge should not be the sole gate. Clarify rubric examples, increase blinded human sampling and pin the judge version in the eval manifest.
Example commands: replace lab values and confirm permissions and software versions before use.
llm-eval run --candidate release-42 --baseline release-41 --dataset support-golden-v7
llm-eval gate --min-win-rate 0.60 --max-critical-regression 0 --max-ttft-p95-ms 900Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Offline score is high but users are unhappy | The eval set misses real intent or difficulty. | Sample production by stratum and compare missing intents and slices. |
| Judge always prefers the candidate | Position, verbosity or model-family bias exists. | Randomize answer order and test against human-labeled calibration cases. |
| Scores vary between runs | Sampling, model aliases or external sources change. | Pin model version, temperature/seed, retrieval snapshot and tool mocks. |
Acceptance checks
- Version datasets by use case and risk slices.
- Run deterministic checks before model judges.
- Calibrate judge rubrics on a human-labeled control set.
- Compare candidate and baseline on paired examples.
- Gate security, latency and cost alongside quality.
- Add canary incidents and feedback to regression datasets.
Related concepts
Application consistency
A bootable copy does not prove application-data consistency. Operating-system caches, database logs and write ordering across disks or services affect the result. A crash-consistent copy resembles recovery after an unexpected shutdown; an application-consistent copy follows supported application preparation and write coordination. Validate transaction integrity, relationships between records and application behavior after recovery, rather than only counting files. Confirm backup integration, application version and reported errors before assuming that consistency was achieved.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Policy lifecycle
A policy needs management as its scope changes, not just when it is enabled. Separate drafting, observation, pilot deployment, enforcement and periodic review. Every exception needs an owner, reason, scope and expiry date. A successful pilot can still cause production false positives if test data does not represent real user behavior. Identify affected users and applications before deploying with measurable acceptance criteria. A rollback path should identify who may reverse the change and under which conditions, rather than merely locating an interface button.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Cost boundaries
Cost includes more than purchase price or a monthly resource bill. Licensing scope, storage, egress, backup, support, operating effort and disruption impact are separate items. Compare equal service levels, capacity and time periods; a cheaper option may exclude an obligation. Compare forecast and actual spending by resource or business unit. Assess underuse risk for commitments and uncontrolled growth for flexible models. Verify pricing and licensing conditions against official documents at decision time.
Version and support lifecycle
Installability does not prove production support. Review the compatibility chain across OS, application, drivers, extensions and management tools. Update plans should record version, support end, restart needs and rollback methods. A test environment that does not reflect production can produce misleading results. Validate service health and existing workflows after a change, not just version numbers. Remember that pinning a version can also prevent future security fixes.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.