Architecture and operating model
A production AI package is more than a model file. Model digest, tokenizer, prompt template, retrieval/embedding version, data snapshot, tool schemas, runtime image, evaluation evidence and policy decision belong in one manifest. The registry records these relationships and stage transitions.
Deploy immutable artifact identities instead of a mutable 'latest' tag; an alias is only a controlled pointer. Training or prompt experiments must be reproducible from a pipeline run, with source-data access and license recorded. Approval uses a role and identity separate from artifact creation.
Rollback is more than starting an old container. Preserve the embedding dimension, prompt variables, data schema and tool contract expected by the old model. Incompatible indexes or feature transforms may require blue/green data planes. Canary, shadow and progressive traffic are tracked at manifest level.
- 1Artifact production and digest
- 2Registry, lineage and approval
- 3Canary deployment and observation
- 4Alias promotion or rollback
Design parameters
- Release manifest
- Bind model, prompt, data, index, image, policy and eval digests with immutable IDs.
- Stage gate
- Require authorized approval, eval evidence and security scan for staging-to-production.
- Compatibility contract
- Version I/O schema, embedding dimensions, tokenizer and tool parameters.
- Rollback objective
- Test RTO, previous artifacts, data compatibility and traffic switching together.
Worked example
A release manifest contains model sha256:m1, prompt p17, embedding e3, index snapshot i88 and runtime image c42. Changing only to p18 still reruns evals. If e4 changes vector dimensions, i88 is incompatible; build and validate a parallel index before moving traffic.
If a 5% canary requires 5,000 requests over at least 20 minutes, the time condition alone is insufficient at low traffic. Require both sample count and time. If errors exceed 1% or p95 TTFT exceeds 900 ms, point the alias back to the previous manifest.
Example commands: replace lab values and confirm permissions and software versions before use.
mlflow models register -m runs:/RUN_ID/model -n support-llm
release-manifest verify release-42.yaml
deploy canary --manifest release-42.yaml --traffic 5 --rollback-on error_rate>0.01Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Answers break after rollback | The old model runs with a new prompt, index or schema. | Compare every live artifact digest, not only model ID, with the prior release. |
| Training data is unknown | Pipeline lineage or dataset snapshot identity is missing. | Verify registry run links, object versions and data access records. |
| Alias points to wrong release | Manual changes or racing pipelines occurred. | Inspect alias audit logs and optimistic concurrency/revision checks. |
Acceptance checks
- Record content digests and owners for every artifact.
- Link pipeline run, source commit and data snapshot.
- Require eval/security gates before production alias changes.
- Automate schema and embedding compatibility tests.
- Use both time and sample thresholds for canaries.
- Exercise rollback to the previous two releases with representative traffic.
Related concepts
Version and support lifecycle
Installability does not prove production support. Review the compatibility chain across OS, application, drivers, extensions and management tools. Update plans should record version, support end, restart needs and rollback methods. A test environment that does not reflect production can produce misleading results. Validate service health and existing workflows after a change, not just version numbers. Remember that pinning a version can also prevent future security fixes.
Dependencies and restart order
Services commonly depend on identity, DNS, time, networking, databases and licensing. Record a dependency graph describing conditions for operation, rather than only a list of equipment. Recovery order follows that graph; circular dependencies may require emergency access paths. Distinguish restored infrastructure from resumed business activity. Assign an owner, validation method and alternative access path to each dependency. Test assumptions by deliberately making one component unavailable in a controlled end-to-end exercise.
Durability and power loss
Survival of committed data depends on write guarantees across the database, OS, filesystem, controller and disks. Do not disable safety settings for speed without understanding caches and flush behavior. Power-loss-protected storage helps but does not prove correctness of the entire chain. Run failure and recovery tests in a controlled lab, not production. Validate application records and supported consistency checks rather than merely opening a sample file.
Application consistency
A bootable copy does not prove application-data consistency. Operating-system caches, database logs and write ordering across disks or services affect the result. A crash-consistent copy resembles recovery after an unexpected shutdown; an application-consistent copy follows supported application preparation and write coordination. Validate transaction integrity, relationships between records and application behavior after recovery, rather than only counting files. Confirm backup integration, application version and reported errors before assuming that consistency was achieved.
Policy lifecycle
A policy needs management as its scope changes, not just when it is enabled. Separate drafting, observation, pilot deployment, enforcement and periodic review. Every exception needs an owner, reason, scope and expiry date. A successful pilot can still cause production false positives if test data does not represent real user behavior. Identify affected users and applications before deploying with measurable acceptance criteria. A rollback path should identify who may reverse the change and under which conditions, rather than merely locating an interface button.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.