Architecture and operating model
Choose cloud APIs versus on-prem open models from the workload, not a privacy slogan. Put data classification and transfer boundaries, quality, context, tools, latency, availability, demand variation, team skill, license and total cost in one matrix.
Cloud APIs may offer managed scaling and rapid model access while introducing quotas, regional/data-processing terms, network dependency and unit-price risk. Local models provide control and fixed capacity but transfer GPU procurement, patching, serving, evaluation, power and on-call operations to the organization. Hybrid routing can separate tasks by data class.
For portability, application logic should not embed provider-specific APIs. Use common message/tool schemas, eval sets, token accounting, error contracts and a policy router. Models are not assumed equivalent; each route passes its own acceptance gates.
- 1Request, identity and data class
- 2Policy router and model route
- 3Cloud API or local serving
- 4Shared eval, telemetry and fallback
Design parameters
- Data boundary
- Contractually verify fields allowed outside, retention/training options and region.
- Demand curve
- Use peak QPS, context/output distribution and seasonality for capacity and cost.
- Operations load
- Include GPUs, energy, cooling, redundancy, patching, 24×7 monitoring and specialists in TCO.
- Exit plan
- Exercise portable APIs, evals, data export and an alternate route at least annually.
Worked example
For 60M monthly input and 8M output tokens, calculate API cost using separate unit prices plus retries, embeddings, rerank and network. For local, include four-GPU depreciation, 35% average utilization, N+1, energy and staff. Comparing purchase price alone with the API bill is invalid.
If 90% of interactive requests are general and low-sensitivity while 10% contain confidential documents, route the first to cloud and the second to local RAG/model serving. Test both on the same task eval; misclassification is a security incident.
Example commands: replace lab values and confirm permissions and software versions before use.
ai-route test --dataset enterprise-routing-v2 --routes cloud,onprem --verify-data-policy
ai-tco compare --horizon 36m --demand demand.csv --include-gpu,energy,staff,api,network
ai-failover exercise --from cloud --to onprem --max-rto 120sTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Cloud route hits quota | Peak traffic or token limits were omitted. | Inspect 429s, queue, quota and token distributions; test backoff and fallback. |
| Local model is not cheaper | Utilization is low and N+1/operations dominate. | Allocate GPU hours, idle share, energy and labor by task and tenant. |
| Fallback behaves differently | The alternate model did not pass the same eval/schema gates. | Run route-specific quality, tool-schema and security regression. |
Acceptance checks
- Encode data-class and regional transfer rules in routing policy.
- Compare candidates on identical task and security evals.
- Compare API cost and local TCO over the same three-year demand scenarios.
- Exercise quota, network and GPU-failure fallbacks.
- Use a provider-neutral application contract.
- Add license, model card and use restrictions to release evidence.
Related concepts
Cost boundaries
Cost includes more than purchase price or a monthly resource bill. Licensing scope, storage, egress, backup, support, operating effort and disruption impact are separate items. Compare equal service levels, capacity and time periods; a cheaper option may exclude an obligation. Compare forecast and actual spending by resource or business unit. Assess underuse risk for commitments and uncontrolled growth for flexible models. Verify pricing and licensing conditions against official documents at decision time.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Trust boundary and failure domain
A trust boundary separates components governed by different access decisions; a failure domain groups resources that one event can affect together. Two VLANs do not create a strong trust boundary when routing between them is unrestricted. Backups in different folders still share a failure domain if one administrator can delete both. Assess physical location, identity provider, management account, network path and power source separately. Test which access paths and recovery options remain available when a component is lost.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Dependencies and restart order
Services commonly depend on identity, DNS, time, networking, databases and licensing. Record a dependency graph describing conditions for operation, rather than only a list of equipment. Recovery order follows that graph; circular dependencies may require emergency access paths. Distinguish restored infrastructure from resumed business activity. Assign an owner, validation method and alternative access path to each dependency. Test assumptions by deliberately making one component unavailable in a controlled end-to-end exercise.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.