Architecture and operating model
DNS failover changes future DNS answers after a health check fails. Google Cloud DNS routing policies, Amazon Route 53 health checks and Azure Traffic Manager implement this with different resources and policies. They do not move an existing TCP session. Recursive and client caches can keep an old answer until TTL expiry, so RTO combines probe detection, TTL, connection lifetime and application retry.
A health endpoint should represent whether the service can serve users, rather than merely returning HTTP 200. Use a controlled, inexpensive synthetic operation when database writes or queue delay matter. An overly deep check can mark both regions unhealthy during dependency instability. A shallow check keeps DNS healthy while the application is broken. Test each provider's all-unhealthy and fail-open behavior.
Compare record and endpoint types, public or private zones, probe sources, expected status codes, TLS/SNI and routing policies. Cloud DNS supports health checks for defined load-balancer and external endpoint cases within routing policies. Route 53 relates records or aliases to health checks. Traffic Manager monitors endpoints in DNS-based profiles, with best-effort behavior when all eligible endpoints degrade.
A single DNS control plane simplifies operations but creates dependency on one authorization and change path. A second authoritative provider requires consistent zone and failover-state replication; independent automation can create split brain. Define primary and secondary endpoints, manual override, automation freeze and failback criteria in one runbook. During game days, test real requests through multiple resolvers, not only authoritative answers.
- 1Multi-region application endpoints
- 2Provider health probes
- 3DNS routing and failover decision
- 4Resolver cache and new client connection
Design parameters
- Detection budget
- Measure probe interval times failure threshold plus decision delay.
- DNS cache
- Observe recursive and client minimums in addition to authoritative TTL.
- Health semantics
- Separate readiness, liveness and dependency checks with distinct thresholds.
- All targets failed
- Lab-test provider fail-open, best-effort or NXDOMAIN behavior.
- Manual control
- Provide an audited override and rollback path for bad probes and maintenance.
Worked example
With 30-second probes, three failures and a 60-second TTL, the theoretical new-connection RTO is 30×3 + 60 = 150 seconds. Probe phase, DNS propagation, client cache and retry can extend it, so measure P50 and P95.
A health path that writes to a database can create thousands of side effects. Use an idempotent synthetic transaction, short timeout and dedicated identity, then chart probe success with endpoint state.
Example commands: replace lab values and confirm permissions and software versions before use.
for r in 1.1.1.1 8.8.8.8 9.9.9.9; do
date -u +%FT%TZ
dig @$r app.example.com A +noall +answer
done
curl -fsS --connect-timeout 3 https://app.example.com/ready
Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Failover never triggers | Probe path, expected code or source firewall may be wrong. | Correlate provider probe logs, endpoint access logs and profile state. |
| DNS changed but users reach the old region | A resolver or client may cache the answer, or an existing session remains. | Query multiple resolvers and measure connection age and application DNS refresh. |
| Both targets become unhealthy | A shared dependency or overly deep check may erase regional independence. | Separate dependency signals and record the provider's all-unhealthy answer. |
Acceptance checks
- Write RTO and data-consistency goals per endpoint.
- Document probe path, status range, timeout and thresholds.
- Measure TTL and actual resolver cache behavior.
- Test all-unhealthy and blocked-probe scenarios.
- Validate manual override and automatic failback.
- Run a game day through three resolvers and real requests.
Related concepts
DNS caching and dependencies
DNS results depend on client and recursive-resolver caches as well as authoritative data. Changing a TTL does not retroactively shorten already cached answers. A/AAAA, CNAME, MX and TXT records serve different purposes. Test internal and external views separately; split DNS may intentionally return different results. Record resolver, response type, TTL and query time when troubleshooting. Successful access by IP address alone does not prove correct DNS configuration.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Failover and failback
Failover moves service to another component; failback returns it to the preferred location. Their risks and sequencing may differ. DNS updates, routing, sessions, replication lag and application dependencies determine user interruption. Define failover triggers, false-alarm behavior and approval for returning service. Simply shutting down a VM does not test every failure type. Measure network, storage and management-plane failures separately.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
RTO and end-to-end recovery time
RTO specifies how soon a service must become usable after an incident. Downloading a backup or booting a VM accounts for only part of that duration. Record detection, approval, infrastructure preparation, data transfer, application startup and business validation separately. Define exactly what starts and stops the test clock. The same technical restore duration can produce different business interruptions when dependencies or access approvals introduce additional waiting.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.