Architecture and operating model
In a private Kubernetes cluster, Pods and nodes do not carry public IPs. An external flow crosses the Pod CNI, node routing/SNAT, subnet route, firewall or policy and an egress component such as Cloud NAT, NAT Gateway or Azure NAT Gateway. Managed service APIs reachable through private endpoints may not require internet NAT, but DNS must resolve the private name correctly.
DNS and egress failures can look alike. Resolve the name from a Pod first, then test TCP/TLS to the returned IP and port. CoreDNS/kube-dns, node resolver, private-zone link or forwarder and upstream resolver are separate layers. A NetworkPolicy that permits UDP 53 but blocks TCP 53 fallback can cause intermittent failures for large answers.
NAT port exhaustion causes loss under high concurrency and long-lived connections. Connection count is replicas times destinations times concurrent sockets, amplified by retries. GKE Cloud NAT port metrics, AWS NAT Gateway allocation/connection metrics and Azure SNAT/NAT metrics use different names but answer the same capacity question.
- 1Pod DNS and NetworkPolicy
- 2Node/CNI SNAT and route
- 3Cloud NAT or private endpoint
- 4Target TLS and return path
Design parameters
- Egress model
- Inventory internet NAT, private endpoint and proxy destinations by FQDN and port.
- SNAT port budget
- Calculate concurrent destination connections and retry amplification per node or NAT IP.
- DNS path
- Document cluster DNS, node resolver, private zone and enterprise forwarder chain.
- Policy order
- Inspect CNI NetworkPolicy, cloud firewall/NSG/security group and route one layer at a time.
Platform implementation
Service names are reference points. Scope, defaults, region availability and operating requirements differ; they are not interchangeable guarantees.
| Control | Google Cloud / GKE | AWS / EKS | Azure / AKS |
|---|---|---|---|
| Internet egress | Cloud NAT and private-node/subnet scope | NAT Gateway or required VPC endpoints | Managed/user-assigned NAT Gateway or UDR/firewall |
| Private service access | Private Google Access/PSC and Cloud DNS | Interface/Gateway VPC endpoints and private DNS | Private Endpoint/Private DNS zone and VNet link |
| First capacity evidence | NAT port allocation and dropped packets | ErrorPortAllocation/PacketsDropCount | SNAT/NAT port and flow-log metrics |
Worked example
With 120 Pods, 20 destinations per Pod and four concurrent connections per destination, theoretical demand is 9,600 flows. Three retry waves can spike to 28,800. Validate per-destination port allocation, node distribution and connection reuse instead of assuming one NAT IP is sufficient.
For an image-pull failure, resolve the registry FQDN with `getent`, test TCP/TLS 443 and compare node and Pod results. If the Pod fails but the node succeeds, inspect cluster DNS and policy. If both resolve but TLS fails, move to route, NAT, firewall or proxy.
Example commands: replace lab values and confirm permissions and software versions before use.
kubectl run netcheck --rm -it --restart=Never --image=busybox:1.36 -- sh
getent hosts registry.example.com
nc -vz registry.example.com 443
cat /etc/resolv.conf
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl get networkpolicy -ATroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| DNS timeout | CoreDNS capacity, UDP/TCP 53 policy or an upstream forwarder may be failing. | Compare Pod and node resolution, CoreDNS logs/metrics and TCP 53. |
| Connections fail under load | NAT ports may be exhausted or retries may amplify demand. | Correlate NAT allocation/drop metrics with application connections and retries. |
| Only a managed API is unreachable | Private-endpoint DNS, route or service-endpoint scope may be incomplete. | Validate resolved IP and the private zone, VPC endpoint or VNet link. |
Acceptance checks
- Record DNS results separately from a Pod and node.
- Map required FQDNs and ports to an egress method.
- Test UDP and TCP 53 under NetworkPolicy.
- Calculate NAT capacity for normal and retry load.
- Validate routes and private-DNS links on GKE, EKS and AKS.
- Compare DNS errors, NAT drops and p95 connect time after the fix.
Related concepts
DNS caching and dependencies
DNS results depend on client and recursive-resolver caches as well as authoritative data. Changing a TTL does not retroactively shorten already cached answers. A/AAAA, CNAME, MX and TXT records serve different purposes. Test internal and external views separately; split DNS may intentionally return different results. Record resolver, response type, TTL and query time when troubleshooting. Successful access by IP address alone does not prove correct DNS configuration.
Layer 2 and Layer 3 boundaries
A VLAN creates a separate broadcast domain; routing and access policies govern communication between VLANs. Review trunk allowed lists, access-port assignment and gateway placement together. A network diagram should show where packets are routed and filtered, not merely which cables connect devices. Sharing a switch does not require sharing privileges. When access fails, confirm VLAN and addressing first, then gateway, route and policy matching.
IT/OT security zones
A zone groups assets with similar operational and security requirements; communications between zones should traverse controlled conduits. An accessible OT device is not necessarily safe to scan or update. Control latency, safety, vendor support and maintenance windows influence decisions. Observe actual PLC and SCADA flows rather than copying IT policies unchanged. Define identity, duration, approval and logging requirements for remote support, and coordinate active tests and cutovers with production owners.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.