Troubleshooting · Cybersecurity
Firewall Asymmetric Routing: Session and Return-Path Diagnosis
A stateful firewall tracks a SYN-initiated session with direction and interface information. If the SYN-ACK returns through another firewall, that device may have no session entry.…
Technical review:
Architecture and operating model
A stateful firewall tracks a SYN-initiated session with direction and interface information. If the SYN-ACK returns through another firewall, that device may have no session entry. Incorrect NAT, policy routing and strict reverse-path checks can produce similar symptoms.
Use synchronized captures at both endpoints and the firewall ingress/egress interfaces. Correlate the five-tuple, translated address, route lookup and session ID. HA alone does not prove session synchronization for every traffic type.
Reproduce the fault with one test source and destination. Make narrowly scoped routing/NAT changes. Disabling stateful inspection or reverse-path protection globally may restore traffic while hiding the routing defect and weakening security.
- 1Client SYN
- 2Outbound route and NAT
- 3Server SYN-ACK
- 4Return session matching
Design parameters
- Flow key
- Record source/destination IP, ports and protocol before and after NAT.
- ECMP and PBR
- Test assumptions about forward/reverse hash or policy decisions using actual captures.
- HA session scope
- Measure TCP, UDP and NAT state synchronization separately during node changes.
Worked example
For client 192.0.2.10:51000 and server 198.51.100.20:443, compare both firewalls when SYN crosses FW-A and SYN-ACK crosses FW-B. Test a pilot PBR rule scoped to this flow to return via FW-A, then correct the permanent topology.
Example commands: replace lab values and confirm permissions and software versions before use.
ip route get 198.51.100.20 from 192.0.2.10
sysctl net.ipv4.conf.all.rp_filter
tcpdump -ni eth0 "host 198.51.100.20 and tcp port 443"Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Repeated SYNs | Wrong return path or no destination listener. | Verify SYN and SYN-ACK at both endpoints. |
| Only fails after failover | Missing session/NAT synchronization. | Test old and new sessions separately. |
| Some sources work | PBR, uRPF or subnet-specific NAT differences. | Compare route and policy decisions for two sources to one destination. |
Acceptance checks
- Prove both paths with captures.
- Match pre/post-NAT addresses.
- Check policy order.
- Test existing sessions during failover.
- Prepare narrowly scoped rollback.
- Validate the fix while retaining stateful inspection.
Related concepts
Layer 2 and Layer 3 boundaries
A VLAN creates a separate broadcast domain; routing and access policies govern communication between VLANs. Review trunk allowed lists, access-port assignment and gateway placement together. A network diagram should show where packets are routed and filtered, not merely which cables connect devices. Sharing a switch does not require sharing privileges. When access fails, confirm VLAN and addressing first, then gateway, route and policy matching.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behaviour. A log describes an event, a metric shows behaviour over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Failover and failback
Failover moves service to another component; failback returns it to the preferred location. Their risks and sequencing may differ. DNS updates, routing, sessions, replication lag and application dependencies determine user interruption. Define failover triggers, false-alarm behaviour and approval for returning service. Simply shutting down a VM does not test every failure type. Measure network, storage and management-plane failures separately.
Trust boundary and failure domain
A trust boundary separates components governed by different access decisions; a failure domain groups resources that one event can affect together. Two VLANs do not create a strong trust boundary when routing between them is unrestricted. Backups in different folders still share a failure domain if one administrator can delete both. Assess physical location, identity provider, management account, network path and power source separately. Test which access paths and recovery options remain available when a component is lost.
Availability versus recovery
High availability aims to keep service running through specified failures with a short interruption; backup recovers lost or corrupted data from an earlier point. A cluster can replicate an accidental deletion to another node. HA therefore does not replace backup. Consider DNS, identity, network, storage and power dependencies together. Successful node failover is insufficient by itself: measure user sessions, application writes and external integrations after the transition as well.
IT/OT security zones
A zone groups assets with similar operational and security requirements; communications between zones should traverse controlled conduits. An accessible OT device is not necessarily safe to scan or update. Control latency, safety, vendor support and maintenance windows influence decisions. Observe actual PLC and SCADA flows rather than copying IT policies unchanged. Define identity, duration, approval and logging requirements for remote support, and coordinate active tests and cutovers with production owners.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.