Architecture and operating model
Stateful nftables rules use the Netfilter conntrack table. NEW describes an initial permitted packet, ESTABLISHED a recognized bidirectional flow and RELATED an associated flow. INVALID is not one failure cause; it can include packets that cannot enter state, fall outside an expected window, are fragmented or lack usable state information.
Troubleshooting correlates firewall counters, conntrack events and packet captures for the same five-tuple. Asymmetric routing, NAT on different nodes, table saturation, timeout, sequence/window validation and fragments are separate hypotheses. Broadly accepting INVALID hides the symptom and expands the security boundary.
In HA designs, conntrackd state synchronization does not guarantee perfect transfer of every flow. On failover, either connections must reconnect or the protocol, NAT and timing limits of state replication need testing. Before changing nftables, export the ruleset and counters atomically and verify a rollback file.
- 1Capture packet and tuple
- 2Conntrack lookup and state
- 3nftables hook and rule decision
- 4Counter, log and application result
Design parameters
- Table capacity
- Monitor nf_conntrack_count/max and insert_failed/drop counters at peak load.
- Timeout
- Measure TCP-state and UDP timeouts against application behavior; arbitrary increases worsen table pressure.
- Path symmetry
- Verify that forward and return packets see the same state domain across routing, ECMP, NAT and HA nodes.
- Log sampling
- Use rate-limited logging with tuple and interface context instead of logging every INVALID packet.
Worked example
With nf_conntrack_max at 262,144 and count at 235,000, utilization is about 89.6%. At 8,000 new short UDP flows per second and a 30-second timeout, the theoretical population is 240,000 records, leaving little room for TCP. Measure timeout distribution and actual flow lifetime first.
If failures occur only when return traffic uses a second WAN, investigate asymmetric routing before rules. Correlate ingress/egress interfaces for the tuple using tcpdump, ip route get, conntrack -E and nft trace; record rp_filter and policy-routing results.
Example commands: replace lab values and confirm permissions and software versions before use.
sudo nft list ruleset
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
sudo conntrack -S
sudo conntrack -E -o timestamp,extended
sudo nft monitor trace
sudo tcpdump -ni any 'host 192.0.2.10 and port 443'Troubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| New connections drop at peak | The conntrack table may be full or inserts may fail. | Graph count/max and nf_conntrack_stat insert_failed/drop values. |
| Only return packets are INVALID | Asymmetric routing, another NAT node or a late packet after timeout may be involved. | Capture both directions and collect conntrack events plus route lookups. |
| Existing sessions break after failover | State/NAT synchronization may be incomplete or outside timing limits. | Compare tuple, mark, zone and NAT translation on active and passive nodes. |
Acceptance checks
- Save nft list ruleset output and a rollback method before changes.
- Baseline nf_conntrack_count/max and error counters.
- Correlate the failing tuple across bidirectional capture and conntrack events.
- Validate NAT, policy routing, rp_filter and ECMP together.
- Use an INVALID log rule with rate limiting in a short pilot.
- After correction, test new, existing and failover sessions separately.
Related concepts
Service lifecycle and dependencies
A running process does not prove that users receive correct service. Review startup order, network or database dependencies, environment variables, file access and health checks together. Restart loops can conceal the root cause. Compare exit codes, recent logs and resource limits, and identify whether systemd or container management triggered the restart. After a controlled restart, validate sessions, writes and dependent services.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Layer 2 and Layer 3 boundaries
A VLAN creates a separate broadcast domain; routing and access policies govern communication between VLANs. Review trunk allowed lists, access-port assignment and gateway placement together. A network diagram should show where packets are routed and filtered, not merely which cables connect devices. Sharing a switch does not require sharing privileges. When access fails, confirm VLAN and addressing first, then gateway, route and policy matching.
Failover and failback
Failover moves service to another component; failback returns it to the preferred location. Their risks and sequencing may differ. DNS updates, routing, sessions, replication lag and application dependencies determine user interruption. Define failover triggers, false-alarm behavior and approval for returning service. Simply shutting down a VM does not test every failure type. Measure network, storage and management-plane failures separately.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.