Architecture and operating model
Kerberos proves identity with time-limited tickets instead of sending a user's password to every service. A client obtains a TGT, then requests a service ticket for the target SPN. The KDC maps that SPN to an account and key; the service decrypts the ticket with its copy. DNS names, SPNs, service accounts, keytabs and time therefore form one identity chain.
Repeated credential prompts, KRB_AP_ERR_MODIFIED, Principal unknown and Clock skew too great do not share one cause. A duplicate SPN prevents reliable account selection. A unique SPN on the wrong account can still yield a ticket that the service cannot decrypt. Clock error can make a ticket appear premature or expired; increasing tolerance does not repair synchronization.
Record the requested URL and canonical DNS name first. Prove that the client attempted Kerberos, which SPN it requested and the ticket start and end times. Search the SPN across the forest, then compare its owner with the running service identity and keytab version. Measure NTP or chrony state on client, server and KDC before refreshing caches.
Keep corrections narrow. Before moving an SPN, document its owner, service startup identity and failover nodes. `setspn -S` checks for duplicates while adding and is safer than bypassing that check. On Linux, keytab principal and KVNO must match the KDC. After key rotation, track nodes that still hold an old keytab.
- 1Client and DNS name
- 2TGT and service ticket from KDC
- 3SPN owner and service key
- 4Application session and audit record
Design parameters
- Target identity
- Record the entered FQDN, DNS canonical target and expected service SPN together.
- SPN uniqueness
- Prove that one forest account owns the SPN and that the service runs as that account.
- Time boundary
- Measure UTC differences across client, KDC and server instead of checking only the configured tolerance.
- Key version
- Match keytab principal, encryption type and KVNO with the KDC record.
- Rollback
- Capture the previous owner and a working ticket example before changing an SPN or service identity.
Worked example
If `https://erp.example.com` requests `HTTP/erp.example.com` but two accounts own that SPN, ticket account selection is unreliable. Find owners with `setspn -F -Q`, verify the process identity and remove only the incorrect record. Open a new session, obtain a ticket and correlate the server event in the same time window.
If the client is at 09:00 UTC and the KDC at 09:06 UTC, the six-minute gap exceeds the common five-minute default. Raising tolerance to ten minutes can hide the symptom while weakening time-dependent checks. Repair the time source, synchronization state and hypervisor clock behavior, then test with a new TGT.
Example commands: replace lab values and confirm permissions and software versions before use.
setspn -F -Q HTTP/erp.example.com
setspn -F -X
klist get HTTP/erp.example.com
env KRB5_TRACE=/dev/stdout kvno HTTP/erp.example.com
klist -e
klist -k -e /etc/krb5.keytab
chronyc trackingTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| KRB_AP_ERR_MODIFIED | The KDC may have used another account's key, or the service may hold an old keytab. | Compare SPN owner, process account, KVNO and the keytab on the affected node. |
| Server not found in Kerberos database | The SPN may be missing, use another name or resolve through the wrong realm. | Use KRB5_TRACE to see the requested principal and compare DNS with the SPN query. |
| Clock skew too great | Client, server or KDC time may be outside tolerance. | Record UTC clocks, chrony or NTP source and the last synchronization error. |
| Silent fallback to NTLM | Kerberos failed but another mechanism may have let the application continue. | Inspect the client ticket cache and the authentication package recorded by the server. |
Acceptance checks
- Record the requested FQDN and actual DNS destination.
- Prove the requested SPN with KRB5_TRACE or client events.
- Search the domain and forest for duplicate SPNs.
- Match service identity, keytab principal and KVNO.
- Measure UTC offset across client, KDC and server.
- Validate a new service ticket and application record for one transaction.
Related concepts
Authentication and sessions
Authentication proves who a user or workload is; authorization determines what that identity may do. Successful sign-in does not grant access to every resource. User sessions, service identities, API tokens and device certificates have different lifecycles. Design session duration, token renewal, employee departure, lost-device handling and emergency access alongside initial sign-in. Measure which existing sessions remain usable and which new access attempts are denied when the identity provider becomes unavailable.
Trust boundary and failure domain
A trust boundary separates components governed by different access decisions; a failure domain groups resources that one event can affect together. Two VLANs do not create a strong trust boundary when routing between them is unrestricted. Backups in different folders still share a failure domain if one administrator can delete both. Assess physical location, identity provider, management account, network path and power source separately. Test which access paths and recovery options remain available when a component is lost.
DNS caching and dependencies
DNS results depend on client and recursive-resolver caches as well as authoritative data. Changing a TTL does not retroactively shorten already cached answers. A/AAAA, CNAME, MX and TXT records serve different purposes. Test internal and external views separately; split DNS may intentionally return different results. Record resolver, response type, TTL and query time when troubleshooting. Successful access by IP address alone does not prove correct DNS configuration.
Telemetry and time correlation
Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.
Least privilege and separation of duties
Least privilege limits the permissions needed for a task to specific resources and time periods. Giving everyone the same administrator role complicates access reviews and investigations. Separate daily-use identities from privileged accounts, and restrict interactive use and unnecessary network access for service identities. An access matrix should show which operation an identity may perform on each resource. Test whether an application keeps working after a permission is removed: unnecessary privileges sometimes become visible only during controlled reduction.
Version and support lifecycle
Installability does not prove production support. Review the compatibility chain across OS, application, drivers, extensions and management tools. Update plans should record version, support end, restart needs and rollback methods. A test environment that does not reflect production can produce misleading results. Validate service health and existing workflows after a change, not just version numbers. Remember that pinning a version can also prevent future security fixes.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.