Troubleshooting Kerberos issues

Each web server has its defaults regarding maximal allowed HTTP request header size. Since a Kerberos ticket can be very large and is sent within the HTTP request headers, the default threshold can be triggered easily. Besides that, users with many group memberships send a much bigger Kerberos ticket, which could lead to issues where some users can access the website while others cannot.

Observed behavior:

• The client sends the Kerberos ticket within the HTTP request in the header Authorization: Negotiate YIIYX... (> 5000 characters).
• The client receives an HTTP response with an HTTP status code 400 (Bad request).
• There are no log entries in the web application log showing that the client has sent this request. This means:
• On Airlock Gateway (WAF): There is no SUMMARY log message for this request.
• On Airlock IAM: There is no log message in the Loginapp log for this request.

Possible reasons:

• On Airlock Gateway (WAF): The maximal HTTP request header size is not configured, so the default settings are active or the configured setting is still too small. Check the step "HTTP request header size" outlined in the chapter Gateway (WAF) Configuration.
• On Airlock IAM: The default allowed HTTP request header size is configured to 16384 bytes. In case this is insufficient, do the following:
• Edit the file instances/<INSTANCE_NAME>/instance.properties
• Adapt the setting iam.webserver.header.bytes.max to the appropriate value
• Restart the instance to make this change active.

Kerberos has strict timing requirements. If time is not synchronized within tight limits, authentication may fail.

Please verify that the time is synchronized on all involved systems:

• Airlock IAM
• Active Directory Domain Controller
• Windows Client

A log message with the information Clock skew too great indicates that the time is not synchronized properly between the mentioned systems.

In case the client tries to authenticate with NTLM over SPNEGO, the Authorization header sent by the client starts with Authorization: Negotiate TlRMTVNT...

Possible reasons:

• The client tries to get a Kerberos service ticket for the FQDN he is accessing. If the corresponding SPN is not registered within the Active Directory, the client might send such an authorization header.
• If the SPN is registered to more than one object (user or machine account).
• The client does not support Kerberos or is not configured correctly.

Observed behavior:

• Login does not work, the user is displayed an error page.
• IAM logs a message like "Client sent NTLM authorization ticket instead of using Kerberos, check the client setup."

Useful checks:

• Run the PowerShell commands below as Domain Administrator on the Active Directory Domain Controller to figure out whether the SPN is registered or not:
copy



# Example for an SPN which is registered.
PS C:\> setspn -Q HTTP/a.airlock.com
Checking domain DC=airlock,DC=com
CN=syskerb-airlock-a,CN=Users,DC=airlock,DC=com
        HTTP/a.airlock.com

Existing SPN found!

# Example for an SPN which is not registered.
PS C:\> setspn -Q HTTP/b.airlock.com
Checking domain DC=airlock,DC=com

No such SPN found.

• Run the PowerShell commands below as Domain Administrator on the Active Directory Domain Controller to search for duplicate SPNs:


# Search domain wide for duplicated SPNs
PS C:\> setspn -X -T *
Checking domain DC=airlock,DC=com
Processing entry 0
found 0 group of duplicate SPNs.

# Search forest wide for duplicated SPNs
PS C:\> setspn -X -F
Checking forest DC=airlock,DC=com
Operation will be performed forestwide, it might take a while.
Processing entry 0
found 0 group of duplicate SPNs.

• Run the PowerShell commands below as a user on the Windows Client to check if a Kerberos ticket for the designated service could be retrieved or not:


# Delete all tickets from the Kerberos ticket cache before testing.
PS C:\> klist purge

Current LogonId is 0:0x48fc2
        Deleting all tickets:
        Ticket(s) purged!

# Request a Kerberos service ticket for the service HTTP/a.airlock.com        
PS C:\> klist get HTTP/a.airlock.com

Current LogonId is 0:0x48fc2
A ticket to HTTP/a.airlock.com has been retrieved successfully.

Cached Tickets: (2)

#0>     Client: tmueller @ AIRLOCK.COM
        Server: krbtgt/AIRLOCK.COM @ AIRLOCK.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
        Start Time: 9/18/2018 14:21:28 (local)
        End Time:   9/19/2018 0:21:28 (local)
        Renew Time: 9/25/2018 14:21:28 (local)
        Session Key Type: RSADSI RC4-HMAC(NT)
        Cache Flags: 0x1 -> PRIMARY
        Kdc Called: dc1.airlock.com

#1>     Client: tmueller @ AIRLOCK.COM
        Server: HTTP/a.airlock.com @ AIRLOCK.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
        Start Time: 9/18/2018 14:21:28 (local)
        End Time:   9/19/2018 0:21:28 (local)
        Renew Time: 9/25/2018 14:21:28 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0
        Kdc Called: dc1.airlock.com

The developer tools from the browser show that no authorization header is sent to Airlock IAM in the HTTP request. Most likely the browser configuration is not correct.

When accessing the website in the browser, a pop-up is asking the user for his credentials. After entering those, authentication is successful. The browser's developer tools show that the header Authorization: Negotiate ... is sent to Airlock IAM and the IAM logs confirm that authentication was done with Kerberos. However, the pop-up is shown because the browser is not configured as mentioned in Browser configuration.

The browser will always request a Kerberos service ticket for an SPN with the exact FQDN used in the browser.

This results in the following requirements:

• Always access the website using the fully qualified domain name (FQDN). Never use shorthand or the IP address.
• Ensure that a DNS A-record is configured on the DNS server for the FQDN.
• It is highly recommended only to configure DNS A-records and not CNAME entries. Aliases (CNAME entries) result in the Active Directory Domain Controller issuing Kerberos service tickets for the A-record the CNAME is referring to.

The keytab file contains for each SPN/encryption type pair the corresponding key material in order to decrypt the Kerberos ticket.

The command below illustrates how the content of the keytab file can be checked:

copy

airlock@iam:/home/airlock/iam/instances/auth$ klist -e -k airlock.com.keytab
Keytab name: FILE:airlock.com.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 HTTP/a.airlock.com@AIRLOCK.COM (aes256-cts-hmac-sha1-96)
   3 HTTP/b.airlock.com@AIRLOCK.COM (aes256-cts-hmac-sha1-96)

A keytab file with invalid key material always results in a failed authentication. However, the cause and the part of what is invalid could differ every time. To figure out what is wrong, analyze the Kerberos ticket sent by the client and compare it with the information in the keytab file (in main focus are KVNO, SPN, and encryption type).

Possible causes:

• KVNO (Key Version Number)
• The Kerberos ticket issued by the Active Directory Domain Controller is encrypted with the Kerberos system user's password (see chapter ). Whenever the Kerberos system user's password is changed or a new key tab file is created (see chapter ), the Active Directory Domain Controller increments the KVNO of this user's password internally. When a user requests a Kerberos service ticket from the Domain Controller, the issued ticket always contains the KVNO used to encrypt the ticket (indicating which password has been used). Normally the KVNO in the key tab file is identical to the one of the Kerberos ticket received by the client since the key tab file and the Kerberos ticket are created with the same key material. A different KVNO could be an indication that probably the Kerberos system user's password has been altered and the key tab file contains outdated key material.
• Nevertheless, it is not a requirement that the KVNO is identical, for as long as the same password is used, Kerberos authentication will work without issues (e.g. resetting the password to the same value would increment the KVNO but the password is still the same). A log message with the information "Mechanism level: Checksum failed" means that the key material used by the Active Directory Domain Controller and the one in the key tab file are different. If unsure whether the key material is still valid, simply re-create the key tab file (see Create the Keytab File).

• SPN and encryption type
• Check in the Kerberos ticket sent by the client for which SPN it was issued. Possible reasons for failed authentication with a log message like "Cannot find key of appropriate type to decrypt AP-REQ - AES256 CTS mode with HMAC SHA1-96" could be:
• The particular SPN is not configured in the key tab file.
• The particular SPN is configured in the key tab file with a different encryption type.
• The particular SPN does not match the Service Principal in the IAM Configuration (depending on the authentication flow that would be in the plugin "Kerberos Config" or "SPNEGO Config").

Useful checks:

• Test whether it is possible on Airlock IAM to retrieve a Kerberos TGT ticket from the Active Directory Domain Controller using the keytab file (network connection on 88/udp and 88/tcp is required).
copy



airlock@iam:/home/airlock/iam/instances/auth$ kdestroy

airlock@iam:/home/airlock/iam/instances/auth$ kinit HTTP/a.airlock.com@AIRLOCK.COM -k -t airlock.com.keytab
airlock@iam:/home/airlock/iam/instances/auth$ klist -e
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: HTTP/a.airlock.local@AIRLOCK.COM

Valid starting     Expires            Service principal
09/06/18 16:17:02  09/07/18 02:17:02  krbtgt/AIRLOCK.COM@AIRLOCK.COM
        renew until 09/07/18 16:17:01, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96



In case that kinit reports "kinit: Resource temporarily unavailable while getting initial credentials", ensure that the kdc configured in the krb5.conf file is accessible, the hostname can be resolved and network connectivity to the kdc via 88/udp and 88/tcp is possible.

The Kerberos ticket cache on the client side could lead to old Kerberos tickets being sent to Airlock IAM. This can be very confusing, especially if the system user's password or the encryption type for the SPN has changed and still the old Kerberos ticket is received on Airlock IAM side. Run the command  klist tickets  on the client in the PowerShell to see which tickets there are in the Kerberos ticket cache. To clear the ticket cache, run  klist purge

During integration, it is highly recommended to clear the Kerberos ticket cache on the client before testing.