Bug 2104185

Summary:	Introduction of URI records for kerberos breaks location functionality
Product:	Red Hat Enterprise Linux 8	Reporter:	Tomasz Kepczynski <tomek>
Component:	ipa	Assignee:	Julien Rische <jrische>
Status:	CLOSED ERRATA	QA Contact:	ipa-qe <ipa-qe>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.6	CC:	frenaud, ftrivino, jonmoore, jrische, mescanfe, rcritten, rjeffman, sumenon, tscherf
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ipa-4.9.10-8.module+el8.8.0+17351+9a3fb056	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2148382 (view as bug list)		Environment:
Last Closed:	2023-05-16 08:28:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2111181
Bug Blocks:	2148382
Deadline:	2022-12-12

Description Tomasz Kepczynski 2022-07-05 18:21:34 UTC

Description of problem:
I believe that introduction of URI records for kerberos service discovery breaks location functionality. This is caused by two facts:
- URI records are NOT location aware;
- URI lookups are performed (by default and this is not actively changed by IPA installers) before SRV lookups and take precedence over SRV records (see: https://web.mit.edu/kerberos/krb5-latest/doc/admin/realm_config.html).

Version-Release number of selected component (if applicable):
ipa-client-4.9.8-7.module_el8.6.0+2881+2f24dc92.x86_64
ipa-client-common-4.9.8-7.module_el8.6.0+2881+2f24dc92.noarch
ipa-common-4.9.8-7.module_el8.6.0+2881+2f24dc92.noarch
ipa-healthcheck-0.7-10.module_el8.6.0+2881+2f24dc92.noarch
ipa-healthcheck-core-0.7-10.module_el8.6.0+2881+2f24dc92.noarch
ipa-selinux-4.9.8-7.module_el8.6.0+2881+2f24dc92.noarch
ipa-server-4.9.8-7.module_el8.6.0+2881+2f24dc92.x86_64
ipa-server-common-4.9.8-7.module_el8.6.0+2881+2f24dc92.noarch
ipa-server-dns-4.9.8-7.module_el8.6.0+2881+2f24dc92.noarch
ipa-server-trust-ad-4.9.8-7.module_el8.6.0+2881+2f24dc92.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Perform URI lookup to discover kerberos KDCs:
vesemir:~> dig _kerberos.ipa.jot23.net uri +noall +answer
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:tcp:xavier.jot23.org."
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:udp:zoltan.jot23.org."
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:udp:filippa.jot23.net."
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:udp:triss.jot23.net."
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:tcp:triss.jot23.net."
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:tcp:filippa.jot23.net."
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:tcp:zoltan.jot23.org."
_kerberos.ipa.jot23.net. 86400  IN      URI     0 100 "krb5srv:m:udp:xavier.jot23.org."
2. Please note equal priority of all records. This is the bug.
3. For comparison perform SRV lookup to dicover kerberos KDCs:
vesemir:~> dig _kerberos._udp.ipa.jot23.net srv +noall +answer
_kerberos._udp.ipa.jot23.net. 86400 IN  CNAME   _kerberos._udp.osowa._locations.ipa.jot23.net.
_kerberos._udp.osowa._locations.ipa.jot23.net. 86400 IN SRV 0 100 88 triss.jot23.net.
_kerberos._udp.osowa._locations.ipa.jot23.net. 86400 IN SRV 50 100 88 xavier.jot23.org.
_kerberos._udp.osowa._locations.ipa.jot23.net. 86400 IN SRV 0 100 88 filippa.jot23.net.
_kerberos._udp.osowa._locations.ipa.jot23.net. 86400 IN SRV 50 100 88 zoltan.jot23.org.
4. Please note differing record priorities. This is how it should looku like for URI records as well.

Actual results:
Location functionality is hindered for URI records.

Expected results:
Location functionality works for URI records.

Additional info:
Please note that SRV records resolve through the CNAME record pointing to appropriate records for location. URI lookup resolves directly. This is (likely) caused by a fact that _kerberos URI records are put alongside _kerberos TXT record and CNAME cannot coexist with other record types. This shouldn't affect _kpasswd URI records but apparently it does.

This can be easily resolved by moving _kerberos TXT records to location aware records and making top level _kerberos also a CNAME there.

Comment 1 Tomasz Kepczynski 2022-07-05 18:22:24 UTC

I forgot to mention this has been discovered on AlmaLinux 8.6, not on RHEL.

Comment 2 Julien Rische 2022-07-21 18:08:59 UTC

Is the _kerberos._udp.ipa.jot23.net. CNAME record generated by IPA? I tried to reproduce this, and I don't get this record. But I checked for ipa-server-4.9.8-7 on RHEL 9.0.

For a certain DOMAIN featuring location "a" (with server-a) and location "b" (with server-b), this is what the zone looks like (on server-a):

_kerberos-master._tcp.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos-master._tcp.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos-master._udp.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos-master._udp.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos._tcp.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos._tcp.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos._udp.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos._udp.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos.DOMAIN. 3600 IN TXT "DOMAIN"
_kerberos.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-a.DOMAIN."
_kerberos.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-b.DOMAIN."
_kerberos.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-a.DOMAIN."
_kerberos.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-b.DOMAIN."
_kpasswd._tcp.DOMAIN. 3600 IN SRV 0 100 464 server-a.DOMAIN.
_kpasswd._tcp.DOMAIN. 3600 IN SRV 0 100 464 server-b.DOMAIN.
_kpasswd._udp.DOMAIN. 3600 IN SRV 0 100 464 server-a.DOMAIN.
_kpasswd._udp.DOMAIN. 3600 IN SRV 0 100 464 server-b.DOMAIN.
_kpasswd.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-a.DOMAIN."
_kpasswd.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-b.DOMAIN."
_kpasswd.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-a.DOMAIN."
_kpasswd.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-b.DOMAIN."
_ldap._tcp.DOMAIN. 3600 IN SRV 0 100 389 server-a.DOMAIN.
_ldap._tcp.DOMAIN. 3600 IN SRV 0 100 389 server-b.DOMAIN.
ipa-ca.DOMAIN. 3600 IN A 1.1.1.1

_kerberos-master._tcp.a._locations.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos-master._tcp.a._locations.DOMAIN. 3600 IN SRV 50 100 88 server-b.DOMAIN.
_kerberos-master._tcp.b._locations.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos-master._tcp.b._locations.DOMAIN. 3600 IN SRV 50 100 88 server-a.DOMAIN.
_kerberos-master._udp.a._locations.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos-master._udp.a._locations.DOMAIN. 3600 IN SRV 50 100 88 server-b.DOMAIN.
_kerberos-master._udp.b._locations.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos-master._udp.b._locations.DOMAIN. 3600 IN SRV 50 100 88 server-a.DOMAIN.
_kerberos._tcp.a._locations.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos._tcp.a._locations.DOMAIN. 3600 IN SRV 50 100 88 server-b.DOMAIN.
_kerberos._tcp.b._locations.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos._tcp.b._locations.DOMAIN. 3600 IN SRV 50 100 88 server-a.DOMAIN.
_kerberos._udp.a._locations.DOMAIN. 3600 IN SRV 0 100 88 server-a.DOMAIN.
_kerberos._udp.a._locations.DOMAIN. 3600 IN SRV 50 100 88 server-b.DOMAIN.
_kerberos._udp.b._locations.DOMAIN. 3600 IN SRV 0 100 88 server-b.DOMAIN.
_kerberos._udp.b._locations.DOMAIN. 3600 IN SRV 50 100 88 server-a.DOMAIN.
_kerberos.a._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-a.DOMAIN."
_kerberos.a._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-a.DOMAIN."
_kerberos.a._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:tcp:server-b.DOMAIN."
_kerberos.a._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:udp:server-b.DOMAIN."
_kerberos.b._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-b.DOMAIN."
_kerberos.b._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-b.DOMAIN."
_kerberos.b._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:tcp:server-a.DOMAIN."
_kerberos.b._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:udp:server-a.DOMAIN."
_kpasswd._tcp.a._locations.DOMAIN. 3600 IN SRV 0 100 464 server-a.DOMAIN.
_kpasswd._tcp.a._locations.DOMAIN. 3600 IN SRV 50 100 464 server-b.DOMAIN.
_kpasswd._tcp.b._locations.DOMAIN. 3600 IN SRV 0 100 464 server-b.DOMAIN.
_kpasswd._tcp.b._locations.DOMAIN. 3600 IN SRV 50 100 464 server-a.DOMAIN.
_kpasswd._udp.a._locations.DOMAIN. 3600 IN SRV 0 100 464 server-a.DOMAIN.
_kpasswd._udp.a._locations.DOMAIN. 3600 IN SRV 50 100 464 server-b.DOMAIN.
_kpasswd._udp.b._locations.DOMAIN. 3600 IN SRV 0 100 464 server-b.DOMAIN.
_kpasswd._udp.b._locations.DOMAIN. 3600 IN SRV 50 100 464 server-a.DOMAIN.
_kpasswd.a._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-a.DOMAIN."
_kpasswd.a._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-a.DOMAIN."
_kpasswd.a._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:tcp:server-b.DOMAIN."
_kpasswd.a._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:udp:server-b.DOMAIN."
_kpasswd.b._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:tcp:server-b.DOMAIN."
_kpasswd.b._locations.DOMAIN. 3600 IN URI 0 100 "krb5srv:m:udp:server-b.DOMAIN."
_kpasswd.b._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:tcp:server-a.DOMAIN."
_kpasswd.b._locations.DOMAIN. 3600 IN URI 50 100 "krb5srv:m:udp:server-a.DOMAIN."
_ldap._tcp.a._locations.DOMAIN. 3600 IN SRV 0 100 389 server-a.DOMAIN.
_ldap._tcp.a._locations.DOMAIN. 3600 IN SRV 50 100 389 server-b.DOMAIN.
_ldap._tcp.b._locations.DOMAIN. 3600 IN SRV 0 100 389 server-b.DOMAIN.
_ldap._tcp.b._locations.DOMAIN. 3600 IN SRV 50 100 389 server-a.DOMAIN.

I think the idea here is to have *._locations.DOMAIN. records prioritized based on the server's location, but not for generic records. We rely on the IPA client to use location-aware records. I think for Kerberos services, the KDC is selected by SSSD's krb5 locator plugin[1] (I am not sure if it's using SRV or URI records though).

So basically I think the issue here is more about the presence or this CNAME record. Is your client an actual IPA one, or a vanilla krb5 client?


[1] https://jhrozek.fedorapeople.org/sssd/2.2.0/man/sssd_krb5_locator_plugin.8.html

Comment 3 Tomasz Kepczynski 2022-07-21 19:28:38 UTC

What you are showing is NOT zone view but the output of the "ipa dns-update-system-records" command and it show somehow simplistic and not exact view of how the DNS looks like.

Use "dig _kpasswd._udp.DOMAIN. srv" based on the input you provided and you will see what I see and the output will be different on dns servers in different locations. I am not going into details but this is basically how bind ldap plugin expands "ipalocation" substitution variable. And going back to your example also try "dig _kerberos.DOMAIN. uri" against DNS servers in different locations. The results should differ but they WILL NOT.

The client is IPA client. But client side is NOT relevant to this issue. This is a problem in either IPA itself or bind ldap plugin. Or both. For more specifics see the output of "ipa dnsrecord-find --all DOMAIN _kpasswd._udp" and "ldapsearch".

Please also note what the man page says:

"SSSD's krb5 auth-provider which is used by the IPA and AD providers as well adds the address of the current KDC or domain controller SSSD is using to this file."

What it doesn't say is HOW sssd finds the KDC. And I bet it will start from looking at URI and SRV records. As long as it hits URI records before SRV records (which I suspect is highly likely as this is what kerberos does), all location functionality for kerberos is gone.

Comment 4 Julien Rische 2022-10-03 16:07:24 UTC

There seems to be a CNAME DNS template attribute missing for the _kerberos record object in LDAP indeed.

If we take this setup:

$ ipa location-find
-----------------------
2 IPA locations matched
-----------------------
  Location name: a

  Location name: b
----------------------------
Number of entries returned 2
----------------------------
$ ipa location-show a
  Location name: a
  Servers: server-a.domain.ipa
  Advertised by servers: server-a.domain.ipa
  Servers details:
    Server name: server-a.domain.ipa
    Service weight: 100
    Service relative weight: 100.0%
    Enabled server roles: CA server, DNS server, IPA master, KRA server
$ ipa location-show b
  Location name: b
  Servers: server-b.domain.ipa
  Advertised by servers: server-b.domain.ipa
  Servers details:
    Server name: server-b.domain.ipa
    Service weight: 100
    Service relative weight: 100.0%
    Enabled server roles: CA server, DNS server, IPA master

And apply the following changes:

$ ldapmodify -xW -D 'cn=directory manager' -H ldaps://server-a.domain.ipa <<EOF
dn: idnsname=_kerberos,idnsname=domain.ipa.,cn=dns,dc=domain,dc=ipa
changetype: modify
add: objectClass
objectClass: idnsTemplateObject
-
add: idnsTemplateAttribute;cnamerecord
idnsTemplateAttribute;cnamerecord: _kerberos.\{substitutionvariable_ipalocation\}._locations
EOF

$ ldapmodify -xW -D 'cn=directory manager' -H ldaps://server-a.domain.ipa <<EOF
dn: idnsname=_kerberos.a._locations,idnsname=domain.ipa.,cn=dns,dc=domain,dc=ipa
changetype: modify
add: tXTRecord
tXTRecord: "DOMAIN.IPA"
EOF

We obtain the appropriate priorities in DNS responses:

$ dig @server-a.domain.ipa. _kerberos.domain.ipa. TXT +noall +answer
_kerberos.domain.ipa.   86400   IN      CNAME   _kerberos.a._locations.domain.ipa.
_kerberos.a._locations.domain.ipa. 86400 IN TXT "DOMAIN.IPA"

$ dig @server-a.domain.ipa. _kerberos.domain.ipa. URI +noall +answer
_kerberos.domain.ipa.   86400   IN      CNAME   _kerberos.a._locations.domain.ipa.
_kerberos.a._locations.domain.ipa. 86400 IN URI 50 100 "krb5srv:m:udp:server-b.domain.ipa."
_kerberos.a._locations.domain.ipa. 86400 IN URI 50 100 "krb5srv:m:tcp:server-b.domain.ipa."
_kerberos.a._locations.domain.ipa. 86400 IN URI 0 100 "krb5srv:m:tcp:server-a.domain.ipa."
_kerberos.a._locations.domain.ipa. 86400 IN URI 0 100 "krb5srv:m:udp:server-a.domain.ipa."

$ dig @server-b.domain.ipa. _kerberos.domain.ipa. TXT +noall +answer
_kerberos.domain.ipa.   86400   IN      CNAME   _kerberos.b._locations.domain.ipa.

$ dig @server-b.domain.ipa. _kerberos.domain.ipa. URI +noall +answer
_kerberos.domain.ipa.   86400   IN      CNAME   _kerberos.b._locations.domain.ipa.
_kerberos.b._locations.domain.ipa. 86400 IN URI 0 100 "krb5srv:m:udp:server-b.domain.ipa."
_kerberos.b._locations.domain.ipa. 86400 IN URI 50 100 "krb5srv:m:tcp:server-a.domain.ipa."
_kerberos.b._locations.domain.ipa. 86400 IN URI 50 100 "krb5srv:m:udp:server-a.domain.ipa."
_kerberos.b._locations.domain.ipa. 86400 IN URI 0 100 "krb5srv:m:tcp:server-b.domain.ipa."

Comment 5 Julien Rische 2022-10-04 15:20:08 UTC

Upstream ticket:
https://pagure.io/freeipa/issue/9257

Comment 8 Rafael Jeffman 2022-11-24 00:16:02 UTC

Ustream PR

master: https://pagure.io/freeipa/c/673d2b82d0c92b7016c7c6a7062ce42627bcf380

Comment 9 Florence Blanc-Renaud 2022-11-24 06:45:29 UTC

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/a0652f5dc8efc4580d8039e70c0e762638d3871d

ipa-4-10:
https://pagure.io/freeipa/c/b0d909968bfa323f16aae46f6126abf7625d11e9

Comment 10 Florence Blanc-Renaud 2022-11-24 06:47:00 UTC

Note to QE: test added in ipatests/test_integration/test_upgrade.py::TestUpgrade::test_krb_uri_txt_to_cname

Comment 18 errata-xmlrpc 2023-05-16 08:28:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (idm:client and idm:DL1 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2794