Bug 2014019 - dnsmasq-2.86-2.fc34.x86_64 breaks kinit
Summary: dnsmasq-2.86-2.fc34.x86_64 breaks kinit
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnsmasq
Version: 34
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Petr Menšík
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-14 09:48 UTC by Viktor Ashirov
Modified: 2021-11-08 01:11 UTC (History)
9 users (show)

Fixed In Version: dnsmasq-2.86-3.fc36 dnsmasq-2.86-3.fc35 dnsmasq-2.86-3.fc34
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-31 01:08:43 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
candidate-patch (1.01 KB, patch)
2021-10-14 19:07 UTC, Petr Menšík
no flags Details | Diff

Description Viktor Ashirov 2021-10-14 09:48:02 UTC
Description of problem:

After upgrading to dnsmasq-2.86-2.fc34.x86_64 I'm not able to get kerberos ticket.

Version-Release number of selected component (if applicable):
dnsmasq-2.86-2.fc34.x86_64

How reproducible:
always

Steps to Reproduce:
1. KRB5_TRACE=/dev/stdout kinit $(whoami)@IPA.REDHAT.COM
2.
3.

Actual results:
$ KRB5_TRACE=/dev/stdout kinit vashirov.COM                                                                                            
[70352] 1634029692.133020: Matching vashirov.COM in collection with result: -1765328243/Can't find client principal vashirov
.COM in cache collection
[70352] 1634029692.133021: Resolving unique ccache of type KCM
[70352] 1634029692.133022: Getting initial credentials for vashirov.COM
[70352] 1634029692.133024: Sending unauthenticated request
[70352] 1634029692.133025: Sending request (205 bytes) to IPA.REDHAT.COM
[70352] 1634029692.133026: Sending DNS URI query for _kerberos.IPA.REDHAT.COM.
[70352] 1634029692.133027: No URI records found
[70352] 1634029692.133028: Sending DNS SRV query for _kerberos._udp.IPA.REDHAT.COM.
[70352] 1634029692.133029: Sending DNS SRV query for _kerberos._tcp.IPA.REDHAT.COM.
[70352] 1634029692.133030: No SRV records found
[70352] 1634029692.133031: Retrying AS request with primary KDC
[70352] 1634029692.133032: Getting initial credentials for vashirov.COM
[70352] 1634029692.133034: Sending unauthenticated request
[70352] 1634029692.133035: Sending request (205 bytes) to IPA.REDHAT.COM (primary)
[70352] 1634029692.133036: Sending DNS URI query for _kerberos.IPA.REDHAT.COM.
[70352] 1634029692.133037: No URI records found
[70352] 1634029692.133038: Sending DNS SRV query for _kerberos-master._udp.IPA.REDHAT.COM.
[70352] 1634029692.133039: Sending DNS SRV query for _kerberos-master._tcp.IPA.REDHAT.COM.
[70352] 1634029692.133040: No SRV records found
kinit: Cannot find KDC for realm "IPA.REDHAT.COM" while getting initial credentials
 
 
dnsmasq logs:
Oct 12 11:08:12 dnsmasq[39007]: query[type=256] _kerberos.IPA.REDHAT.COM from 127.0.0.1
Oct 12 11:08:12 dnsmasq[39007]: forwarded _kerberos.ipa.redhat.com to 10.45.248.15
Oct 12 11:08:12 dnsmasq[39007]: reply _kerberos.IPA.REDHAT.COM is NODATA
Oct 12 11:08:12 dnsmasq[39007]: query[SRV] _kerberos._udp.IPA.REDHAT.COM from 127.0.0.1
Oct 12 11:08:12 dnsmasq[39007]: cached _kerberos._udp.IPA.REDHAT.COM is NXDOMAIN
Oct 12 11:08:12 dnsmasq[39007]: query[SRV] _kerberos._tcp.IPA.REDHAT.COM from 127.0.0.1
Oct 12 11:08:12 dnsmasq[39007]: cached _kerberos._tcp.IPA.REDHAT.COM is NXDOMAIN
Oct 12 11:08:12 dnsmasq[39007]: query[type=256] _kerberos.IPA.REDHAT.COM from 127.0.0.1
Oct 12 11:08:12 dnsmasq[39007]: forwarded _kerberos.ipa.redhat.com to 10.45.248.15
Oct 12 11:08:12 dnsmasq[39007]: reply _kerberos.IPA.REDHAT.COM is NODATA
Oct 12 11:08:12 dnsmasq[39007]: query[SRV] _kerberos-master._udp.IPA.REDHAT.COM from 127.0.0.1
Oct 12 11:08:12 dnsmasq[39007]: cached _kerberos-master._udp.IPA.REDHAT.COM is NODATA
Oct 12 11:08:12 dnsmasq[39007]: query[SRV] _kerberos-master._tcp.IPA.REDHAT.COM from 127.0.0.1
Oct 12 11:08:12 dnsmasq[39007]: cached _kerberos-master._tcp.IPA.REDHAT.COM is NODATA


Expected results:
dnsmasq should return DNS entries.

Additional info:
It seems to be an issue with case sensitivity:

This doesn't return any entries:
$ dig srv _kerberos._tcp.IPA.REDHAT.COM

And this does:
$ dig srv _kerberos._tcp.IPA.redhat.com

As a workaround I downgraded to dnsmasq-2.86-2.fc34.x86_64.

Comment 1 Viktor Ashirov 2021-10-14 10:49:04 UTC
I've reproduced this on a clean F34 install with the latest updates.

systemd-resolved has to be disabled:
$ systemctl disable --now systemd-resolved

And NetworkManager uses dnsmasq:

$ cat /etc/NetworkManager/conf.d/50-dns.conf
[main]
dns=dnsmasq

Comment 2 Viktor Ashirov 2021-10-14 11:01:31 UTC
It seems that the problematic patch is this one https://src.fedoraproject.org/rpms/dnsmasq/blob/rawhide/f/dnsmasq-2.86-domain-match-local.patch 
I did a mockbuild without it and dnsmasq now returns entries when an uppercase domain is used.

Comment 3 Viktor Ashirov 2021-10-14 11:20:22 UTC
Here's a copr build that has the patch above commented out: https://copr.fedorainfracloud.org/coprs/vashirov/dnsmasq-bz2014019/

Comment 4 Petr Menšík 2021-10-14 13:30:57 UTC
I tested it on my second laptop and confirm kinit *@REDHAT.COM on our VPN does not work as it should. But it does not seem to be better even with dnsmasq-2.86-1.fc34.x86_64, where patch mentioned in comment #2 were not yet present. Their behavior is different, but both seems to be broken.

According to log-queries output, it mangles case of sent query to lowercase. Then it sends it to redhat.com domain specific resolvers. It seems though reply from them is NOT accepted because of mismatching case.

Then there is a new regression, it sends redhat.com query to normal nameservers without domain specified. This time without mangled case, which makes these queries successful. But public resolvers does not know internal kerberos servers, so the response is NXDOMAIN.

Which makes three problems:

1) original query is mangled to lower case. It would not be usually a problem, but it would break 0x20 bit randomization [1]. Not important.
2) reply with mangled case name is not accepted as a valid response. This is important.
3) non-domain specific resolvers are tried on domain, which has domain-specific resolvers defined. This means privacy issue regression. It did not leak domain specific before.

[1] https://datatracker.ietf.org/doc/html/draft-vixie-dnsext-dns0x20-00

Comment 5 Petr Menšík 2021-10-14 13:50:47 UTC
Reported upstream in thread:
https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q4/015854.html

Comment 7 Petr Menšík 2021-10-14 14:41:58 UTC
Even latest 2.87test4 prerelease has still the same issue. Tested my build at COPR, with log-queries added to /etc/NetworkManager/dnsmasq.d/log.conf

https://copr.fedorainfracloud.org/coprs/pemensik/dnsmasq/

Comment 8 Petr Menšík 2021-10-14 18:07:13 UTC
Oh, I think have I missed important detail. Our dig +tcp SRV _kerberos-master._tcp.*.REDHAT.COM gives response of size 1454. Because kinit does not use EDNS0 extension, it receives truncated UDP packet. TCP request is retried, that is a reason for changed PID in log. But that TCP query selects WRONG server to query. It receives different response than it would via UDP, because different server is queried. TCP query result is NXDOMAIN, because that name is not defined on public internet.

Primary issue is therefore TCP query selects server different way than UDP would.

Comment 9 Petr Menšík 2021-10-14 19:06:20 UTC
Found simple way to reproduce it:

dnsmasq -d --conf-file=/dev/null --port 2053 --server=127.0.0.1 --no-resolv --server='/test/::1' --log-queries &

dig +tcp @localhost -p 2053 srv _tcp.TEST
dig @localhost -p 2053 srv _udp.TEST

Results in log:
dnsmasq: started, version 2.87test4-11-g80fae3c cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile
dnsmasq: using nameserver 127.0.0.1#53
dnsmasq: using nameserver ::1#53 for domain test 
dnsmasq: read /etc/hosts - 16 addresses
dnsmasq: query[SRV] _udp.TEST from ::1
dnsmasq: forwarded _udp.test to ::1
dnsmasq: reply _udp.TEST is NXDOMAIN
dnsmasq: query[SRV] _tcp.TEST from ::1
dnsmasq: forwarded _tcp.TEST to ::1
dnsmasq: reply _tcp.TEST is NXDOMAIN

Note forwarded name differs in case, UDP is forwarded lowercase. But TCP query is forwarded as received without modified case. It then requires case insensitive comparison strcasecmp in order() function in domain-match.c, where strcmp is used now.

Without a patch, previous version would forward it to 127.0.0.1.

Comment 10 Petr Menšík 2021-10-14 19:07:49 UTC
Created attachment 1833100 [details]
candidate-patch

Comment 11 Dmitry Tantsur 2021-10-26 13:52:27 UTC
Hi folks! Is this the same issue:

$ KRB5_TRACE=/dev/stdout kinit -V dtantsur.COM
...
[2508625] 1635256242.418026: Response was from primary KDC
[2508625] 1635256242.418027: Received error from KDC: -1765328359/Additional pre-authentication required
[2508625] 1635256242.418030: Preauthenticating using KDC method data
[2508625] 1635256242.418031: Processing preauth types: PA-PK-AS-REQ (16), PA-PK-AS-REP_OLD (15), PA-PK-AS-REQ_OLD (14), PA-FX-FAST (136), PA-PKINIT-KX (147), PA-FX-COOKIE (133)
[2508625] 1635256242.418032: Received cookie: MIT
[2508625] 1635256242.418033: PKINIT client has no configured identity; giving up
[2508625] 1635256242.418034: Preauth module pkinit (147) (info) returned: 0/Success
[2508625] 1635256242.418035: PKINIT client has no configured identity; giving up
[2508625] 1635256242.418036: Preauth module pkinit (16) (real) returned: 22/Invalid argument
kinit: Pre-authentication failed: Invalid argument while getting initial credentials

There seem to be DNS records above that, so maybe it's another problem?

Comment 12 Fedora Update System 2021-10-27 16:22:02 UTC
FEDORA-2021-bfb01154ce has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-bfb01154ce

Comment 13 Fedora Update System 2021-10-27 16:23:01 UTC
FEDORA-2021-83554ef1c7 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-83554ef1c7

Comment 14 Fedora Update System 2021-10-27 19:00:33 UTC
FEDORA-2021-bfb01154ce has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-bfb01154ce`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-bfb01154ce

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 15 Petr Menšík 2021-10-27 19:16:17 UTC
(In reply to Dmitry Tantsur from comment #11)
> Hi folks! Is this the same issue:
> 
> $ KRB5_TRACE=/dev/stdout kinit -V dtantsur.COM
> ...
> [2508625] 1635256242.418026: Response was from primary KDC
> [2508625] 1635256242.418027: Received error from KDC: -1765328359/Additional
> pre-authentication required
> [2508625] 1635256242.418030: Preauthenticating using KDC method data
> [2508625] 1635256242.418031: Processing preauth types: PA-PK-AS-REQ (16),
> PA-PK-AS-REP_OLD (15), PA-PK-AS-REQ_OLD (14), PA-FX-FAST (136), PA-PKINIT-KX
> (147), PA-FX-COOKIE (133)
> [2508625] 1635256242.418032: Received cookie: MIT
> [2508625] 1635256242.418033: PKINIT client has no configured identity;
> giving up
> [2508625] 1635256242.418034: Preauth module pkinit (147) (info) returned:
> 0/Success
> [2508625] 1635256242.418035: PKINIT client has no configured identity;
> giving up
> [2508625] 1635256242.418036: Preauth module pkinit (16) (real) returned:
> 22/Invalid argument
> kinit: Pre-authentication failed: Invalid argument while getting initial
> credentials
> 
> There seem to be DNS records above that, so maybe it's another problem?

It seems this is different error. This bug is only about inability to find master server. If kerberos fail in pre-authentication, if is already past this test. This error would report Cannot find KDC for realm "IPA.REDHAT.COM", stopping before it. I only guess some parameters or password are wrong.

Comment 16 Fedora Update System 2021-10-28 20:12:08 UTC
FEDORA-2021-83554ef1c7 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-83554ef1c7`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-83554ef1c7

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2021-10-31 01:08:43 UTC
FEDORA-2021-bfb01154ce has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 18 Aleksandar Kostadinov 2021-11-02 09:22:40 UTC
Wrorks for me, thanks!

Comment 19 Fedora Update System 2021-11-08 01:11:39 UTC
FEDORA-2021-83554ef1c7 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.