Bug 1484451

Summary: nsupdate fails with GSSAPI
Product: [Fedora] Fedora Reporter: Lukas Slebodnik <lslebodn>
Component: bindAssignee: Petr Menšík <pemensik>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: lslebodn, marka, mruprich, msehnout, pemensik, ssorce, thozza, vonsch, zdohnal
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: bind-9.11.1-2.P3.fc26 bind-9.11.1-7.P3.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-17 22:52:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
output of KRB5_TRACE=/dev/stderr nsupdate -D -d -g < add_remove.msg >nsupdate_f26.log 2>&1
none
nsupdate_f26.log
none
nsupdate_f25.log
none
nsupdate_f26_debug.log
none
nsupdate_f25_debug.log
none
reproducer.sh none

Description Lukas Slebodnik 2017-08-23 15:41:40 UTC
Created attachment 1317146 [details]
output of KRB5_TRACE=/dev/stderr nsupdate -D -d -g < add_remove.msg >nsupdate_f26.log 2>&1

Description of problem:
I was debugging unrelated BZ with ssd on fedora 26 and I noticed that dns records
were not updated on f26 even thought it works on f26. f26 and f25 have the same version of sssd and sssd only call nsupdate for updating dns records in AD/IPA.
 
Version-Release number of selected component (if applicable):
sh# rpm -q bind-utils
bind-utils-9.11.1-2.P2.fc26.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
1. # get krb5 ticket which can be used for updating DNS records in AD or another
   # DNS server which supports update via gssapi
2. create dns update message add_remove.msg (e.g.
  realm SSSDAD2012R2.COM
  update delete fserver-037334.sssdad2012r2.com. in A
  update add fserver-037334.sssdad2012r2.com. 3600 in A 10.16.41.18
  send

3. nsupdate -g < add_remove.msg || echo bad

Actual results:
sh# cat add_remove.msg 
realm SSSDAD2012R2.COM
update delete fserver-037334.sssdad2012r2.com. in A
update add fserver-037334.sssdad2012r2.com. 3600 in A 10.16.41.18
send

sh# klist 
Ticket cache: FILE:/tmp/ccache_SSSDAD2012R2.COM
Default principal: FSERVER-2017334$@SSSDAD2012R2.COM

Valid starting       Expires              Service principal
08/23/2017 11:18:31  08/23/2017 21:18:31  krbtgt/SSSDAD2012R2.COM@SSSDAD2012R2.COM
        renew until 08/30/2017 11:18:30
08/23/2017 11:18:31  08/23/2017 21:18:31  ldap/fire.sssdad2012r2.com@SSSDAD2012R2.COM
        renew until 08/30/2017 11:18:30
08/23/2017 11:21:35  08/23/2017 21:18:31  DNS/fire.sssdad2012r2.com@SSSDAD2012R2.COM
        renew until 08/30/2017 11:18:30

sh# nsupdate -V
nsupdate 9.11.1-P2-RedHat-9.11.1-2.P2.fc26

sh# rpm -q bind-utils
bind-utils-9.11.1-2.P2.fc26.x86_64

sh# nsupdate -g < add_remove.msg || echo bad
; TSIG error with server: tsig verify failure
update failed: REFUSED
bad

sh# dig fserver-037334.sssdad2012r2.com. +short

Expected results:
sh# dig fserver-037334.sssdad2012r2.com. +short
10.16.41.18

Additional info:

Comment 1 Lukas Slebodnik 2017-08-23 15:45:18 UTC
It works on fedora26 with nsupdate from fedora25 (yes it is not very trivail to achieve it :-)

sh# cat add_remove.msg 
realm SSSDAD2012R2.COM
update delete fserver-037334.sssdad2012r2.com. in A
update add fserver-037334.sssdad2012r2.com. 3600 in A 10.16.41.18
send
sh# klist 
Ticket cache: FILE:/tmp/ccache_SSSDAD2012R2.COM
Default principal: FSERVER-037334$@SSSDAD2012R2.COM

Valid starting       Expires              Service principal
08/23/2017 11:18:31  08/23/2017 21:18:31  krbtgt/SSSDAD2012R2.COM@SSSDAD2012R2.COM
        renew until 08/30/2017 11:18:30
08/23/2017 11:18:31  08/23/2017 21:18:31  ldap/fire.sssdad2012r2.com@SSSDAD2012R2.COM
        renew until 08/30/2017 11:18:30
08/23/2017 11:21:35  08/23/2017 21:18:31  DNS/fire.sssdad2012r2.com@SSSDAD2012R2.COM
        renew until 08/30/2017 11:18:30

sh# KRB5_TRACE=/dev/stderr nsupdate -d -g < add_remove.msg >nsupdate_f25.log 2>&1  || echo bad

sh# nsupdate -V
nsupdate 9.10.5-P2-RedHat-9.10.5-2.P2.fc25

sh# dig fserver-037334.sssdad2012r2.com +short
10.16.41.18

Comment 2 Lukas Slebodnik 2017-08-23 15:51:55 UTC
Created attachment 1317147 [details]
nsupdate_f26.log

Output of:
KRB5_TRACE=/dev/stderr nsupdate -D -d -g < add_remove.msg |& tee nsupdate_f26.log

Comment 3 Lukas Slebodnik 2017-08-23 15:54:11 UTC
Created attachment 1317148 [details]
nsupdate_f25.log

Attaching also output of "nsupdate -D -d -g" for nsupdate-9.10.5-P2-RedHat-9.10.5-2.P2

Comment 5 Simo Sorce 2017-08-23 16:53:50 UTC
Do I read the logs right? The signature is broken with the new nsupdate tool ?
Was anything changed in how TSIG is handled in latest versions ?

Comment 6 Petr Menšík 2017-08-23 19:10:23 UTC
Thanks for the report. What is the version of the server, that nsupdate runs against? Is it always the same version? Is it possible to obtain kerberos debug info also from server on failure? Is that server bind with IPA?

I could not find any relevant change that were not already in 9.10.5-2.P2 version of Fedora 25. The only change I can remember is change of default value of dns_canonicalize_hostname in krb5.conf. I guess that is not this case.

Comment 8 Lukas Slebodnik 2017-08-24 08:52:56 UTC
(In reply to Petr Menšík from comment #6)
> Thanks for the report. What is the version of the server, that nsupdate runs
> against?
Server was Active Directory 2012-r2.
 
> Is it always the same version?

Yes

> Is it possible to obtain kerberos
> debug info also from server on failure?
>
nsupdate_f26.log contains client krb5 debug logs. I just forgot to include in in nsupdate_f25.log. 

> Is that server bind with IPA?
I didn't try to reproduce with freeIPA. 

> I could not find any relevant change that were not already in 9.10.5-2.P2
> version of Fedora 25. The only change I can remember is change of default
> value of dns_canonicalize_hostname in krb5.conf. I guess that is not this
> case.

I do not think it is related to any other change in f26
because nsupdate from f25 on f26 works well.

Comment 9 Petr Menšík 2017-08-24 13:49:09 UTC
First of all, thanks for a great reproducer. I could use git bisect against upstream sources to find more.

Upstream sources are affected as well, it is not packaging error of Fedora.
Branches v9_10 are ok, passing without trouble. v9_11 is broken, it fails to authenticate. master are ok again. It does not matter whether they are built against OpenSSL 1.1 or OpenSSL 1.0.

According to my bisecting, failing commit is this:
https://source.isc.org/cgi-bin/gitweb.cgi?p=bind9.git;a=commit;h=a8da00ef95ba37b9d071c2b8db1a0c967e060106

This commit is not yet present in v9_10 branch. I am working on to find correct commit for fix missing in v9_11 branch

Comment 10 Petr Menšík 2017-08-24 14:16:35 UTC
Problem is fixed by commit:
https://source.isc.org/cgi-bin/gitweb.cgi?p=bind9.git;a=commit;h=03be5a6b4e6311b14a12dec5b15a62f55586aaf4

Unfortunately it is quite big change. According to commit message, there was no intention to fix anything.

Comment 11 Petr Menšík 2017-08-24 15:08:16 UTC
I think I missed important detail with -L 10:
24-Aug-2017 10:56:13.904 GSS verify error: GSSAPI error: Major = A token had an 
invalid Message Integrity Check (MIC), Minor = Packet was replayed in wrong dire
ction.
24-Aug-2017 10:56:13.904 tsig key '1729469141.sig-jetfire.sssdad2012r2.com' (<nu
ll>): signature failed to verify(1)
; TSIG error with server: tsig verify failure

Bug reported as 
[ISC-Bugs #45854], https://bugs.isc.org/Public/Bug/Display.html?id=45854

Would be accessible when made public.

Comment 12 Petr Menšík 2017-08-24 15:16:31 UTC
Created attachment 1317765 [details]
nsupdate_f26_debug.log

Comment 13 Petr Menšík 2017-08-24 15:17:14 UTC
Created attachment 1317767 [details]
nsupdate_f25_debug.log

Comment 14 Petr Menšík 2017-08-24 15:22:46 UTC
Created attachment 1317769 [details]
reproducer.sh

Comment 15 Mark Andrews 2017-08-25 06:41:05 UTC
The workaround for Windows mishandling of TSIG records got lost
with the case preservation change.  Windows fails to correctly
hash the incoming message.  The owner name of the TSIG record is
supposed to be added *to the hash* in canonical format but this
isn't happening if you send the request with a compressed owner
name.  I suspect that the compressed owner name is being fed into
the hash rather than the expanded form.

RFC 2845, 3.4.2. TSIG Variables

TSIG RR      NAME             Key name, in canonical wire format

All owner names going over the wire are allowed to be compressed.

Mark

Comment 17 Petr Menšík 2017-09-13 18:32:19 UTC
Can you please verify that new build fixes authentication to Active Directory server?

https://koji.fedoraproject.org/koji/taskinfo?taskID=21847011

Comment 18 Lukas Slebodnik 2017-09-15 12:38:46 UTC
(In reply to Petr Menšík from comment #17)
> Can you please verify that new build fixes authentication to Active
> Directory server?
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=21847011

Yes it works. Thank you very much btw bind-utils-9.11.1-6.P3.fc27
probably does not have fix because it does not work

Comment 19 Fedora Update System 2017-09-15 13:16:53 UTC
bind-9.11.1-2.P3.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-af0763ad6f

Comment 20 Fedora Update System 2017-09-15 13:19:05 UTC
bind-9.11.1-7.P3.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-64ae6659ca

Comment 21 Fedora Update System 2017-09-15 16:55:59 UTC
bind-9.11.1-7.P3.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-64ae6659ca

Comment 22 Fedora Update System 2017-09-16 04:25:38 UTC
bind-9.11.1-2.P3.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-af0763ad6f

Comment 23 Fedora Update System 2017-09-17 22:52:06 UTC
bind-9.11.1-2.P3.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 24 Fedora Update System 2017-09-30 06:34:55 UTC
bind-9.11.1-7.P3.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.