Bug 1843926

Summary: satellite-change-hostname fails when running nsupdate
Product: Red Hat Satellite Reporter: Lukas Pramuk <lpramuk>
Component: InstallationAssignee: Ewoud Kohl van Wijngaarden <ekohlvan>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: apatel, jlenz, jomitsch, kgaikwad, pcreech
Target Milestone: 6.8.0Keywords: AutomationBlocker, Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-2.1.0-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 13:03:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukas Pramuk 2020-06-04 12:33:13 UTC 
Description of problem:
On Satellite with DNS capsule satellite-change-hostname fails when running nsupdate:

update failed: REFUSED
update failed: REFUSED

Version-Release number of selected component (if applicable):
Satellite 6.8.0 Snap3
rubygem-foreman_maintain-0.6.4-1.el7sat.noarch

How reproducible:
100% in my setup

Steps to Reproduce:
1. Have satellite set up with DNS Capsule

2. Try to change the hostname

# satellite-change-hostname satellite-new.example.com -y -u admin -p changeme

Checking hostname validity

Checking overall health of server

Checking credentials

Assembling data for DNS update
updating DNS records with nsupdate:
local 127.0.0.1
zone example.com
update add example.com 10800 SOA satellite-new.example.com. root.example.com 2 86400 3600 604800 3600
update add example.com. 3600 IN NS satellite-new.example.com.
update delete example.com. IN NS satellite.example.com
update delete satellite.example.com A
update add satellite-new.example.com 10800 A 192.168.100.1
send

zone 100.168.192.in-addr.arpa
update add 100.168.192.in-addr.arpa 10800 SOA satellite-new.example.com. root.100.168.192.in-addr.arpa 2 86400 3600 604800 3600
update add 100.168.192.in-addr.arpa. 3600 IN NS satellite-new.example.com.
update delete 100.168.192.in-addr.arpa. IN NS satellite.example.com
send
update failed: REFUSED
update failed: REFUSED

Failed 'echo -e "local 127.0.0.1
zone example.com
update add example.com 10800 SOA satellite-new.example.com. root.example.com 2 86400 3600 604800 3600
update add example.com. 3600 IN NS satellite-new.example.com.
update delete example.com. IN NS satellite.example.com
update delete satellite.example.com A
update add satellite-new.example.com 10800 A 192.168.100.1
send

zone 100.168.192.in-addr.arpa
update add 100.168.192.in-addr.arpa 10800 SOA satellite-new.example.com. root.100.168.192.in-addr.arpa 2 86400 3600 604800 3600
update add 100.168.192.in-addr.arpa. 3600 IN NS satellite-new.example.com.
update delete 100.168.192.in-addr.arpa. IN NS satellite.example.com
send
" | nsupdate -l -k /etc/rndc.key' with exit code 2


Actual results:
satellite-change-hostname fails

Expected results:
satellite-change-hostname succeeds and dns record are updated
Comment 1 Lukas Pramuk 2020-06-04 12:40:24 UTC 
Adding  "update-policy { grant rndc-key zonesub ANY; };" 
to every zone helped to fix the issue
Comment 2 Lukas Pramuk 2020-06-19 07:56:54 UTC 
The very similar issue is manifesting during host creation when it fail to create DNS record for the host:

  Create IPv4 DNS record for host1.example.com task failed with the following error: ERF12-2357 [ProxyAPI::ProxyException]: Unable to set DNS entry ([RestClient::BadRequest]: 400 Bad Request) for Capsule https://sat.example.com:9090/dns


I tracked the issue down and it's the same problem with nsupdate (update failed: REFUSED)

/var/log/foreman-proxy/proxy.log:
---------------------------------------------------
2020-06-18T20:04:32 f9c81df5 [I] Started POST /dns/ 
2020-06-18T20:04:32 f9c81df5 [D] verifying remote client 192.168.100.1 against trusted_hosts ["sat.example.com"]
2020-06-18T20:04:32 f9c81df5 [D] Finished DNS query getresources for 'host1.example.com' in 1.83 ms
2020-06-18T20:04:32 f9c81df5 [D] running /usr/bin/nsupdate -k /etc/rndc.key 
2020-06-18T20:04:32 f9c81df5 [D] nsupdate: executed - server 127.0.0.1
2020-06-18T20:04:32 f9c81df5 [D] nsupdate: executed - update add host1.example.com. 86400 A 192.168.100.165
2020-06-18T20:04:32 f9c81df5 [D] nsupdate: errors
Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id:   6369

;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;example.com.	IN	SOA



;; TSIG PSEUDOSECTION:

rndc-key.		0	ANY	TSIG	hmac-md5.sig-alg.reg.int. 1592525072 300 16 /RDSr6OnJvtUo9oRAovA8Q== 6369 NOERROR 0 



2020-06-18T20:04:32 f9c81df5 [E] Update errors: Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id:   6369

;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;example.com.	IN	SOA



;; TSIG PSEUDOSECTION:

rndc-key.		0	ANY	TSIG	hmac-md5.sig-alg.reg.int. 1592525072 300 16 /RDSr6OnJvtUo9oRAovA8Q== 6369 NOERROR 0 



2020-06-18T20:04:32 f9c81df5 [W] Error details for Update errors: Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id:   6369

;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;example.com.	IN	SOA



;; TSIG PSEUDOSECTION:

rndc-key.		0	ANY	TSIG	hmac-md5.sig-alg.reg.int. 1592525072 300 16 /RDSr6OnJvtUo9oRAovA8Q== 6369 NOERROR 0 


: <Proxy::Dns::Error>: Update errors: Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id:   6369

;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;example.com.	IN	SOA



;; TSIG PSEUDOSECTION:

rndc-key.		0	ANY	TSIG	hmac-md5.sig-alg.reg.int. 1592525072 300 16 /RDSr6OnJvtUo9oRAovA8Q== 6369 NOERROR 0 



/usr/share/foreman-proxy/modules/dns_nsupdate/dns_nsupdate_main.rb:56:in `nsupdate_disconnect'
/usr/share/foreman-proxy/modules/dns_nsupdate/dns_nsupdate_main.rb:18:in `do_create'
/usr/share/foreman-proxy/modules/dns_common/dns_common.rb:37:in `create_a_record'
/usr/share/foreman-proxy/modules/dns/dns_api.rb:25:in `block in <class:Api>'
---------------------------------------------------

Running nsupdate manually ends up with the same error:

# /usr/bin/nsupdate -k /etc/rndc.key
> server 127.0.0.1
> update add host1.example.com. 86400 A 192.168.100.165
> send
update failed: REFUSED
> quit


I dont think we need separete BZ for this another occurence of the nsupdate problem.
Comment 4 Ewoud Kohl van Wijngaarden 2020-06-29 15:57:09 UTC 
Created redmine issue https://projects.theforeman.org/issues/30240 from this bug
Comment 5 Bryan Kearney 2020-06-29 16:05:44 UTC 
Upstream bug assigned to ekohlvan
Comment 6 Bryan Kearney 2020-06-29 16:05:46 UTC 
Upstream bug assigned to ekohlvan
Comment 7 Bryan Kearney 2020-07-04 20:06:01 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/30240 has been resolved.
Comment 8 Lukas Pramuk 2020-07-09 23:12:07 UTC 
VERIFIED.

@Satellite 6.8.0 Snap8
foreman-installer-2.1.0-1.el7sat.noarch


# satellite-change-hostname new-satellite.example.com -y -u admin -p changeme sudo: false

Checking hostname validity

Checking overall health of server

Checking credentials

Assembling data for DNS update
updating DNS records with nsupdate:
local 127.0.0.1
zone example.com
update add example.com 10800 SOA new-satellite.example.com. root.example.com 2 86400 3600 604800 3600
update add example.com. 3600 IN NS new-satellite.example.com.
update delete example.com. IN NS satellite.example.com
update delete satellite.example.com A
update add new-satellite.example.com 10800 A 192.168.100.1
send

zone 100.168.192.in-addr.arpa
update add 100.168.192.in-addr.arpa 10800 SOA new-satellite.example.com. root.100.168.192.in-addr.arpa 2 86400 3600 604800 3600
update add 100.168.192.in-addr.arpa. 3600 IN NS new-satellite.example.com.
update delete 100.168.192.in-addr.arpa. IN NS satellite.example.com
send
updating dynamic zone files...
DNS records updated
updating hostname in /etc/hostname
setting hostname
checking if hostname was changed

Updating default Capsule
Updating installation media paths
stopping services
removing old cert rpms
No Match for argument: satellite.example.com-tomcat*
deleting old certs
backed up /var/www/html/pub to /var/www/html/pub/satellite.example.com-20200709121013.backup
updating hostname in /etc/hosts
updating hostname in foreman installer scenarios
updating hostname in hammer configuration
backing up last_scenario.yaml
removing last_scenario.yaml
re-running the installer
foreman-installer --scenario satellite -v --disable-system-checks --certs-regenerate=true --foreman-proxy-register-in-foreman true
cleaning up temporary files
...

>>> satellite-change-hostname finished successfully
Comment 11 errata-xmlrpc 2020-10-27 13:03:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366