Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1998917

Summary: rabbitmq cert and keys are injected incorretly after certmonger regenerates them
Product: Red Hat OpenStack Reporter: Julia Marciano <jmarcian>
Component: puppet-tripleoAssignee: OSP Team <rhos-maint>
Status: CLOSED ERRATA QA Contact: David Rosenfeld <drosenfe>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: dciabrin, jhajyahy, jjoyce, jschluet, lmiccini, slinaber, tvignaud
Target Milestone: z2Keywords: Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: puppet-tripleo-11.7.0-2.20211224004900.be47189.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1999702 (view as bug list) Environment:
Last Closed: 2022-03-23 22:11:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1999702    

Description Julia Marciano 2021-08-30 00:32:05 UTC 
Description of problem:
1.When certmonger resubmit a certificate to the IPA server, it calls a post_save script that fails to inject the updated certificate and key because it fetches an invalid hiera key.

2. 'podman cp'command is still being used in certmonger-rabbitmq-refresh.sh, certmonger-redis-refresh.sh (which is not desired, according to https://bugzilla.redhat.com/show_bug.cgi?id=1935621).

Please see the run of post_save script:
[root@controller-0 ~]# /usr/bin/certmonger-rabbitmq-refresh.sh
++ hiera -c /etc/puppet/hiera.yaml container_cli docker
+ container_cli=podman
++ podman ps '--format={{.Names}}'
++ grep -w -E 'rabbitmq(-bundle-.*-[0-9]+)?'
+ container_name=rabbitmq-bundle-podman-1
++ hiera -c /etc/puppet/hiera.yaml tripleo::rabbitmq::service_certificate.service_certificate
Traceback (most recent call last):
	9: from /bin/hiera:246:in `<main>'
	8: from /usr/share/ruby/vendor_ruby/hiera.rb:116:in `lookup'
	7: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup'
	6: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `each'
	5: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `block in lookup'
	4: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `catch'
	3: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:279:in `block (2 levels) in lookup'
	2: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `qualified_lookup'
	1: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `each'
/usr/share/ruby/vendor_ruby/hiera/backend.rb:329:in `block in qualified_lookup': Hiera type mismatch: Got String when a hash-like object was expected to access value using 'service_certificate' from key 'tripleo::rabbitmq::service_certificate.service_certificate' (Exception)
+ service_crt=
++ hiera -c /etc/puppet/hiera.yaml tripleo::rabbitmq::service_certificate.service_key
Traceback (most recent call last):
	9: from /bin/hiera:246:in `<main>'
	8: from /usr/share/ruby/vendor_ruby/hiera.rb:116:in `lookup'
	7: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup'
	6: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `each'
	5: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `block in lookup'
	4: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `catch'
	3: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:279:in `block (2 levels) in lookup'
	2: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `qualified_lookup'
	1: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `each'
/usr/share/ruby/vendor_ruby/hiera/backend.rb:329:in `block in qualified_lookup': Hiera type mismatch: Got String when a hash-like object was expected to access value using 'service_key' from key 'tripleo::rabbitmq::service_certificate.service_key' (Exception)
+ service_key=
+ echo rabbitmq-bundle-podman-1
+ grep -q '^rabbitmq-bundle'
+ tar -c '' ''
+ podman exec -i rabbitmq-bundle-podman-1 tar -C / -xv
tar: Substituting `.' for empty member name
tar: : Cannot stat: No such file or directory
tar: Substituting `.' for empty member name
tar: : Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
+ podman exec rabbitmq-bundle-podman-1 cp /var/lib/kolla/config_files/src-tls ''
cp: -r not specified; omitting directory '/var/lib/kolla/config_files/src-tls'
+ podman exec -u root rabbitmq-bundle-podman-1 chown rabbitmq:rabbitmq ''
chown: cannot access '': No such file or directory
+ podman exec -u root rabbitmq-bundle-podman-1 chown rabbitmq:rabbitmq ''
chown: cannot access '': No such file or directory
+ podman exec rabbitmq-bundle-podman-1 rabbitmqctl eval 'ssl:clear_pem_cache().'
ok


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Damien Ciabrini 2021-08-30 10:03:02 UTC 
Fixed upstream, instructions for testing:

1. check the current certificate for rabbitmq
# md5sum /etc/pki/tls/certs/rabbitmq.crt
921441554988987b997a67978c2e4689  /etc/pki/tls/certs/rabbitmq.crt

2. generate a new certificate with certmonger
#  getcert resubmit -i rabbitmq
Resubmitting "rabbitmq" to "IPA".

3. verify that a new certificate has been retrieved by certmonger on the host
# md5sum /etc/pki/tls/certs/rabbitmq.crt
439789f66ce25332786891aace6f18da  /etc/pki/tls/certs/rabbitmq.crt

4. verify that the cert got injected in the rabbitmq container as expected
# podman exec $(podman ps -q --filter name=rabbitmq) md5sum /etc/pki/tls/certs/rabbitmq.crt
439789f66ce25332786891aace6f18da  /etc/pki/tls/certs/rabbitmq.crt

5. double check that the cert injection uses "tar" instead of "podman cp"
# /usr/bin/certmonger-rabbitmq-refresh.sh
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
etc/pki/tls/certs/rabbitmq.crt
etc/pki/tls/private/rabbitmq.key
ok


Note: it's ok for certmonger-redis-refresh.sh to still use 'podman cp', as it
injects the new certificate into a container managed by paunch (redis_tls_proxy)
and not the redis container managed by pacemaker.
Comment 11 errata-xmlrpc 2022-03-23 22:11:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1001