+++ This bug was initially created as a clone of Bug #1998917 +++

Description of problem:
1.When certmonger resubmit a certificate to the IPA server, it calls a post_save script that fails to inject the updated certificate and key because it fetches an invalid hiera key.

2. 'podman cp'command is still being used in certmonger-rabbitmq-refresh.sh, certmonger-redis-refresh.sh (which is not desired, according to https://bugzilla.redhat.com/show_bug.cgi?id=1935621).

Please see the run of post_save script:
[root@controller-0 ~]# /usr/bin/certmonger-rabbitmq-refresh.sh
++ hiera -c /etc/puppet/hiera.yaml container_cli docker
+ container_cli=podman
++ podman ps '--format={{.Names}}'
++ grep -w -E 'rabbitmq(-bundle-.*-[0-9]+)?'
+ container_name=rabbitmq-bundle-podman-1
++ hiera -c /etc/puppet/hiera.yaml tripleo::rabbitmq::service_certificate.service_certificate
Traceback (most recent call last):
	9: from /bin/hiera:246:in `<main>'
	8: from /usr/share/ruby/vendor_ruby/hiera.rb:116:in `lookup'
	7: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup'
	6: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `each'
	5: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `block in lookup'
	4: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `catch'
	3: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:279:in `block (2 levels) in lookup'
	2: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `qualified_lookup'
	1: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `each'
/usr/share/ruby/vendor_ruby/hiera/backend.rb:329:in `block in qualified_lookup': Hiera type mismatch: Got String when a hash-like object was expected to access value using 'service_certificate' from key 'tripleo::rabbitmq::service_certificate.service_certificate' (Exception)
+ service_crt=
++ hiera -c /etc/puppet/hiera.yaml tripleo::rabbitmq::service_certificate.service_key
Traceback (most recent call last):
	9: from /bin/hiera:246:in `<main>'
	8: from /usr/share/ruby/vendor_ruby/hiera.rb:116:in `lookup'
	7: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup'
	6: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `each'
	5: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `block in lookup'
	4: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `catch'
	3: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:279:in `block (2 levels) in lookup'
	2: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `qualified_lookup'
	1: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `each'
/usr/share/ruby/vendor_ruby/hiera/backend.rb:329:in `block in qualified_lookup': Hiera type mismatch: Got String when a hash-like object was expected to access value using 'service_key' from key 'tripleo::rabbitmq::service_certificate.service_key' (Exception)
+ service_key=
+ echo rabbitmq-bundle-podman-1
+ grep -q '^rabbitmq-bundle'
+ tar -c '' ''
+ podman exec -i rabbitmq-bundle-podman-1 tar -C / -xv
tar: Substituting `.' for empty member name
tar: : Cannot stat: No such file or directory
tar: Substituting `.' for empty member name
tar: : Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
+ podman exec rabbitmq-bundle-podman-1 cp /var/lib/kolla/config_files/src-tls ''
cp: -r not specified; omitting directory '/var/lib/kolla/config_files/src-tls'
+ podman exec -u root rabbitmq-bundle-podman-1 chown rabbitmq:rabbitmq ''
chown: cannot access '': No such file or directory
+ podman exec -u root rabbitmq-bundle-podman-1 chown rabbitmq:rabbitmq ''
chown: cannot access '': No such file or directory
+ podman exec rabbitmq-bundle-podman-1 rabbitmqctl eval 'ssl:clear_pem_cache().'
ok


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Damien Ciabrini on 2021-08-30 10:03:02 UTC ---

Fixed upstream, instructions for testing:

1. check the current certificate for rabbitmq
# md5sum /etc/pki/tls/certs/rabbitmq.crt
921441554988987b997a67978c2e4689  /etc/pki/tls/certs/rabbitmq.crt

2. generate a new certificate with certmonger
#  getcert resubmit -i rabbitmq
Resubmitting "rabbitmq" to "IPA".

3. verify that a new certificate has been retrieved by certmonger on the host
# md5sum /etc/pki/tls/certs/rabbitmq.crt
439789f66ce25332786891aace6f18da  /etc/pki/tls/certs/rabbitmq.crt

4. verify that the cert got injected in the rabbitmq container as expected
# podman exec $(podman ps -q --filter name=rabbitmq) md5sum /etc/pki/tls/certs/rabbitmq.crt
439789f66ce25332786891aace6f18da  /etc/pki/tls/certs/rabbitmq.crt

5. double check that the cert injection uses "tar" instead of "podman cp"
# /usr/bin/certmonger-rabbitmq-refresh.sh
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
etc/pki/tls/certs/rabbitmq.crt
etc/pki/tls/private/rabbitmq.key
ok


Note: it's ok for certmonger-redis-refresh.sh to still use 'podman cp', as it
injects the new certificate into a container managed by paunch (redis_tls_proxy)
and not the redis container managed by pacemaker.