Description of problem: 1.When certmonger resubmit a certificate to the IPA server, it calls a post_save script that fails to inject the updated certificate and key because it fetches an invalid hiera key. 2. 'podman cp'command is still being used in certmonger-rabbitmq-refresh.sh, certmonger-redis-refresh.sh (which is not desired, according to https://bugzilla.redhat.com/show_bug.cgi?id=1935621). Please see the run of post_save script: [root@controller-0 ~]# /usr/bin/certmonger-rabbitmq-refresh.sh ++ hiera -c /etc/puppet/hiera.yaml container_cli docker + container_cli=podman ++ podman ps '--format={{.Names}}' ++ grep -w -E 'rabbitmq(-bundle-.*-[0-9]+)?' + container_name=rabbitmq-bundle-podman-1 ++ hiera -c /etc/puppet/hiera.yaml tripleo::rabbitmq::service_certificate.service_certificate Traceback (most recent call last): 9: from /bin/hiera:246:in `<main>' 8: from /usr/share/ruby/vendor_ruby/hiera.rb:116:in `lookup' 7: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup' 6: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `each' 5: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `block in lookup' 4: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `catch' 3: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:279:in `block (2 levels) in lookup' 2: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `qualified_lookup' 1: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `each' /usr/share/ruby/vendor_ruby/hiera/backend.rb:329:in `block in qualified_lookup': Hiera type mismatch: Got String when a hash-like object was expected to access value using 'service_certificate' from key 'tripleo::rabbitmq::service_certificate.service_certificate' (Exception) + service_crt= ++ hiera -c /etc/puppet/hiera.yaml tripleo::rabbitmq::service_certificate.service_key Traceback (most recent call last): 9: from /bin/hiera:246:in `<main>' 8: from /usr/share/ruby/vendor_ruby/hiera.rb:116:in `lookup' 7: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup' 6: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:267:in `each' 5: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `block in lookup' 4: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:272:in `catch' 3: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:279:in `block (2 levels) in lookup' 2: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `qualified_lookup' 1: from /usr/share/ruby/vendor_ruby/hiera/backend.rb:316:in `each' /usr/share/ruby/vendor_ruby/hiera/backend.rb:329:in `block in qualified_lookup': Hiera type mismatch: Got String when a hash-like object was expected to access value using 'service_key' from key 'tripleo::rabbitmq::service_certificate.service_key' (Exception) + service_key= + echo rabbitmq-bundle-podman-1 + grep -q '^rabbitmq-bundle' + tar -c '' '' + podman exec -i rabbitmq-bundle-podman-1 tar -C / -xv tar: Substituting `.' for empty member name tar: : Cannot stat: No such file or directory tar: Substituting `.' for empty member name tar: : Cannot stat: No such file or directory tar: Exiting with failure status due to previous errors + podman exec rabbitmq-bundle-podman-1 cp /var/lib/kolla/config_files/src-tls '' cp: -r not specified; omitting directory '/var/lib/kolla/config_files/src-tls' + podman exec -u root rabbitmq-bundle-podman-1 chown rabbitmq:rabbitmq '' chown: cannot access '': No such file or directory + podman exec -u root rabbitmq-bundle-podman-1 chown rabbitmq:rabbitmq '' chown: cannot access '': No such file or directory + podman exec rabbitmq-bundle-podman-1 rabbitmqctl eval 'ssl:clear_pem_cache().' ok Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Fixed upstream, instructions for testing: 1. check the current certificate for rabbitmq # md5sum /etc/pki/tls/certs/rabbitmq.crt 921441554988987b997a67978c2e4689 /etc/pki/tls/certs/rabbitmq.crt 2. generate a new certificate with certmonger # getcert resubmit -i rabbitmq Resubmitting "rabbitmq" to "IPA". 3. verify that a new certificate has been retrieved by certmonger on the host # md5sum /etc/pki/tls/certs/rabbitmq.crt 439789f66ce25332786891aace6f18da /etc/pki/tls/certs/rabbitmq.crt 4. verify that the cert got injected in the rabbitmq container as expected # podman exec $(podman ps -q --filter name=rabbitmq) md5sum /etc/pki/tls/certs/rabbitmq.crt 439789f66ce25332786891aace6f18da /etc/pki/tls/certs/rabbitmq.crt 5. double check that the cert injection uses "tar" instead of "podman cp" # /usr/bin/certmonger-rabbitmq-refresh.sh tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets etc/pki/tls/certs/rabbitmq.crt etc/pki/tls/private/rabbitmq.key ok Note: it's ok for certmonger-redis-refresh.sh to still use 'podman cp', as it injects the new certificate into a container managed by paunch (redis_tls_proxy) and not the redis container managed by pacemaker.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1001