Is the installer currently following the instructions to enable the selinux port? 

Specifically, what is returned from "getsebool -a | grep candlepin" ?

If that is returning  candlepin_can_bind_activemq_port --> off

Toggle the bool as desired via "sudo setsebool candlepin_can_bind_activemq_port on" or "sudo setsebool candlepin_can_bind_activemq_port off"

Since this is working now according to Mike's findings (and my own) I'll close this out. We can re-open once we have a clear reproducer.

*** Bug 1901983 has been marked as a duplicate of this bug. ***

Reassigning this bug to the Installer component. On the reproducer I found that the Candlepin truststore doesn't get updated by the installer with the new client cert which is used by the Foreman rails app to connect to the Artemis message broker embedded in Candlepin. That's why candlepin_events shows up as FAIL via hammer ping.

Here's what I found:

The java-client cert and key in /etc/pki/katello are correctly updated, and are a valid pair =>

[root@dhcp-2-190 certs]# openssl x509 -noout -modulus -in java-client.crt  | openssl md5
(stdin)= d74483a4ae79b6b2a6ea09afe1b21095
[root@dhcp-2-190 certs]# openssl rsa -noout -modulus -in ../private/java-client.key | openssl md5
(stdin)= d74483a4ae79b6b2a6ea09afe1b21095

However, candlepin's truststore doesn't know about the new java-client.crt (called 'artemis-client' in the store) =>

[root@dhcp-2-190 certs]# keytool -list -keystore truststore
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

artemis-client, Dec 10, 2020, trustedCertEntry, 
Certificate fingerprint (SHA1): 17:91:F0:47:4C:18:8B:19:57:49:D3:4C:1E:05:38:D9:59:66:82:3B

Compare that fingerprint to /etc/pki/katello/certs/java-client.crt =>

[root@dhcp-2-190 certs]# openssl x509 -noout -fingerprint -sha1 -inform pem -in java-client.crt
SHA1 Fingerprint=2C:E3:3C:D1:B3:A5:01:EF:B7:5E:00:5D:6B:87:DF:6B:CA:28:A3:56

They should match, but don't

*** Bug 1906747 has been marked as a duplicate of this bug. ***

Created redmine issue https://projects.theforeman.org/issues/31574 from this bug

*** Bug 1914122 has been marked as a duplicate of this bug. ***

In my testing, this issue seems to come up only when running satellite-change-hostname, because /usr/share/katello/hostname-change.rb deletes the tomcat and candlepin certs and keystore file, but not the truststore, before running foreman-installer / satellite-installer. I think deleting the truststore here is the resolution, instead of adding extra logic to the puppet classes to check if the stores need updating.

This patch resolves the issue in my testing:

# diff -pruN /usr/share/katello/hostname-change.rb.bak /usr/share/katello/hostname-change.rb
--- /usr/share/katello/hostname-change.rb.bak   2021-01-25 10:25:33.325936156 -0500
+++ /usr/share/katello/hostname-change.rb   2021-01-25 10:26:18.088624628 -0500
@@ -476,9 +476,12 @@ If not done, all hosts will lose connect
         self.run_cmd("rm -rf /etc/candlepin/certs/amqp{,.bak}")
         self.run_cmd("rm -f /etc/candlepin/certs/candlepin-ca.crt /etc/candlepin/certs/candlepin-ca.key")
         self.run_cmd("rm -f /etc/candlepin/certs/keystore")
+        self.run_cmd("rm -f /etc/candlepin/certs/truststore")
         self.run_cmd("rm -f /etc/tomcat/keystore")
+        self.run_cmd("rm -f /etc/tomcat/truststore")
         self.run_cmd("rm -rf /etc/foreman/old-certs")
         self.run_cmd("rm -f /etc/pki/katello/keystore")
+        self.run_cmd("rm -f /etc/pki/katello/truststore")
         self.run_cmd("rm -rf #{@scenario_answers["foreman"]["client_ssl_cert"]}")
         self.run_cmd("rm -rf #{@scenario_answers["foreman"]["client_ssl_key"]}")
       end

After running satellite-change-hostname with this patch, katello's candlepin events listener successfully connects to candlepin:

# hammer ping
[...]
candlepin:        
    Status:          ok
    Server Response: Duration: 391ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
[...]

Automation is still seeing the candlepin issue on snap 12 when attempting to run satellite-change-hostname

Verified on 6.9 Snap 12.

Verification points:

1- Install 6.9 Snap 12 on the Satlab machine.
2- Checked the hammer ping and satellite services status, I didn't see any candlepin issue.
3- the changes that made in /usr/share/katello/hostname-change.rb file is reflected.
4- Checked the fix package.

rpm -qa|grep katello-3.18.1-2
katello-3.18.1-2.el7sat.noarch


Can we mark this bug as verified and start a separate discussion on pondrejk reported problem in the new bug?

There are two grouped but entirely separate issues that have been in this bug.

 1) a candlepin SELinux denial that can appear when doing a change hostname sometimes
 2) truststore not getting updated with new certificates when they change

There is a fix for the SELinux issue -- https://github.com/Katello/katello-selinux/pull/31  but I don't think it made it into downstream yet. Then there is the truststore issue fixes which has been the primary target of this BZ. I would be happy if we split them as they are different concerns. They are both caused by change hostname but they are different issues.

pondrejk @ehelms I have created a separate issue BZ#1925616 for "candlepin SELinux denial" that can appear when doing a satellite hostname change.

As per the @ehelms comment, this bug is more related to trustcore certificates update, So I am marking this bug as verified based on BZ#1897360#c20 comment.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313