Bug 2058532 - certs-regenerate breaks qpidd certificates, resulting in qpidd start-up failures: Couldn't find any network address to listen to
Summary: certs-regenerate breaks qpidd certificates, resulting in qpidd start-up failu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Installation
Version: 6.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.11.0
Assignee: Eric Helms
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-25 09:16 UTC by Lukas Pramuk
Modified: 2023-09-15 01:52 UTC (History)
4 users (show)

Fixed In Version: foreman-installer-3.1.2.3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2081814 (view as bug list)
Environment:
Last Closed: 2022-07-05 14:33:57 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 34860 0 Normal Closed Resetting nssdb certificate does not update private key and breaks Qpid 2022-05-03 18:17:05 UTC
Red Hat Product Errata RHSA-2022:5498 0 None None None 2022-07-05 14:34:13 UTC

Description Lukas Pramuk 2022-02-25 09:16:44 UTC 
Description of problem:
Upgrade fails during installer run with qpidd start-up failed: Couldn't find any network address to listen to

Version-Release number of selected component (if applicable):
7.0.0 Snap11

How reproducible:
deterministic

Steps to Reproduce:
1. Upgrade Sat 6.10.3 to 7.0.0

# satellite-maintain upgrade run --target-version 7.0 -w repositories-validate,repositories-setup -y
...

Running Migration scripts to Satellite 7.0
================================================================================
Setup repositories:                                                   [ALREADY RUN]
The step was skipped as it was already run and it is marked as run_once. Use --force to enforce the execution.
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [FAIL]
Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 6:
 2022-02-24 11:15:39 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-02-24 11:15:47 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-02-24 11:15:47 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-02-24 11:15:55 [WARN  ] [pre] Skipping system checks.
2022-02-24 11:15:55 [WARN  ] [pre] Skipping system checks.
2022-02-24 11:16:05 [NOTICE] [configure] Starting system configuration.
2022-02-24 11:16:22 [NOTICE] [configure] 250 configuration steps out of 2101 steps complete.
2022-02-24 11:16:59 [NOTICE] [configure] 500 configuration steps out of 2105 steps complete.
2022-02-24 11:17:07 [NOTICE] [configure] 750 configuration steps out of 2107 steps complete.
2022-02-24 11:17:14 [NOTICE] [configure] 1000 configuration steps out of 2112 steps complete.
2022-02-24 11:17:16 [NOTICE] [configure] 1250 configuration steps out of 2116 steps complete.
2022-02-24 11:18:46 [ERROR ] [configure] /Service[qpidd]: Failed to call refresh: Systemd restart for qpidd failed!
2022-02-24 11:18:46 [ERROR ] [configure] journalctl log for qpidd:
2022-02-24 11:18:46 [ERROR ] [configure] -- Logs begin at Thu 2022-02-17 04:35:44 EST, end at Thu 2022-02-24 11:18:46 EST. --
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: Stopping An AMQP message broker daemon....
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: Stopped An AMQP message broker daemon..
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: Starting An AMQP message broker daemon....
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Broker (pid=26158) start-up failed: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Broker (pid=26158) start-up failed: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Unexpected error: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Unexpected error: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: qpidd.service start-post operation timed out. Stopping.
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: Failed to start An AMQP message broker daemon..
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: Unit qpidd.service entered failed state.
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: qpidd.service failed.
2022-02-24 11:18:46 [ERROR ] [configure] /Service[qpidd]: Systemd restart for qpidd failed!
2022-02-24 11:18:46 [ERROR ] [configure] journalctl log for qpidd:
2022-02-24 11:18:46 [ERROR ] [configure] -- Logs begin at Thu 2022-02-17 04:35:44 EST, end at Thu 2022-02-24 11:18:46 EST. --
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: Stopping An AMQP message broker daemon....
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: Stopped An AMQP message broker daemon..
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: Starting An AMQP message broker daemon....
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Broker (pid=26158) start-up failed: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Broker (pid=26158) start-up failed: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Unexpected error: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com qpidd[26158]: 2022-02-24 11:17:16 [Broker] critical Unexpected error: Couldn't find any network address to listen to
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:17:16 sat.example.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: qpidd.service start-post operation timed out. Stopping.
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: Failed to start An AMQP message broker daemon..
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: Unit qpidd.service entered failed state.
2022-02-24 11:18:46 [ERROR ] [configure] Feb 24 11:18:46 sat.example.com systemd[1]: qpidd.service failed.
2022-02-24 11:29:23 [NOTICE] [configure] 1500 configuration steps out of 2117 steps complete.
2022-02-24 11:30:32 [NOTICE] [configure] 1750 configuration steps out of 2117 steps complete.
2022-02-24 11:33:09 [NOTICE] [configure] 2000 configuration steps out of 2117 steps complete.
2022-02-24 11:33:40 [NOTICE] [configure] System configuration has finished.


Actual results:
upgrade fails during installer run

Expected results:
upgrade successfully finishes
Comment 2 Evgeni Golov 2022-03-15 09:23:18 UTC 
As there is no reproducer and no sosreport for this, the issue is a tad hard to diagnose.
Additionally, Lukas reports that this same DB passes fine in automation :/

My original thought was IPv4-only vs IPv4/6 setups, as qpidd by default listens on both v4 and v6 localhost (via interface=lo in qpidd.conf).

But I couldn't find any hints for irregularities in the existing customer backup.

Thus I would like to request QE to try to reproduce this again and if possible leave the VM running so that we can inspect it.
Comment 7 Lukas Pramuk 2022-04-26 09:46:12 UTC 
It occurred also in upgrade automation:

1) Upgrade QE template to 6.11.0

2) Enable cockpit feature by running installer

# satellite-installer --enable-foreman-plugin-remote-execution-cockpit
Comment 8 Evgeni Golov 2022-04-26 10:43:18 UTC 
Hah, with a reproducer machine in place, this gets much better.

The issue is not so much that it can't find a network address, but that it seems it can't load the certs and then can't create an SSL listening socket.

I still don't know *why*, but well, progress nevertheless.
Comment 9 Evgeni Golov 2022-04-26 10:44:26 UTC 
With log-enable=debug in the config, we see:

2022-04-26 06:43:34 [Network] debug Using interface: 127.0.0.1
2022-04-26 06:43:34 [System] debug Exception constructed: Failed to retrieve private key from certificate (/builddir/build/BUILD/qpid-cpp-1.36.0/src/qpid/sys/ssl/SslSocket.cpp:234)
2022-04-26 06:43:34 [Network] debug Using interface: ::1
2022-04-26 06:43:34 [System] debug Exception constructed: Failed to retrieve private key from certificate (/builddir/build/BUILD/qpid-cpp-1.36.0/src/qpid/sys/ssl/SslSocket.cpp:234)
2022-04-26 06:43:34 [System] debug Exception constructed: Couldn't find any network address to listen to
2022-04-26 06:43:34 [Broker] critical Broker (pid=96725) start-up failed: Couldn't find any network address to listen to
2022-04-26 06:43:34 [Broker] critical Unexpected error: Couldn't find any network address to listen to
Comment 10 Lukas Pramuk 2022-04-26 13:03:57 UTC 
This is not Upgrades related as I have reproducer with fresh 6.11:

1) Have a Satellite 6.11.0

2) Enable katello-agent by installer run

# satellite-installer --foreman-proxy-content-enable-katello-agent

3) Run installer again

# satellite-installer 

>>> after enabling katello-agent all sucessive installer runs are failing
Comment 11 Evgeni Golov 2022-04-26 13:39:48 UTC 
it seems (to me), that it even doesn't try to load the password file:

on a working system:
# strace -f /usr/sbin/qpidd --config /etc/qpid/qpidd.conf 2>&1 |grep nss_db
[pid 51190] openat(AT_FDCWD, "/etc/pki/katello/nss_db_password-file", O_RDONLY) = 26

on a broken one:
# strace -f /usr/sbin/qpidd --config /etc/qpid/qpidd.conf 2>&1 |grep nss_db
<empty>
Comment 12 Evgeni Golov 2022-04-26 15:01:22 UTC 
Coming closer. Looking at a puppet run on the reproducer, the certs changed for some reason:

2022-04-26 08:04:25 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Cert[dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker]: Starting to evaluate the resource (365 of 1762)
2022-04-26 08:04:25 [DEBUG ] [configure] Executing: '/bin/katello-ssl-tool --gen-server --dir /root/ssl-build --set-hostname dhcp-3-82.vms.sat.rdu2.redhat.com --server-cert dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt --server
-cert-req dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt.req --server-key dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.key --server-rpm dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker -p file:/etc/pki/katello/private/katello-defa
ult-ca.pwd --set-hostname dhcp-3-82.vms.sat.rdu2.redhat.com --set-common-name dhcp-3-82.vms.sat.rdu2.redhat.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katel
lo-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org pulp --set-org-unit SomeOrgUnit --set-email  --cert-expiration 7300 --set-cname localhost'
2022-04-26 08:04:26 [INFO  ] [configure] /Stage[main]/Certs::Qpid/Cert[dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker]/ensure: created
2022-04-26 08:04:26 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Cert[dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker]: The container Class[Certs::Qpid] will propagate my refresh event
2022-04-26 08:04:26 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Cert[dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker]: Evaluated in 1.57 seconds
2022-04-26 08:04:26 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nss_db_password-file]: Starting to evaluate the resource (366 of 1762)
2022-04-26 08:04:26 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nss_db_password-file]: Evaluated in 0.00 seconds
2022-04-26 08:04:26 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/Nssdb[/etc/pki/katello/nssdb]: Starting to evaluate the resource (367 of 1762)
2022-04-26 08:04:26 [DEBUG ] [configure] Executing: '/bin/certutil -K -d /etc/pki/katello/nssdb -f /etc/pki/katello/nss_db_password-file'
2022-04-26 08:04:26 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/Nssdb[/etc/pki/katello/nssdb]: Evaluated in 0.05 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb]: Starting to evaluate the resource (368 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb]: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/cert8.db]: Starting to evaluate the resource (369 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/cert8.db]: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/key3.db]: Starting to evaluate the resource (370 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/key3.db]: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/secmod.db]: Starting to evaluate the resource (371 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/secmod.db]: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /etc/pki/katello/nssdb: Starting to evaluate the resource (372 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] /etc/pki/katello/nssdb: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] Class[Certs::Ssltools::Nssdb]: Starting to evaluate the resource (373 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] Class[Certs::Ssltools::Nssdb]: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/File[/etc/pki/katello/private/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.key]: Starting to evaluate the resource (374 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/File[/etc/pki/katello/private/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.key]: Nothing to manage: no ensure and the resource doesn't exist
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/File[/etc/pki/katello/private/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.key]: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/File[/etc/pki/katello/certs/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt]: Starting to evaluate the resource (375 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/File[/etc/pki/katello/certs/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt]: Nothing to manage: no ensure and the resource doesn't exist
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/File[/etc/pki/katello/certs/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt]: Evaluated in 0.00 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Nssdb_certificate[/etc/pki/katello/nssdb:ca]: Starting to evaluate the resource (376 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/certutil -L -a -d /etc/pki/katello/nssdb -n ca'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/certutil -L -a -d /etc/pki/katello/nssdb -n ca'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/openssl x509 -sha256 -noout -fingerprint -in /tmp/cert20220426-69708-5x8erg'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/openssl x509 -sha256 -noout -fingerprint -in /etc/pki/katello/certs/katello-default-ca.crt'
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Nssdb_certificate[/etc/pki/katello/nssdb:ca]: Evaluated in 0.14 seconds
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Nssdb_certificate[/etc/pki/katello/nssdb:broker]: Starting to evaluate the resource (377 of 1766)
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/certutil -L -a -d /etc/pki/katello/nssdb -n broker'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/certutil -L -a -d /etc/pki/katello/nssdb -n broker'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/openssl x509 -sha256 -noout -fingerprint -in /tmp/cert20220426-69708-1h3z7ov'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/openssl x509 -sha256 -noout -fingerprint -in /root/ssl-build/dhcp-3-82.vms.sat.rdu2.redhat.com/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/certutil -L -a -d /etc/pki/katello/nssdb -n broker'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/certutil -D -d /etc/pki/katello/nssdb -n broker'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/certutil -A -a -d /etc/pki/katello/nssdb -n broker -t ,, -i /root/ssl-build/dhcp-3-82.vms.sat.rdu2.redhat.com/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt -f /etc/pki/k
atello/nss_db_password-file'
2022-04-26 08:04:27 [DEBUG ] [configure] Executing: '/bin/openssl x509 -sha256 -noout -fingerprint -in /root/ssl-build/dhcp-3-82.vms.sat.rdu2.redhat.com/dhcp-3-82.vms.sat.rdu2.redhat.com-qpid-broker.crt'
2022-04-26 08:04:27 [INFO  ] [configure] /Stage[main]/Certs::Qpid/Nssdb_certificate[/etc/pki/katello/nssdb:broker]/certificate: certificate changed 'C0:B5:FF:BB:4E:63:37:E8:34:E7:25:00:AE:F5:32:9F:A9:F4:23:7F:36:F2:2D:11:19:68:3C:A
6:7A:F6:B0:70' to 'FB:68:D5:F5:D0:BB:22:6A:73:DD:39:45:51:B9:2C:68:E3:3F:36:D6:FE:D6:58:C9:06:41:7F:FB:FB:66:83:4A'
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Nssdb_certificate[/etc/pki/katello/nssdb:broker]: The container Class[Certs::Qpid] will propagate my refresh event
2022-04-26 08:04:27 [DEBUG ] [configure] /Stage[main]/Certs::Qpid/Nssdb_certificate[/etc/pki/katello/nssdb:broker]: Evaluated in 0.31 seconds


Now someone smarter than me needs to tell me *why* it decided to re-do the certs?!
Comment 14 Evgeni Golov 2022-04-27 09:17:06 UTC 
DING DING DING!

2022-04-26 08:04:13 [DEBUG ] [configure] Found key: "certs::regenerate" value: true

And it's true, /etc/foreman-installer/scenarios.d/satellite-answers.yaml has:

certs:
  regenerate: true

Which means it regenerates the certs on every run.

So I'd argue there are two bugs here:
- it should not regenerate the certs on every run
- if it is regenerating, it should make qpidd happy afterwards
Comment 15 Evgeni Golov 2022-04-27 09:52:36 UTC 
Right. And the real reproducer is:

- deploy 6.11
- enable agent support: satellite-installer --foreman-proxy-content-enable-katello-agent true
- regen certs: satellite-installer --certs-regenerate true

I am guessing for Lukas the origin of the problem is the fact that this is a QE-template machine, that got deployed as sat-template.example.com and then used katello-change-hostname to be renamed to satellite.example.com. This left the regen=true in the answers. The first run to enable agent succeeded (there were no old certs), and the second regenerated the certs again and boom.
Comment 16 Evgeni Golov 2022-04-27 10:02:34 UTC 
Workaround: rm -rf /etc/pki/katello/nss*
Comment 17 Evgeni Golov 2022-04-27 11:00:22 UTC 
Actually, just purging /etc/pki/katello/nssdb/key3.db is enough.
Comment 18 Evgeni Golov 2022-04-27 11:23:38 UTC 
Yeah, so (I am talking to myself, right?)

The problem is that the private key is not updated in the nssdb:

from a run that properly generated the db:
[root@sat-6-11-qa-rhel7 ~]# grep -E '(pkcs12|pk12).*nss' /var/log/foreman-installer/satellite.log 
2022-04-27 10:57:39 [DEBUG ] [configure] Executing: '/bin/openssl pkcs12 -export -in /root/ssl-build/sat-6-11-qa-rhel7.tanso.example.com/sat-6-11-qa-rhel7.tanso.example.com-qpid-broker.crt -inkey /root/ssl-build/sat-6-11-qa-rhel7.tanso.example.com/sat-6-11-qa-rhel7.tanso.example.com-qpid-broker.key -out /tmp/pkcs1220220427-30609-jpxpdb -password file:/etc/pki/katello/nss_db_password-file -name broker'
2022-04-27 10:57:39 [DEBUG ] [configure] Executing: '/bin/pk12util -i /tmp/pkcs1220220427-30609-jpxpdb -d /etc/pki/katello/nssdb -w /etc/pki/katello/nss_db_password-file -k /etc/pki/katello/nss_db_password-file'

from a failing one:
[root@sat-6-11-qa-rhel7 ~]# grep -E '(pkcs12|pk12).*nss' /var/log/foreman-installer/satellite.log 
<empty>

But at that point my understanding of Puppet providers ends and I'm handing over to @ehelms
Comment 19 Lukas Pramuk 2022-05-19 13:22:57 UTC 
VERIFIED.

@Satellite 6.11.0 Snap20
foreman-installer-3.1.2.5-1.el7sat.noarch

by the manual reproducer described in comment#15:

1) Install Satellite 6.11.0

2) Enable optional katello-agent support
# satellite-installer --foreman-proxy-content-enable-katello-agent true

3) Run the installer again to regenerate certs
# satellite-installer --certs-regenerate true
2022-05-19 05:57:19 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-05-19 05:57:25 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-05-19 05:57:25 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Package versions are locked. Continuing with unlock.
2022-05-19 05:57:43 [NOTICE] [configure] Starting system configuration.
2022-05-19 05:57:57 [NOTICE] [configure] 250 configuration steps out of 1762 steps complete.
2022-05-19 05:58:09 [NOTICE] [configure] 500 configuration steps out of 2570 steps complete.
2022-05-19 05:58:10 [NOTICE] [configure] 750 configuration steps out of 2570 steps complete.
2022-05-19 05:58:10 [NOTICE] [configure] 1000 configuration steps out of 2570 steps complete.
2022-05-19 05:58:11 [NOTICE] [configure] 1250 configuration steps out of 2570 steps complete.
2022-05-19 05:58:22 [NOTICE] [configure] 1500 configuration steps out of 2572 steps complete.
2022-05-19 05:58:29 [NOTICE] [configure] 1750 configuration steps out of 2575 steps complete.
2022-05-19 05:58:30 [NOTICE] [configure] 2000 configuration steps out of 2580 steps complete.
2022-05-19 05:58:32 [NOTICE] [configure] 2250 configuration steps out of 2581 steps complete.
2022-05-19 05:59:47 [NOTICE] [configure] 2500 configuration steps out of 2581 steps complete.
2022-05-19 05:59:59 [NOTICE] [configure] System configuration has finished.
  Success!
  * Satellite is running at https://sat.example.com
...

>>> there are no qpidd failures since not only cert but also private key is now being regenerated
Comment 22 errata-xmlrpc 2022-07-05 14:33:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498
Comment 23 Red Hat Bugzilla 2023-09-15 01:52:15 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days
Note You need to log in before you can comment on or make changes to this bug.