Bug 2028733

Looking at the proxy log, it did start up fine at 2021-12-02T12:38:26 and 2021-12-02T13:13:13, but failed at 2021-12-02T23:26:41.

So whatever happened in between, would be interesting to know.

Looking at more logs, there seem to be more certs related issues on that machine:

tomcat:
Dec  2 23:25:09 dhcp-2-109 server: SEVERE: Failed to load keystore type PKCS12 with path /etc/candlepin/certs/keystore due to /etc/candlepin/certs/keystore (No such file or directory)
Dec  2 23:25:09 dhcp-2-109 server: java.io.FileNotFoundException: /etc/candlepin/certs/keystore (No such file or directory)

The problem is not the installer, but `katello-change-hostname`.

When trying to change the hostname of an existing system, the tool bails out:

[root@ci-vm-10-0-101-91 ~]# satellite-change-hostname newhost.example.com -u admin -p changeme

Checking hostname validity

Checking overall health of server

Checking credentials

***WARNING*** This script will modify your system.
You will need to re-register any satellite clients registered to this system after script completion.
Capsules will have to be re-registered and reinstalled. If you are using custom certificates,
you will have to run the satellite-installer again with custom certificate options after this script completes.
 Have you taken the necessary precautions (backups, snapshots, etc...)?
Proceed with changing your hostname? [y/n]
y
Precheck passed
updating hostname in /etc/hostname
setting hostname
checking if hostname was changed

Updating default Capsule
Updating installation media paths
stopping services
removing old cert rpms
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-apache*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-foreman-client*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-foreman-proxy*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-foreman-proxy-client*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-puppet-client*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-broker*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-client-cert*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-router-client*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-router-server*
No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-tomcat*
deleting old certs
backed up /var/www/html/pub to /var/www/html/pub/ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-20211207054627.backup
updating hostname in /etc/hosts
updating hostname in foreman installer scenarios
updating hostname in hammer configuration
backing up last_scenario.yaml
removing last_scenario.yaml
re-running the installer
satellite-installer --scenario satellite -v --disable-system-checks --certs-regenerate=true --foreman-proxy-register-in-foreman true
/usr/share/katello/hostname-change.rb:159:in `next_steps_message': undefined local variable or method `proxy' for #<KatelloUtilities::HostnameChange:0x00000001b9c408> (NameError)
	from /usr/share/katello/hostname-change.rb:570:in `installer_failure_message'
	from /usr/share/katello/hostname-change.rb:541:in `run'
	from /usr/sbin/satellite-change-hostname:23:in `<main>'

And *THEN* the certs are indeed missing:

[root@ci-vm-10-0-101-91 ~]# systemctl restart foreman-proxy
Job for foreman-proxy.service failed because the control process exited with error code. See "systemctl status foreman-proxy.service" and "journalctl -xe" for details.
[root@ci-vm-10-0-101-91 ~]# systemctl status foreman-proxy
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/foreman-proxy.service.d
           └─90-limits.conf
   Active: failed (Result: exit-code) since Tue 2021-12-07 05:48:12 EST; 5s ago
  Process: 4130 ExecStart=/usr/share/foreman-proxy/bin/smart-proxy --no-daemonize (code=exited, status=1/FAILURE)
 Main PID: 4130 (code=exited, status=1/FAILURE)

Dec 07 05:48:10 newhost.example.com systemd[1]: Starting Foreman Proxy...
Dec 07 05:48:12 newhost.example.com smart-proxy[4130]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/sqlite.rb:114: warning: rb_check_safe_obj will be removed in Ruby 3.0
Dec 07 05:48:12 newhost.example.com smart-proxy[4130]: Errors detected on startup, see log for details. Exiting: No such file or directory @ rb_sysopen - /etc/foreman-proxy/ssl_key.pem
Dec 07 05:48:12 newhost.example.com systemd[1]: foreman-proxy.service: main process exited, code=exited, status=1/FAILURE
Dec 07 05:48:12 newhost.example.com systemd[1]: Failed to start Foreman Proxy.
Dec 07 05:48:12 newhost.example.com systemd[1]: Unit foreman-proxy.service entered failed state.
Dec 07 05:48:12 newhost.example.com systemd[1]: foreman-proxy.service failed.

Created redmine issue https://projects.theforeman.org/issues/34098 from this bug

Verified on Satellite 7.0 snap 4

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498