Description of problem: foreman-proxy service failed with error "Exiting: No such file or directory @ rb_sysopen - /etc/foreman-proxy/ssl_key.pem" Version-Release number of selected component (if applicable): 7.0 Snap1.1 How reproducible: always Steps to Reproduce: 1. Install Satellite 7.0 Snap1.1 2. Check the satellite services status. 3. All the services are in running state except foreman-proxy and which is getting failed with error "Exiting: No such file or directory @ rb_sysopen - /etc/foreman-proxy/ssl_key.pem" Actual results: Foreman-proxy service failed to start. Expected results: All the satellite services should be up and running. Additional info:
Looking at the proxy log, it did start up fine at 2021-12-02T12:38:26 and 2021-12-02T13:13:13, but failed at 2021-12-02T23:26:41. So whatever happened in between, would be interesting to know.
Looking at more logs, there seem to be more certs related issues on that machine: tomcat: Dec 2 23:25:09 dhcp-2-109 server: SEVERE: Failed to load keystore type PKCS12 with path /etc/candlepin/certs/keystore due to /etc/candlepin/certs/keystore (No such file or directory) Dec 2 23:25:09 dhcp-2-109 server: java.io.FileNotFoundException: /etc/candlepin/certs/keystore (No such file or directory)
The problem is not the installer, but `katello-change-hostname`. When trying to change the hostname of an existing system, the tool bails out: [root@ci-vm-10-0-101-91 ~]# satellite-change-hostname newhost.example.com -u admin -p changeme Checking hostname validity Checking overall health of server Checking credentials ***WARNING*** This script will modify your system. You will need to re-register any satellite clients registered to this system after script completion. Capsules will have to be re-registered and reinstalled. If you are using custom certificates, you will have to run the satellite-installer again with custom certificate options after this script completes. Have you taken the necessary precautions (backups, snapshots, etc...)? Proceed with changing your hostname? [y/n] y Precheck passed updating hostname in /etc/hostname setting hostname checking if hostname was changed Updating default Capsule Updating installation media paths stopping services removing old cert rpms No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-apache* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-foreman-client* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-foreman-proxy* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-foreman-proxy-client* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-puppet-client* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-broker* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-client-cert* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-router-client* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-qpid-router-server* No Match for argument: ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-tomcat* deleting old certs backed up /var/www/html/pub to /var/www/html/pub/ci-vm-10-0-101-91.hosted.upshift.rdu2.redhat.com-20211207054627.backup updating hostname in /etc/hosts updating hostname in foreman installer scenarios updating hostname in hammer configuration backing up last_scenario.yaml removing last_scenario.yaml re-running the installer satellite-installer --scenario satellite -v --disable-system-checks --certs-regenerate=true --foreman-proxy-register-in-foreman true /usr/share/katello/hostname-change.rb:159:in `next_steps_message': undefined local variable or method `proxy' for #<KatelloUtilities::HostnameChange:0x00000001b9c408> (NameError) from /usr/share/katello/hostname-change.rb:570:in `installer_failure_message' from /usr/share/katello/hostname-change.rb:541:in `run' from /usr/sbin/satellite-change-hostname:23:in `<main>' And *THEN* the certs are indeed missing: [root@ci-vm-10-0-101-91 ~]# systemctl restart foreman-proxy Job for foreman-proxy.service failed because the control process exited with error code. See "systemctl status foreman-proxy.service" and "journalctl -xe" for details. [root@ci-vm-10-0-101-91 ~]# systemctl status foreman-proxy ● foreman-proxy.service - Foreman Proxy Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/foreman-proxy.service.d └─90-limits.conf Active: failed (Result: exit-code) since Tue 2021-12-07 05:48:12 EST; 5s ago Process: 4130 ExecStart=/usr/share/foreman-proxy/bin/smart-proxy --no-daemonize (code=exited, status=1/FAILURE) Main PID: 4130 (code=exited, status=1/FAILURE) Dec 07 05:48:10 newhost.example.com systemd[1]: Starting Foreman Proxy... Dec 07 05:48:12 newhost.example.com smart-proxy[4130]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/sqlite.rb:114: warning: rb_check_safe_obj will be removed in Ruby 3.0 Dec 07 05:48:12 newhost.example.com smart-proxy[4130]: Errors detected on startup, see log for details. Exiting: No such file or directory @ rb_sysopen - /etc/foreman-proxy/ssl_key.pem Dec 07 05:48:12 newhost.example.com systemd[1]: foreman-proxy.service: main process exited, code=exited, status=1/FAILURE Dec 07 05:48:12 newhost.example.com systemd[1]: Failed to start Foreman Proxy. Dec 07 05:48:12 newhost.example.com systemd[1]: Unit foreman-proxy.service entered failed state. Dec 07 05:48:12 newhost.example.com systemd[1]: foreman-proxy.service failed.
Created redmine issue https://projects.theforeman.org/issues/34098 from this bug
Verified on Satellite 7.0 snap 4
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5498