Bug 2052493

Summary:	restore on another machine fails with ERROR: web server's SSL certificate generation/signing failed
Product:	Red Hat Satellite	Reporter:	Lukas Pramuk <lpramuk>
Component:	Satellite Maintain	Assignee:	Eric Helms <ehelms>
Status:	CLOSED ERRATA	QA Contact:	Lukas Pramuk <lpramuk>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.11.0	CC:	ahumbe, apatel, aupadhye, egolov, gtalreja, kgaikwad, lpramuk, pcreech
Target Milestone:	6.11.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rubygem-foreman_maintain-1.0.10	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-07-05 14:32:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lukas Pramuk 2022-02-09 11:53:10 UTC

Description of problem:
restore on another machine fails with ERROR: web server's SSL certificate generation/signing failed 

while restore on the same (source) machine worked
restore on another machine failed with the same error even though I had changed the hostname by satellite-change-hostname prior restore

Version-Release number of selected component (if applicable):
7.0.0 Snap8


How reproducible:
deterministic

Steps to Reproduce:
SOURCE
1. Create an offline backup 
# satellite-maintain backup offline /var/backup

TARGET
2. Change satellite hostname
# echo "<target ip> <source fqdn>" >> /etc/hosts
# satellite-change-hostname <source fqdn> -y -u admin -p changeme

3. Restore from offline backup
# satellite-maintain restore -y /mnt/satellite-backup-*/
Running Restore backup
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Validate backup has appropriate files:                                [OK]
--------------------------------------------------------------------------------
Validate hostname is the same as backup:                              [OK]
--------------------------------------------------------------------------------
Validate network interfaces match the backup:                         [OK]
--------------------------------------------------------------------------------
Confirm dropping databases and running restore: 

WARNING: This script will drop and restore your database.
Your existing installation will be replaced with the backup database.
Once this operation is complete there is no going back.
Do you want to proceed? (assuming yes)
                                                                      [OK]      
--------------------------------------------------------------------------------
Setting file security: 
/ Restoring SELinux context                                           [OK]      
--------------------------------------------------------------------------------
Restore configs from backup: 
- Restoring configs                                                   [OK]      
--------------------------------------------------------------------------------
Run installer reset: 
| Installer reset                                                     [FAIL]    
Failed executing yes | satellite-installer -v --reset-data --disable-system-checks , exit status 6:
 2022-02-08 11:58:07 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-02-08 11:58:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-02-08 11:58:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Are you sure you want to continue? This will drop the databases, reset all configurations that you have made and bring all application data back to a fresh install. [y/n]
Package versions are locked. Continuing with unlock.
2022-02-08 11:59:22 [NOTICE] [pre] Dropping foreman database!
2022-02-08 11:59:22 [NOTICE] [pre] Dropping candlepin database!
2022-02-08 11:59:22 [NOTICE] [pre] Dropping pulpcore database!
2022-02-08 11:59:22 [WARN  ] [pre] Pulpcore content directory not present at '/var/lib/pulp/docroot'
2022-02-08 11:59:22 [WARN  ] [pre] Skipping system checks.
2022-02-08 11:59:22 [WARN  ] [pre] Skipping system checks.
2022-02-08 11:59:28 [NOTICE] [configure] Starting system configuration.
2022-02-08 11:59:42 [NOTICE] [configure] 250 configuration steps out of 1649 steps complete.
2022-02-08 11:59:44 [ERROR ] [configure] Execution of '/bin/katello-ssl-tool --gen-client --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-foreman-client.crt --server-cert-req sat.example.com-foreman-client.crt.req --server-key sat.example.com-foreman-client.key --server-rpm sat.example.com-foreman-client -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org FOREMAN --set-org-unit PUPPET --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:44 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:44 [ERROR ] [configure] 140476065654592:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:44 [ERROR ] [configure] 140476065654592:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-foreman-client.key
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.key --> sat.example.com-foreman-client.key.1
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-foreman-client.crt.req
2022-02-08 11:59:44 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:44 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:44 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:44 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:44 [ERROR ] [configure] --set-org          = "FOREMAN"
2022-02-08 11:59:44 [ERROR ] [configure] --set-org-unit     = "PUPPET"
2022-02-08 11:59:44 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:44 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.crt.req --> sat.example.com-foreman-client.crt.req.1
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-foreman-client.crt
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.crt --> sat.example.com-foreman-client.crt.1
...

--------------------------------------------------------------------------------
Scenario [Restore backup] failed.

The following steps ended up in failing state:

  [restore-installer-reset]
...


Actual results:
installer reset failed

Expected results:
successful restore

Comment 1 Lukas Pramuk 2022-02-09 11:55:34 UTC

Cert regen failed for all:

Run installer reset: 
| Installer reset                                                     [FAIL]    
Failed executing yes | satellite-installer -v --reset-data --disable-system-checks , exit status 6:
 2022-02-08 11:58:07 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-02-08 11:58:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-02-08 11:58:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Are you sure you want to continue? This will drop the databases, reset all configurations that you have made and bring all application data back to a fresh install. [y/n]
Package versions are locked. Continuing with unlock.
2022-02-08 11:59:22 [NOTICE] [pre] Dropping foreman database!
2022-02-08 11:59:22 [NOTICE] [pre] Dropping candlepin database!
2022-02-08 11:59:22 [NOTICE] [pre] Dropping pulpcore database!
2022-02-08 11:59:22 [WARN  ] [pre] Pulpcore content directory not present at '/var/lib/pulp/docroot'
2022-02-08 11:59:22 [WARN  ] [pre] Skipping system checks.
2022-02-08 11:59:22 [WARN  ] [pre] Skipping system checks.
2022-02-08 11:59:28 [NOTICE] [configure] Starting system configuration.
2022-02-08 11:59:42 [NOTICE] [configure] 250 configuration steps out of 1649 steps complete.
2022-02-08 11:59:44 [ERROR ] [configure] Execution of '/bin/katello-ssl-tool --gen-client --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-foreman-client.crt --server-cert-req sat.example.com-foreman-client.crt.req --server-key sat.example.com-foreman-client.key --server-rpm sat.example.com-foreman-client -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org FOREMAN --set-org-unit PUPPET --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:44 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:44 [ERROR ] [configure] 140476065654592:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:44 [ERROR ] [configure] 140476065654592:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-foreman-client.key
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.key --> sat.example.com-foreman-client.key.1
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-foreman-client.crt.req
2022-02-08 11:59:44 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:44 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:44 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:44 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:44 [ERROR ] [configure] --set-org          = "FOREMAN"
2022-02-08 11:59:44 [ERROR ] [configure] --set-org-unit     = "PUPPET"
2022-02-08 11:59:44 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:44 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.crt.req --> sat.example.com-foreman-client.crt.req.1
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-foreman-client.crt
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.crt --> sat.example.com-foreman-client.crt.1
2022-02-08 11:59:44 [ERROR ] [configure] /Stage[main]/Certs::Foreman/Cert[sat.example.com-foreman-client]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/katello-ssl-tool --gen-client --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-foreman-client.crt --server-cert-req sat.example.com-foreman-client.crt.req --server-key sat.example.com-foreman-client.key --server-rpm sat.example.com-foreman-client -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org FOREMAN --set-org-unit PUPPET --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:44 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:44 [ERROR ] [configure] 140476065654592:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:44 [ERROR ] [configure] 140476065654592:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-foreman-client.key
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.key --> sat.example.com-foreman-client.key.1
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-foreman-client.crt.req
2022-02-08 11:59:44 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:44 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:44 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:44 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:44 [ERROR ] [configure] --set-org          = "FOREMAN"
2022-02-08 11:59:44 [ERROR ] [configure] --set-org-unit     = "PUPPET"
2022-02-08 11:59:44 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:44 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.crt.req --> sat.example.com-foreman-client.crt.req.1
2022-02-08 11:59:44 [ERROR ] [configure] 
2022-02-08 11:59:44 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-foreman-client.crt
2022-02-08 11:59:44 [ERROR ] [configure] Rotated: sat.example.com-foreman-client.crt --> sat.example.com-foreman-client.crt.1
2022-02-08 11:59:45 [ERROR ] [configure] Execution of '/bin/katello-ssl-tool --gen-server --dir /root/ssl-build --set-hostname localhost --server-cert localhost-tomcat.crt --server-cert-req localhost-tomcat.crt.req --server-key localhost-tomcat.key --server-rpm localhost-tomcat -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname localhost --set-common-name localhost --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org Katello --set-org-unit SomeOrgUnit --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:45 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:45 [ERROR ] [configure] 140260282152768:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:45 [ERROR ] [configure] 140260282152768:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/localhost/localhost-tomcat.key
2022-02-08 11:59:45 [ERROR ] [configure] Rotated: localhost-tomcat.key --> localhost-tomcat.key.1
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/localhost/localhost-tomcat.crt.req
2022-02-08 11:59:45 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:45 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:45 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:45 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:45 [ERROR ] [configure] --set-org          = "Katello"
2022-02-08 11:59:45 [ERROR ] [configure] --set-org-unit     = "SomeOrgUnit"
2022-02-08 11:59:45 [ERROR ] [configure] --set-hostname     = "localhost"
2022-02-08 11:59:45 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:45 [ERROR ] [configure] Rotated: localhost-tomcat.crt.req --> localhost-tomcat.crt.req.1
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Generating/signing web server's SSL certificate: localhost-tomcat.crt
2022-02-08 11:59:45 [ERROR ] [configure] Rotated: localhost-tomcat.crt --> localhost-tomcat.crt.1
2022-02-08 11:59:45 [ERROR ] [configure] /Stage[main]/Certs::Candlepin/Cert[localhost-tomcat]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/katello-ssl-tool --gen-server --dir /root/ssl-build --set-hostname localhost --server-cert localhost-tomcat.crt --server-cert-req localhost-tomcat.crt.req --server-key localhost-tomcat.key --server-rpm localhost-tomcat -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname localhost --set-common-name localhost --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org Katello --set-org-unit SomeOrgUnit --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:45 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:45 [ERROR ] [configure] 140260282152768:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:45 [ERROR ] [configure] 140260282152768:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/localhost/localhost-tomcat.key
2022-02-08 11:59:45 [ERROR ] [configure] Rotated: localhost-tomcat.key --> localhost-tomcat.key.1
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/localhost/localhost-tomcat.crt.req
2022-02-08 11:59:45 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:45 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:45 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:45 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:45 [ERROR ] [configure] --set-org          = "Katello"
2022-02-08 11:59:45 [ERROR ] [configure] --set-org-unit     = "SomeOrgUnit"
2022-02-08 11:59:45 [ERROR ] [configure] --set-hostname     = "localhost"
2022-02-08 11:59:45 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:45 [ERROR ] [configure] Rotated: localhost-tomcat.crt.req --> localhost-tomcat.crt.req.1
2022-02-08 11:59:45 [ERROR ] [configure] 
2022-02-08 11:59:45 [ERROR ] [configure] Generating/signing web server's SSL certificate: localhost-tomcat.crt
2022-02-08 11:59:45 [ERROR ] [configure] Rotated: localhost-tomcat.crt --> localhost-tomcat.crt.1
2022-02-08 11:59:47 [ERROR ] [configure] Execution of '/bin/katello-ssl-tool --gen-server --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-apache.crt --server-cert-req sat.example.com-apache.crt.req --server-key sat.example.com-apache.key --server-rpm sat.example.com-apache -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org Katello --set-org-unit SomeOrgUnit --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:47 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:47 [ERROR ] [configure] 140047844509504:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:47 [ERROR ] [configure] 140047844509504:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-apache.key
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: sat.example.com-apache.key --> sat.example.com-apache.key.1
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-apache.crt.req
2022-02-08 11:59:47 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:47 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:47 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:47 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org          = "Katello"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org-unit     = "SomeOrgUnit"
2022-02-08 11:59:47 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:47 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: sat.example.com-apache.crt.req --> sat.example.com-apache.crt.req.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-apache.crt
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: sat.example.com-apache.crt --> sat.example.com-apache.crt.1
2022-02-08 11:59:47 [ERROR ] [configure] /Stage[main]/Certs::Apache/Cert[sat.example.com-apache]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/katello-ssl-tool --gen-server --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-apache.crt --server-cert-req sat.example.com-apache.crt.req --server-key sat.example.com-apache.key --server-rpm sat.example.com-apache -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org Katello --set-org-unit SomeOrgUnit --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:47 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:47 [ERROR ] [configure] 140047844509504:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:47 [ERROR ] [configure] 140047844509504:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-apache.key
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: sat.example.com-apache.key --> sat.example.com-apache.key.1
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-apache.crt.req
2022-02-08 11:59:47 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:47 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:47 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:47 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org          = "Katello"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org-unit     = "SomeOrgUnit"
2022-02-08 11:59:47 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:47 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: sat.example.com-apache.crt.req --> sat.example.com-apache.crt.req.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-apache.crt
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: sat.example.com-apache.crt --> sat.example.com-apache.crt.1
2022-02-08 11:59:47 [ERROR ] [configure] Execution of '/bin/katello-ssl-tool --gen-client --dir /root/ssl-build --set-hostname sat.example.com --server-cert pulp-client.crt --server-cert-req pulp-client.crt.req --server-key pulp-client.key --server-rpm pulp-client -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name admin --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org PULP --set-org-unit NODES --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:47 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:47 [ERROR ] [configure] 139913021384512:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:47 [ERROR ] [configure] 139913021384512:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/pulp-client.key
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: pulp-client.key --> pulp-client.key.1
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/pulp-client.crt.req
2022-02-08 11:59:47 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:47 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:47 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:47 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org          = "PULP"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org-unit     = "NODES"
2022-02-08 11:59:47 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:47 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: pulp-client.crt.req --> pulp-client.crt.req.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating/signing web server's SSL certificate: pulp-client.crt
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: pulp-client.crt --> pulp-client.crt.1
2022-02-08 11:59:47 [ERROR ] [configure] /Stage[main]/Certs::Pulp_client/Cert[pulp-client]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/katello-ssl-tool --gen-client --dir /root/ssl-build --set-hostname sat.example.com --server-cert pulp-client.crt --server-cert-req pulp-client.crt.req --server-key pulp-client.key --server-rpm pulp-client -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name admin --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org PULP --set-org-unit NODES --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:47 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:47 [ERROR ] [configure] 139913021384512:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:47 [ERROR ] [configure] 139913021384512:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/pulp-client.key
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: pulp-client.key --> pulp-client.key.1
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/pulp-client.crt.req
2022-02-08 11:59:47 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:47 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:47 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:47 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org          = "PULP"
2022-02-08 11:59:47 [ERROR ] [configure] --set-org-unit     = "NODES"
2022-02-08 11:59:47 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:47 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: pulp-client.crt.req --> pulp-client.crt.req.1
2022-02-08 11:59:47 [ERROR ] [configure] 
2022-02-08 11:59:47 [ERROR ] [configure] Generating/signing web server's SSL certificate: pulp-client.crt
2022-02-08 11:59:47 [ERROR ] [configure] Rotated: pulp-client.crt --> pulp-client.crt.1
2022-02-08 11:59:51 [NOTICE] [configure] 500 configuration steps out of 2453 steps complete.
2022-02-08 11:59:51 [NOTICE] [configure] 750 configuration steps out of 2453 steps complete.
2022-02-08 11:59:52 [NOTICE] [configure] 1000 configuration steps out of 2453 steps complete.
2022-02-08 11:59:53 [NOTICE] [configure] 1250 configuration steps out of 2453 steps complete.
2022-02-08 11:59:54 [ERROR ] [configure] Execution of '/bin/katello-ssl-tool --gen-server --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-foreman-proxy.crt --server-cert-req sat.example.com-foreman-proxy.crt.req --server-key sat.example.com-foreman-proxy.key --server-rpm sat.example.com-foreman-proxy -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org FOREMAN --set-org-unit SMART_PROXY --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:54 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:54 [ERROR ] [configure] 140331280918336:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:54 [ERROR ] [configure] 140331280918336:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy.key
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy.key --> sat.example.com-foreman-proxy.key.1
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy.crt.req
2022-02-08 11:59:54 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:54 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:54 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:54 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:54 [ERROR ] [configure] --set-org          = "FOREMAN"
2022-02-08 11:59:54 [ERROR ] [configure] --set-org-unit     = "SMART_PROXY"
2022-02-08 11:59:54 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:54 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy.crt.req --> sat.example.com-foreman-proxy.crt.req.1
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-foreman-proxy.crt
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy.crt --> sat.example.com-foreman-proxy.crt.1
2022-02-08 11:59:54 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Cert[sat.example.com-foreman-proxy]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/katello-ssl-tool --gen-server --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-foreman-proxy.crt --server-cert-req sat.example.com-foreman-proxy.crt.req --server-key sat.example.com-foreman-proxy.key --server-rpm sat.example.com-foreman-proxy -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org FOREMAN --set-org-unit SMART_PROXY --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:54 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:54 [ERROR ] [configure] 140331280918336:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:54 [ERROR ] [configure] 140331280918336:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy.key
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy.key --> sat.example.com-foreman-proxy.key.1
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy.crt.req
2022-02-08 11:59:54 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:54 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:54 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:54 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:54 [ERROR ] [configure] --set-org          = "FOREMAN"
2022-02-08 11:59:54 [ERROR ] [configure] --set-org-unit     = "SMART_PROXY"
2022-02-08 11:59:54 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:54 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy.crt.req --> sat.example.com-foreman-proxy.crt.req.1
2022-02-08 11:59:54 [ERROR ] [configure] 
2022-02-08 11:59:54 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-foreman-proxy.crt
2022-02-08 11:59:54 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy.crt --> sat.example.com-foreman-proxy.crt.1
2022-02-08 11:59:56 [ERROR ] [configure] Execution of '/bin/katello-ssl-tool --gen-client --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-foreman-proxy-client.crt --server-cert-req sat.example.com-foreman-proxy-client.crt.req --server-key sat.example.com-foreman-proxy-client.key --server-rpm sat.example.com-foreman-proxy-client -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org FOREMAN --set-org-unit FOREMAN_PROXY --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:56 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:56 [ERROR ] [configure] 140714426926912:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:56 [ERROR ] [configure] 140714426926912:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy-client.key
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy-client.key --> sat.example.com-foreman-proxy-client.key.1
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy-client.crt.req
2022-02-08 11:59:56 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:56 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:56 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:56 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:56 [ERROR ] [configure] --set-org          = "FOREMAN"
2022-02-08 11:59:56 [ERROR ] [configure] --set-org-unit     = "FOREMAN_PROXY"
2022-02-08 11:59:56 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:56 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy-client.crt.req --> sat.example.com-foreman-proxy-client.crt.req.1
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-foreman-proxy-client.crt
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy-client.crt --> sat.example.com-foreman-proxy-client.crt.1
2022-02-08 11:59:56 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Cert[sat.example.com-foreman-proxy-client]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/katello-ssl-tool --gen-client --dir /root/ssl-build --set-hostname sat.example.com --server-cert sat.example.com-foreman-proxy-client.crt --server-cert-req sat.example.com-foreman-proxy-client.crt.req --server-key sat.example.com-foreman-proxy-client.key --server-rpm sat.example.com-foreman-proxy-client -p file:/etc/pki/katello/private/katello-default-ca.pwd --set-hostname sat.example.com --set-common-name sat.example.com --ca-cert /etc/pki/katello-certs-tools/certs/katello-default-ca.crt --ca-key /etc/pki/katello-certs-tools/private/katello-default-ca.key --set-country US --set-state North Carolina --set-city Raleigh --set-org FOREMAN --set-org-unit FOREMAN_PROXY --set-email  --cert-expiration 7300' returned 22: ERROR: web server's SSL certificate generation/signing failed:
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Using configuration from /root/ssl-build/katello-ca-openssl.cnf
2022-02-08 11:59:56 [ERROR ] [configure] unable to load CA private key
2022-02-08 11:59:56 [ERROR ] [configure] 140714426926912:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-02-08 11:59:56 [ERROR ] [configure] 140714426926912:error:0906A065:PEM routines:PEM_do_header:bad decrypt:crypto/pem/pem_lib.c:461:
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Generating the web server's SSL private key: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy-client.key
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy-client.key --> sat.example.com-foreman-proxy-client.key.1
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: katello-server-openssl.cnf --> katello-server-openssl.cnf.1
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Generating web server's SSL certificate request: /root/ssl-build/sat.example.com/sat.example.com-foreman-proxy-client.crt.req
2022-02-08 11:59:56 [ERROR ] [configure] Using distinguished names:
2022-02-08 11:59:56 [ERROR ] [configure] --set-country      = "US"
2022-02-08 11:59:56 [ERROR ] [configure] --set-state        = "North Carolina"
2022-02-08 11:59:56 [ERROR ] [configure] --set-city         = "Raleigh"
2022-02-08 11:59:56 [ERROR ] [configure] --set-org          = "FOREMAN"
2022-02-08 11:59:56 [ERROR ] [configure] --set-org-unit     = "FOREMAN_PROXY"
2022-02-08 11:59:56 [ERROR ] [configure] --set-hostname     = "sat.example.com"
2022-02-08 11:59:56 [ERROR ] [configure] --set-email        = ""
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy-client.crt.req --> sat.example.com-foreman-proxy-client.crt.req.1
2022-02-08 11:59:56 [ERROR ] [configure] 
2022-02-08 11:59:56 [ERROR ] [configure] Generating/signing web server's SSL certificate: sat.example.com-foreman-proxy-client.crt
2022-02-08 11:59:56 [ERROR ] [configure] Rotated: sat.example.com-foreman-proxy-client.crt --> sat.example.com-foreman-proxy-client.crt.1
2022-02-08 11:59:57 [NOTICE] [configure] 1500 configuration steps out of 2455 steps complete.
2022-02-08 11:59:58 [ERROR ] [configure] Failed to add certificate to keystore: Execution of '/bin/openssl pkcs12 -export -in /root/ssl-build/localhost/localhost-tomcat.crt -inkey /root/ssl-build/localhost/localhost-tomcat.key -out #<File:0x000000000730c580> -name tomcat -CAfile /etc/candlepin/certs/candlepin-ca.crt -password file:/etc/pki/katello/keystore_password-file' returned 1: No certificate matches private key
2022-02-08 11:59:58 [ERROR ] [configure] Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias candlepin-ca -file /etc/candlepin/certs/candlepin-ca.crt -storepass:file /etc/pki/katello/truststore_password-file' returned 1: keytool error: java.io.IOException: keystore password was incorrect
2022-02-08 11:59:58 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2079)
2022-02-08 11:59:58 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:836)
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:377)
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:370)
2022-02-08 11:59:58 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2022-02-08 11:59:58 [ERROR ] [configure] ... 5 more
2022-02-08 11:59:58 [ERROR ] [configure] /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:candlepin-ca]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias candlepin-ca -file /etc/candlepin/certs/candlepin-ca.crt -storepass:file /etc/pki/katello/truststore_password-file' returned 1: keytool error: java.io.IOException: keystore password was incorrect
2022-02-08 11:59:58 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2079)
2022-02-08 11:59:58 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:836)
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:377)
2022-02-08 11:59:58 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:370)
2022-02-08 11:59:58 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2022-02-08 11:59:58 [ERROR ] [configure] ... 5 more
2022-02-08 12:00:03 [NOTICE] [configure] 1750 configuration steps out of 2460 steps complete.
2022-02-08 12:00:05 [NOTICE] [configure] 2000 configuration steps out of 2466 steps complete.
2022-02-08 12:00:49 [NOTICE] [configure] 2250 configuration steps out of 2466 steps complete.
2022-02-08 12:01:48 [NOTICE] [configure] System configuration has finished.

Comment 7 Eric Helms 2022-05-06 14:13:25 UTC

Created redmine issue https://projects.theforeman.org/issues/34874 from this bug

Comment 8 Bryan Kearney 2022-05-06 16:04:42 UTC

Upstream bug assigned to ehelms

Comment 9 Bryan Kearney 2022-05-06 16:04:44 UTC

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34874 has been resolved.

Comment 10 Amit Upadhye 2022-05-11 12:06:34 UTC

We need to get 1.0.10 with cherry picks: https://github.com/theforeman/foreman-packaging/pull/7887

Comment 11 Lukas Pramuk 2022-05-20 11:17:34 UTC

VERIFIED.

@Satellite 6.11.0 Snap21
rubygem-foreman_maintain-1.0.10-1.el7sat.noarch

by the reproducer described in comment#0:

SOURCE
1) Create an offline backup 
# satellite-maintain backup offline /var/backup

TARGET
2) Change satellite hostname to the one in the backup
# echo "<target ip> <source fqdn>" >> /etc/hosts
# satellite-change-hostname <source fqdn> -y -u admin -p changeme

3. Restore from offline backup
# scp -r <source fqdn>:/var/backup /mnt
# satellite-maintain restore -y /mnt/satellite-backup-*/
Running Restore backup
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Validate backup has appropriate files:                                [OK]
--------------------------------------------------------------------------------
Validate hostname is the same as backup:                              [OK]
--------------------------------------------------------------------------------
Validate network interfaces match the backup:                         [OK]
--------------------------------------------------------------------------------
Confirm dropping databases and running restore: 

WARNING: This script will drop and restore your database.
Your existing installation will be replaced with the backup database.
Once this operation is complete there is no going back.
Do you want to proceed? (assuming yes)
                                                                      [OK]      
--------------------------------------------------------------------------------
Setting file security: 
/ Restoring SELinux context                                           [OK]      
--------------------------------------------------------------------------------
Restore configs from backup: 
\ Restoring configs                                                   [OK]      
--------------------------------------------------------------------------------
Run installer reset: 
| Installer reset                                                     [OK]      
--------------------------------------------------------------------------------
Stop applicable services: 

Stopping the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-api.socket, pulpcore-content.socket, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
\ All services stopped                                                [OK]      
--------------------------------------------------------------------------------
Extract any existing tar files in backup: 
\ Extracting pgsql data                                               [OK]      
--------------------------------------------------------------------------------
Migrate pulpcore db: 
/ Migrating pulpcore database                                         [OK]      
--------------------------------------------------------------------------------
Ensure Candlepin runs all migrations after restoring the database:    [OK]
--------------------------------------------------------------------------------
Start applicable services: 

Starting the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
- All services started                                                [OK]      
--------------------------------------------------------------------------------
Run daemon reload:                                                    [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [OK]
--------------------------------------------------------------------------------
Execute upgrade:run rake task:                                        [OK]
--------------------------------------------------------------------------------

>>> restore succeeded successfully

# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 43ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 36ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 1ms
pulp3:            
    Status:          ok
    Server Response: Duration: 72ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 59ms
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 4ms

>>> there are no issues after restore

Comment 14 errata-xmlrpc 2022-07-05 14:32:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498