Bug 2049893

Summary:	Capsule upgrade from 6.10 to 6.11 fails due to changed certs deployment
Product:	Red Hat Satellite	Reporter:	Vladimír Sedmík <vsedmik>
Component:	Documentation	Assignee:	Marie Hornickova <mdolezel>
Documentation sub component:	default	QA Contact:
Status:	CLOSED CURRENTRELEASE	Docs Contact:
Severity:	high
Priority:	high	CC:	ahumbe, bangelic, bbuckingham, egolov, ehelms, mdolezel, smallamp, vferschm
Version:	6.11.0	Keywords:	Triaged, Upgrades
Target Milestone:	6.11.0	Flags:	mdolezel: needinfo? (vferschm)
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2052867 (view as bug list)		Environment:
Last Closed:	2022-07-22 19:35:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2052867

Description Vladimír Sedmík 2022-02-02 21:11:55 UTC

Description of problem:
Capsule upgrade from 6.10.2 to 7.0 fails in Procedures::Installer::Upgrade


Version-Release number of selected component (if applicable):
original Sat/Cap - 6.10.2 snap 2
DF repos for upg - 7.0.0 snap 7


How reproducible:
always


Steps to Reproduce:
1. Have a blank SAT with registered external CAPSuple
2. Upgrade the Satellite to 7.0 (succeeds)
3. Try to upgrade the Capsule to 7.0
# satellite-maintain upgrade list-versions
# satellite-maintain upgrade check --target-version=7.0 --whitelist="repositories-validate,repositories-setup"
# satellite-maintain upgrade run --target-version=7.0 --whitelist="repositories-validate,repositories-setup"


Actual results:
Fails in Procedures::Installer::Upgrade step

2022-02-02 11:53:29 [NOTICE] [configure] Starting system configuration.
2022-02-02 11:53:42 [NOTICE] [configure] 250 configuration steps out of 1640 steps complete.
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Ca/File[/etc/pki/katello/certs/katello-default-ca.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/katello-default-ca.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Ca/File[/etc/pki/katello/certs/katello-server-ca.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/katello-server-ca.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Cert_key_bundle[/etc/pki/katello/private/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client-bundle.pem]: Could not evaluate: No such file or directory @ rb_sysopen - /root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy]/File[/etc/foreman-proxy/ssl_key.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy.key
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy]/File[/etc/foreman-proxy/ssl_cert.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client]/File[/etc/foreman-proxy/foreman_ssl_key.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client.key
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client]/File[/etc/foreman-proxy/foreman_ssl_cert.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Apache/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-apache]/File[/etc/pki/katello/private/katello-apache.key]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-apache.key
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Apache/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-apache]/File[/etc/pki/katello/certs/katello-apache.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-apache.crt
2022-02-02 11:53:54 [NOTICE] [configure] 500 configuration steps out of 1645 steps complete.
2022-02-02 11:53:54 [ERROR ] [configure] /Stage[main]/Certs::Puppet/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client]/File[/etc/pki/katello/puppet/puppet_client.key]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client.key
2022-02-02 11:53:54 [ERROR ] [configure] /Stage[main]/Certs::Puppet/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client]/File[/etc/pki/katello/puppet/puppet_client.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client.crt
2022-02-02 11:53:55 [NOTICE] [configure] 750 configuration steps out of 1646 steps complete.
2022-02-02 11:53:56 [NOTICE] [configure] 1000 configuration steps out of 1653 steps complete.
2022-02-02 11:53:56 [NOTICE] [configure] 1250 configuration steps out of 1653 steps complete.
2022-02-02 11:53:57 [NOTICE] [configure] 1500 configuration steps out of 1653 steps complete.
2022-02-02 11:54:35 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.


Expected results:
No errors detected and successful upgrade

Comment 2 Evgeni Golov 2022-02-04 12:54:23 UTC

Did you generate a new certs bundle using capsule-certs-generate/foreman-proxy-certs-generate, or did it us the old 6.10 bundle?

Comment 3 Vladimír Sedmík 2022-02-04 15:05:33 UTC

While on 6.10, I generated certs this way:

# capsule-certs-generate --foreman-proxy-fqdn $CAPS --certs-tar $CAPS-certs.tar

Comment 4 Evgeni Golov 2022-02-07 09:24:42 UTC

(In reply to Vladimír Sedmík from comment #3)
> While on 6.10, I generated certs this way:
> 
> # capsule-certs-generate --foreman-proxy-fqdn $CAPS --certs-tar
> $CAPS-certs.tar

So you did not regenerate them on 7.0, correct?

Comment 5 Evgeni Golov 2022-02-07 09:42:11 UTC

I think what happens here, we changed certs deployment in 7.0 and if you have an old, 6.10 bundle, it doesn't have all the needed files.

Comment 6 Vladimír Sedmík 2022-02-08 22:06:06 UTC

Sorry, I missed your comment yesterday. No, I did not regenerate them on 7.0.

Comment 7 Evgeni Golov 2022-02-09 08:49:54 UTC

Can you easily rerun this *with* regeneration of the certs?

Comment 8 Vladimír Sedmík 2022-02-09 11:33:17 UTC

Yes, after certs regeneration on the upgraded SAT (7.0) and copying them on 6.10 capsule, the upgrade succeeded.

Should we cover this need in docs?

Comment 9 Evgeni Golov 2022-02-09 14:30:14 UTC

after talking to Brad, we need to:
- update docs
- add a check in foreman maintain to verify the new cert layout is present