2049893 – Capsule upgrade from 6.10 to 6.11 fails due to changed certs deployment

Bug 2049893 - Capsule upgrade from 6.10 to 6.11 fails due to changed certs deployment [NEEDINFO]

Summary: Capsule upgrade from 6.10 to 6.11 fails due to changed certs deployment

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Documentation
Sub Component:
Version:	6.11.0
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	6.11.0
Assignee:	Marie Hornickova
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	2052867
TreeView+	depends on / blocked

Reported:	2022-02-02 21:11 UTC by Vladimír Sedmík
Modified:	2022-07-22 19:35 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2052867 (view as bug list)
Environment:
Last Closed:	2022-07-22 19:35:32 UTC
Target Upstream Version:
Embargoed:
Flags:	mdolezel: needinfo? (vferschm)

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	SATDOC-691	0	None	None	None	2022-03-14 19:30:00 UTC

Description Vladimír Sedmík 2022-02-02 21:11:55 UTC

Description of problem:
Capsule upgrade from 6.10.2 to 7.0 fails in Procedures::Installer::Upgrade


Version-Release number of selected component (if applicable):
original Sat/Cap - 6.10.2 snap 2
DF repos for upg - 7.0.0 snap 7


How reproducible:
always


Steps to Reproduce:
1. Have a blank SAT with registered external CAPSuple
2. Upgrade the Satellite to 7.0 (succeeds)
3. Try to upgrade the Capsule to 7.0
# satellite-maintain upgrade list-versions
# satellite-maintain upgrade check --target-version=7.0 --whitelist="repositories-validate,repositories-setup"
# satellite-maintain upgrade run --target-version=7.0 --whitelist="repositories-validate,repositories-setup"


Actual results:
Fails in Procedures::Installer::Upgrade step

2022-02-02 11:53:29 [NOTICE] [configure] Starting system configuration.
2022-02-02 11:53:42 [NOTICE] [configure] 250 configuration steps out of 1640 steps complete.
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Ca/File[/etc/pki/katello/certs/katello-default-ca.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/katello-default-ca.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Ca/File[/etc/pki/katello/certs/katello-server-ca.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/katello-server-ca.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Cert_key_bundle[/etc/pki/katello/private/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client-bundle.pem]: Could not evaluate: No such file or directory @ rb_sysopen - /root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy]/File[/etc/foreman-proxy/ssl_key.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy.key
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy]/File[/etc/foreman-proxy/ssl_cert.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client]/File[/etc/foreman-proxy/foreman_ssl_key.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client.key
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Foreman_proxy/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client]/File[/etc/foreman-proxy/foreman_ssl_cert.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-foreman-proxy-client.crt
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Apache/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-apache]/File[/etc/pki/katello/private/katello-apache.key]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-apache.key
2022-02-02 11:53:50 [ERROR ] [configure] /Stage[main]/Certs::Apache/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-apache]/File[/etc/pki/katello/certs/katello-apache.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-apache.crt
2022-02-02 11:53:54 [NOTICE] [configure] 500 configuration steps out of 1645 steps complete.
2022-02-02 11:53:54 [ERROR ] [configure] /Stage[main]/Certs::Puppet/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client]/File[/etc/pki/katello/puppet/puppet_client.key]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client.key
2022-02-02 11:53:54 [ERROR ] [configure] /Stage[main]/Certs::Puppet/Certs::Keypair[dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client]/File[/etc/pki/katello/puppet/puppet_client.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///root/ssl-build/dhcp-2-76.vms.sat.rdu2.redhat.com/dhcp-2-76.vms.sat.rdu2.redhat.com-puppet-client.crt
2022-02-02 11:53:55 [NOTICE] [configure] 750 configuration steps out of 1646 steps complete.
2022-02-02 11:53:56 [NOTICE] [configure] 1000 configuration steps out of 1653 steps complete.
2022-02-02 11:53:56 [NOTICE] [configure] 1250 configuration steps out of 1653 steps complete.
2022-02-02 11:53:57 [NOTICE] [configure] 1500 configuration steps out of 1653 steps complete.
2022-02-02 11:54:35 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.


Expected results:
No errors detected and successful upgrade

Comment 2 Evgeni Golov 2022-02-04 12:54:23 UTC

Did you generate a new certs bundle using capsule-certs-generate/foreman-proxy-certs-generate, or did it us the old 6.10 bundle?

Comment 3 Vladimír Sedmík 2022-02-04 15:05:33 UTC

While on 6.10, I generated certs this way:

# capsule-certs-generate --foreman-proxy-fqdn $CAPS --certs-tar $CAPS-certs.tar

Comment 4 Evgeni Golov 2022-02-07 09:24:42 UTC

(In reply to Vladimír Sedmík from comment #3)
> While on 6.10, I generated certs this way:
> 
> # capsule-certs-generate --foreman-proxy-fqdn $CAPS --certs-tar
> $CAPS-certs.tar

So you did not regenerate them on 7.0, correct?

Comment 5 Evgeni Golov 2022-02-07 09:42:11 UTC

I think what happens here, we changed certs deployment in 7.0 and if you have an old, 6.10 bundle, it doesn't have all the needed files.

Comment 6 Vladimír Sedmík 2022-02-08 22:06:06 UTC

Sorry, I missed your comment yesterday. No, I did not regenerate them on 7.0.

Comment 7 Evgeni Golov 2022-02-09 08:49:54 UTC

Can you easily rerun this *with* regeneration of the certs?

Comment 8 Vladimír Sedmík 2022-02-09 11:33:17 UTC

Yes, after certs regeneration on the upgraded SAT (7.0) and copying them on 6.10 capsule, the upgrade succeeded.

Should we cover this need in docs?

Comment 9 Evgeni Golov 2022-02-09 14:30:14 UTC

after talking to Brad, we need to:
- update docs
- add a check in foreman maintain to verify the new cert layout is present

Note You need to log in before you can comment on or make changes to this bug.