Bug 2083485

Summary: [RHOS 16.2][TLS] Failed to launch a nova instance with "Cannot load certificate ..."
Product: Red Hat OpenStack Reporter: lkuchlan <lkuchlan>
Component: openstack-tripleo-commonAssignee: Adriano Petrich <apetrich>
Status: CLOSED DUPLICATE QA Contact: David Rosenfeld <drosenfe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16.2 (Train)CC: alifshit, bdobreli, cjeanner, mburns, slinaber
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-18 09:50:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description lkuchlan 2022-05-10 08:30:37 UTC
Description of problem:
Creating a nova instance in TLS environment is failed with
"Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/serv
er-key.pem': Error while reading file"

Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20220427.n.3
openstack-tripleo-common-containers-11.7.1-2.20220318011205.b5ef9a5.el8ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy TLS environment
2. Try to create a nova instance

Actual results:
Instance creation is failed with:
Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/serv
er-key.pem': Error while reading file

Expected results:
Creating an instance should be successful.

Additional info:

From compute.log
=================
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [req-e80d2b13-1ac8-480a-b36c-60d1c21ed379 6a47f16d7eb84c32906f7dfbca7de9fd 485a8c5de5314bc087cd68ec1d96cf50 - default default] [instance: d0f97193-7400-42f
c-83c9-01c0995d7daa] Failed to build and run instance: libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-05-09T13:20:55.121551Z qemu-kvm: -object tls-creds-x509,id=vnc-tls
-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa] Traceback (most recent call last):
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2485, in _build_and_run_instance
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     block_device_info=block_device_info)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3780, in spawn
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     cleanup_instance_disks=created_disks)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6683, in _create_domain_and_ne
twork
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     cleanup_instance_disks=cleanup_instance_disks)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     self.force_reraise()
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     six.reraise(self.type_, self.value, self.tb)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     raise value
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6649, in _create_domain_and_ne
twork
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     post_xml_callback=post_xml_callback)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6578, in _create_domain
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     guest.launch(pause=pause)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 149, in launch
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     self._encoded_xml, errors='ignore')
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     self.force_reraise()
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     six.reraise(self.type_, self.value, self.tb)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     raise value
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     return self._domain.createWithFlags(flags)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     rv = execute(f, *args, **kwargs)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     six.reraise(c, e, tb)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     raise value
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     rv = meth(*args, **kwargs)
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1385, in createWithFlags
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]     raise libvirtError('virDomainCreateWithFlags() failed')
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa] libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-05-09T13:20:55.1215
51Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/serv
er-key.pem': Error while reading file.
2022-05-09 13:20:56.723 7 ERROR nova.compute.manager [instance: d0f97193-7400-42fc-83c9-01c0995d7daa]

Comment 3 Artom Lifshitz 2022-05-16 14:54:47 UTC
Indeed, can you retry with puppet-tripleo-11.7.0-2.20220405015037.el8ost and report back if the same problem continues?

Comment 4 Bogdan Dobrelya 2022-05-17 14:00:15 UTC
Please let us know if the aforementioned fix helped

Comment 5 lkuchlan 2022-05-18 06:13:55 UTC
(In reply to Artom Lifshitz from comment #3)
> Indeed, can you retry with puppet-tripleo-11.7.0-2.20220405015037.el8ost and
> report back if the same problem continues?

The latest compose (RHOS-16.2-RHEL-8-20220513.n.2) uses puppet-tripleo-11.7.0-2.20220405015037.el8ost.noarch
and according to the CI results the problem was fixed.

Comment 6 Artom Lifshitz 2022-05-18 09:50:36 UTC

*** This bug has been marked as a duplicate of bug 2079767 ***