Bug 1817880

To me this looks like libvirt is already configured for RHV, but the needed files are missing (the CA according to the error message).

Could you please do the following:
Check /etc/libvirt/libvirtd.conf

Delete everything below this line:
## beginning of configuration section by vdsm-*

Do the same for qemu.conf

Restart libvirtd. This should work now. You can then restart the ovirt-hosted-engine setup.

Alternatively you can reinstall the host from scratch.

The error looks very much like this system started doing the ovirt-hosted-engine and was aborted after the first step, which leaves libvirt configuration in a broken state.
You can check for other config files that already have been adjusted for RHV by just looking for the above string.

Let me know if this helped.

duplicate of bug 1810882?

or is this at the beginning of the deployment? then martin's comment is most likely, does it happen on a clean host for a first run?

This happened to me when trying to re-deploy a hosted-engine.
Not sure what part of configuration is not removed from previous installation but I would guess it's something like that.

The issue why libvirt wont start is:
libvirtd[34641]: Cannot read certificate '/etc/pki/libvirt/servercert.pem': No such file or directory

As I see during HE deployment, libvirt is stopped and reconfigured.
Post reconfiguration, libvirtd never comes up.

May 13 05:32:45 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: Stopping Virtualization daemon...
May 13 05:32:45 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: Stopped Virtualization daemon.
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: libvirtd.service: Found left-over process 2993 (dnsmasq) in control group while starting unit. Ignoring.
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: libvirtd.service: Found left-over process 2994 (dnsmasq) in control group while starting unit. Ignoring.
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: Starting Virtualization daemon...
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com libvirtd[14000]: libvirt version: 6.0.0, package: 17.module+el8.2.0+6257+0d066c28 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-04-08-16:>
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com libvirtd[14000]: hostname: rhsqa-grafton11-nic2.lab.eng.blr.redhat.com
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com libvirtd[14000]: Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: libvirtd.service: Main process exited, code=exited, status=6/NOTCONFIGURED
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: libvirtd.service: Failed with result 'exit-code'.
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: Failed to start Virtualization daemon.
May 13 05:32:54 rhsqa-grafton11-nic2.lab.eng.blr.redhat.com systemd[1]: libvirtd.service: Service RestartSec=100ms expired, scheduling restart.


I will also upload the HE deploy logs shortly

I have started libvirtd with debug logs enabled, and what I could observe as error is, /etc/pki/CA/cacert.pem is missing in the configuration, 
because of which libvirtd is unable to start

2020-05-13 05:44:25.258+0000: 14192: error : virNetTLSContextCheckCertFile:110 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory
2020-05-13 05:44:25.470+0000: 14208: error : virNetTLSContextCheckCertFile:110 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory
2020-05-13 05:44:25.837+0000: 14224: error : virNetTLSContextCheckCertFile:110 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory
2020-05-13 05:44:26.086+0000: 14240: error : virNetTLSContextCheckCertFile:110 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory
2020-05-13 05:44:26.339+0000: 14256: error : virNetTLSContextCheckCertFile:110 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory

One more additional information that I observed.

For Hyperconverged case, we are trying to generate the multipath configuration file - /etc/multipath.conf,
for that purpose - `# vdsm-tool configure --force` command is run , after which if Hosted Engine deployment is attempted,
then it fails.


Is it the right approach to make use of 'vdsm-tool configure --force' to generate - /etc/multipath.con ?

I see that we can configure only multipath using 'vdsm-tool'
# vdsm-tool configure --module multipath

I am trying to validate this scenario with RHHI-V deployment, if this works well,
then instead of configuring everything using 'vdsm-tool configure', we can only
configure 'multipath', which would resolve the problem

Let me test and update the results

I have done with my testing, with this latest configuration to generate multipath configuration
with 'vdsm-tool configure --module multipath' and with this I do not see this problem.

I think we can close this bug in this case, after addressing Petr Matyas's problem

@Petr Matyas, Do you still see this problem with reinstallation of Hosted Engine ?

I haven't tried since, but I would assume it's still there.
Not sure how much is reinstallation of hosted engine supported, but it can surely cause problems and not everyone want's to reinstall their machine every time there is some problem/change.

Hosted-Engine can be reinstalled after performing cleanup with this utility:
/usr/sbin/ovirt-hosted-engine-cleanup

this also should fix libvirtd missing cert failures .
SATHEESARAN,can you please confirm?

duplicate

*** This bug has been marked as a duplicate of bug 1634742 ***

(In reply to Evgeny Slutsky from comment #12)
> Hosted-Engine can be reinstalled after performing cleanup with this utility:
> /usr/sbin/ovirt-hosted-engine-cleanup
> 
> this also should fix libvirtd missing cert failures .
> SATHEESARAN,can you please confirm?

Yes, Evgeny, that works