Bug 1324016

Summary: "Failed to Establish Libvirt Connection" on RHEV-H 7.2 for RHEV 3.6.4
Product: Red Hat Enterprise Virtualization Manager Reporter: cshao <cshao>
Component: ovirt-nodeAssignee: Douglas Schilling Landgraf <dougsland>
Status: CLOSED CURRENTRELEASE QA Contact: cshao <cshao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.4CC: cshao, fdeutsch, gklein, huzhao, leiwang, lsurette, weiwang, yaniwang, ycui, ykaul
Target Milestone: ovirt-3.6.9   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 04:28:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
failed to Establish libvirt connection
none
/var/log/*.*
none
all log info
none
0602 none

Description cshao 2016-04-05 10:48:51 UTC
Description of problem:
"Failed to Establish Libvirt Connection" on RHEV-H 7.2 for RHEV 3.6.4, it is hard to reproduced.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160328.0.el7ev)
ovirt-node-3.6.1-8.0.el7ev.noarch
libvirt-1.2.17-13.el7_2.3.x86_64
vdsm-4.17.23.2-1.el7ev.noarch


How reproducible:
5%, it is hard to reproduced.

Steps to Reproduce:
1. TUI clean installed rhevh successful
2. Login TUI
3. Setup host name
4. Setup network via dhcp
5. Please see attachment for more detail.

Actual results:
"Failed to Establish Libvirt Connection" on RHEV-H 7.2 for RHEV 3.6.4,

Expected results:
without such error



Additional info:
# systemctl status libvirtd vdsmd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
           └─unlimited-core.conf
   Active: failed (Result: start-limit) since Tue 2016-04-05 07:16:49 UTC; 8min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
  Process: 18450 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=6)
 Main PID: 18450 (code=exited, status=6)

Apr 05 07:16:49 localhost systemd[1]: Failed to start Virtualization daemon.
Apr 05 07:16:49 localhost systemd[1]: Unit libvirtd.service entered failed state.
Apr 05 07:16:49 localhost systemd[1]: libvirtd.service failed.
Apr 05 07:16:49 localhost systemd[1]: libvirtd.service holdoff time over, scheduling restart.
Apr 05 07:16:49 localhost systemd[1]: start request repeated too quickly for libvirtd.service
Apr 05 07:16:49 localhost systemd[1]: Failed to start Virtualization daemon.
Apr 05 07:16:49 localhost systemd[1]: Unit libvirtd.service entered failed state.
Apr 05 07:16:49 localhost systemd[1]: libvirtd.service failed.

● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Apr 05 07:16:48 localhost systemd[1]: Dependency failed for Virtual Desktop Server Manager.
Apr 05 07:16:48 localhost systemd[1]: Job vdsmd.service/start failed with result 'dependency'.
Apr 05 07:16:48 localhost systemd[1]: Dependency failed for Virtual Desktop Server Manager.
Apr 05 07:16:48 localhost systemd[1]: Job vdsmd.service/start failed with result 'dependency'.
Apr 05 07:16:49 localhost systemd[1]: Dependency failed for Virtual Desktop Server Manager.
Apr 05 07:16:49 localhost systemd[1]: Job vdsmd.service/start failed with result 'dependency'.
Apr 05 07:16:49 localhost systemd[1]: Dependency failed for Virtual Desktop Server Manager.
Apr 05 07:16:49 localhost systemd[1]: Job vdsmd.service/start failed with result 'dependency'.
Apr 05 07:16:49 localhost systemd[1]: Dependency failed for Virtual Desktop Server Manager.
Apr 05 07:16:49 localhost systemd[1]: Job vdsmd.service/start failed with result 'dependency'.
[root@localhost admin]# 


# systemctl restart libvirtd
Job for libvirtd.service failed because the control process exited with error code. See "systemctl status libvirtd.service" and "journalctl -xe" for details.
[root@localhost admin]# 



# systemctl status supervdsmd
● supervdsmd.service - Auxiliary vdsm service for running helper functions as root
   Loaded: loaded (/usr/lib/systemd/system/supervdsmd.service; static; vendor preset: enabled)
   Active: failed (Result: start-limit) since Tue 2016-04-05 07:25:57 UTC; 22s ago
  Process: 18679 ExecStart=/usr/share/vdsm/daemonAdapter /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock (code=killed, signal=TERM)
 Main PID: 18679 (code=killed, signal=TERM)

Apr 05 07:25:57 localhost systemd[1]: Started Auxiliary vdsm service for running helper functions as root.
Apr 05 07:25:57 localhost systemd[1]: Starting Auxiliary vdsm service for running helper functions as root...
Apr 05 07:25:57 localhost systemd[1]: Stopping Auxiliary vdsm service for running helper functions as root...
Apr 05 07:25:57 localhost systemd[1]: start request repeated too quickly for supervdsmd.service
Apr 05 07:25:57 localhost systemd[1]: Failed to start Auxiliary vdsm service for running helper functions as root.
Apr 05 07:25:57 localhost systemd[1]: Unit supervdsmd.service entered failed state.
Apr 05 07:25:57 localhost systemd[1]: supervdsmd.service failed.
[root@localhost admin]#

Comment 1 cshao 2016-04-05 10:49:42 UTC
Created attachment 1143772 [details]
failed to Establish libvirt connection

Comment 2 cshao 2016-04-05 10:50:14 UTC
Created attachment 1143774 [details]
/var/log/*.*

Comment 3 Fabian Deutsch 2016-04-06 13:21:55 UTC
Can you please retry this with RHEV 3.6.5?

And can VMs be spawned?

Comment 4 cshao 2016-04-07 10:16:13 UTC
(In reply to Fabian Deutsch from comment #3)
> Can you please retry this with RHEV 3.6.5?
> 
The bug is hard to reproduce, and I can't reproduce this issue on rhev-hypervisor7-7.2-20160406.0 at present.


> And can VMs be spawned?
I will give result ASAP once reproduced.

Comment 5 Ryan Barry 2016-04-07 12:16:39 UTC
Can you please also post the contents of /etc/pki?

It looks like the clock on the system was out of sync when the vdsm server certificate was generated, and libvirt is refusing it (since the date is probably in the future), so libvirt doesn't start, and vdsm can't communicate with it.

Comment 6 cshao 2016-04-08 02:28:28 UTC
(In reply to Ryan Barry from comment #5)
> Can you please also post the contents of /etc/pki?
> 
> It looks like the clock on the system was out of sync when the vdsm server
> certificate was generated, and libvirt is refusing it (since the date is
> probably in the future), so libvirt doesn't start, and vdsm can't
> communicate with it.

The original env has been destroyed due to new build testing, but I still will tries to provide the contents of /etc/pki if I can reproduce this issue.

Comment 7 cshao 2016-06-06 05:50:28 UTC
Created attachment 1165049 [details]
all log info

Reproduce this issue on RHEV-H 7.2 for RHEV 3.6.7 (rhev-hypervisor7-7.2-20160602.0.iso ) + rhev-hypervisor7-7.2-20160602.0.iso.

Attach all log info for reference.
/var/log/*.*
/tmp/ovirt.log
/etc/pki/*
sosreport

Comment 8 cshao 2016-06-06 05:51:54 UTC
Created attachment 1165050 [details]
0602

Comment 9 Fabian Deutsch 2016-07-26 09:10:17 UTC
Chen, can you please retry this on one of the more recent 3.6.z builds?

Comment 10 cshao 2016-07-26 09:25:45 UTC
(In reply to Fabian Deutsch from comment #9)
> Chen, can you please retry this on one of the more recent 3.6.z builds?

ok, the bug is hard to reproduce, but I will try and let you know the result ASAP.

Comment 11 cshao 2016-07-27 04:18:10 UTC
(In reply to Fabian Deutsch from comment #9)
> Chen, can you please retry this on one of the more recent 3.6.z builds?

Test version:
rhev-hypervisor7-7.2-20160711.0
ovirt-node-3.6.1-13.0.el7ev.noarch

After several times testing on different types of machine, can't reproduce this bug.

Comment 12 Douglas Schilling Landgraf 2016-07-27 04:28:44 UTC
(In reply to shaochen from comment #11)
> (In reply to Fabian Deutsch from comment #9)
> > Chen, can you please retry this on one of the more recent 3.6.z builds?
> 
> Test version:
> rhev-hypervisor7-7.2-20160711.0
> ovirt-node-3.6.1-13.0.el7ev.noarch
> 
> After several times testing on different types of machine, can't reproduce
> this bug.

Thanks shaochen, closing this bug for now, fell free to re-open anytime.