Bug 1019360

Summary: vdsm not starts properly after clean installation.
Product: Red Hat Enterprise Virtualization Manager Reporter: Leonid Natapov <lnatapov>
Component: vdsmAssignee: Yaniv Bronhaim <ybronhei>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: urgent Docs Contact:
Priority: urgent    
Version: unspecifiedCC: abaron, acathrow, bazulay, eedri, iheim, lnatapov, lpeer, pstehlik, yeylon
Target Milestone: ---Keywords: Regression, Reopened, Triaged
Target Release: 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: is22 Doc Type: Bug Fix
Doc Text:
VDSM did not start properly upon clean installation, which could lead to failure when deploying the hosted engine. This update changes the return method from the vdsmd start function, so VDSM starts properly.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-21 16:18:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1032811    
Attachments:
Description Flags
vdsm log file none

Description Leonid Natapov 2013-10-15 14:48:10 UTC
Description of problem:
vdsm not starts properly after clean installation.
when starting vdsm daemon for the first time getting following exception:
------------

[root@nott-vds1 ~]# service vdsmd start
Starting libvirtd daemon:                                  [  OK  ]
supervdsm start                                            [  OK  ]
vdsm: Running run_init_hooks                                      
vdsm: Running gencerts                                            
vdsm: Running reconfigure_sanlock                                  
vdsm: Running reconfigure_libvirt                                  
Stopping libvirtd daemon: [  OK  ]                                
Reconfiguration of libvirt is done.                                
 
To start working with the new configuration, execute:
'vdsm-tool libvirt-configure-services-restart'      
This will manage restarting of the following services:
libvirtd, supervdsmd                                  
 
diff: /etc/init/libvirtd.conf: No such file or directory
vdsm: Running syslog_available                          
vdsm: Running nwfilter                                  
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
raceback (most recent call last):                                                                            
  File "/usr/bin/vdsm-tool", line 143, in <module>                                                            
    sys.exit(main())                                                                                          
  File "/usr/bin/vdsm-tool", line 140, in main                                                                
    return tool_command[cmd]["command"](*args[1:])                                                            
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/nwfilter.py", line 35, in main                          
    conn = libvirtconnection.get(None, False)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 132, in get
    conn = utils.retry(libvirtOpenAuth, timeout=10, sleep=0.2)
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 903, in retry
    return func()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
vdsm: failed to execute nwfilter, error code 1
Starting up vdsm daemon:
vdsm start                                                 [  OK  ]

---------

As a result /var/run/vdsm/pools directory not created and host can't connect to pool. Restart vdsm solves the problem.

Version-Release number of selected component (if applicable):
vdsm-4.13.0-0.2.beta1.el6ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1.remove vdsm from the host
2.install vdsm
3.start vdsm daemon



Additional info:

Comment 1 Leonid Natapov 2013-10-15 14:48:43 UTC
vdsm.log attached

Comment 2 Leonid Natapov 2013-10-15 14:49:05 UTC
Created attachment 812567 [details]
vdsm log file

Comment 4 Barak 2013-10-16 12:40:25 UTC
If you install & start vdsm manually it explains simply what needs to be done in order to start it.

This applies only for manual work - through host-deploy it is done automatically.

Hence moving to CLOSE NOTABUG

Comment 6 Dan Kenigsberg 2013-10-16 13:23:00 UTC
Frankly, this is a dup of bug 1013371. But for ovirt-3.3.1 we should not attempt to backport the relevant patches from master. We'd better just re-introduce libvirtd restart after an actual configuration thereof.

Comment 8 Yaniv Bronhaim 2013-10-23 15:35:29 UTC
This is the same issue as reported in Bug 1013371. This was fixed and the exception won't appear with http://gerrit.ovirt.org/#/c/19884. 

This fix was not backported to ovirt-3.3, for that please ack rhevm-3.3 flag

Comment 9 Yaniv Bronhaim 2013-10-23 15:37:49 UTC
btw, Dan, I prefer not to re-introduce the libvirt restart. It's simple patch to merge in without any expected regressions that I can think of..

Comment 11 Yaniv Bronhaim 2013-10-27 11:51:47 UTC
Still needs appropriate acks to proceed.

Comment 12 Leonid Natapov 2013-11-17 14:37:33 UTC
is23. same error on vdsmd start.

vdsm-4.13.0-0.8.beta1.el6ev.x86_64
libvirt-0.10.2-29.el6.x86_64

Comment 13 Yaniv Bronhaim 2013-11-17 15:18:28 UTC
Please share copy paste of output, libvirt log, and status of libvirtd. It means during vdsmd service start you could not connect to libvirt. It can be many issues related to libvirt but this bug talks about failing the start and not move forward with the other pre-start tasks. If not, lets investigate 

please provide the information asap.

Comment 14 Leonid Natapov 2013-11-17 15:57:45 UTC
[root@green-vdsc libvirt]# initctl status libvirtd
initctl: Unknown job: libvirtd
[root@green-vdsc libvirt]# service vdsmd status
VDS daemon is not running
[root@green-vdsc ~]# service vdsmd start
Starting libvirtd daemon:                                  [  OK  ]
supervdsm start                                            [  OK  ]
vdsm: Running run_init_hooks
vdsm: Running gencerts
vdsm: Running reconfigure_sanlock
vdsm: Running reconfigure_libvirt
Stopping libvirtd daemon: [  OK  ]
Reconfiguration of libvirt is done.

To start working with the new configuration, execute:
'vdsm-tool libvirt-configure-services-restart'
This will manage restarting of the following services:
libvirtd, supervdsmd

diff: /etc/init/libvirtd.conf: No such file or directory
vdsm: Running prepare_transient_repository
vdsm: Running syslog_available
vdsm: Running nwfilter
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Traceback (most recent call last):
  File "/usr/bin/vdsm-tool", line 143, in <module>
    sys.exit(main())
  File "/usr/bin/vdsm-tool", line 140, in main
    return tool_command[cmd]["command"](*args[1:])
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/nwfilter.py", line 35, in main
    conn = libvirtconnection.get(None, False)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 132, in get
    conn = utils.retry(libvirtOpenAuth, timeout=10, sleep=0.2)
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 903, in retry
    return func()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
vdsm: failed to execute nwfilter, error code 1
vdsm start                                                 [FAILED]
------------------------------------------
after unsuccessful vdsdm start
libvirt log includes only the following:

013-11-17 15:54:48.847+0000: 30726: info : libvirt version: 0.10.2, package: 29.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2013-10-09-06:25:35, x86-026.build.eng.bos.redhat.com)
2013-11-17 15:54:48.847+0000: 30726: error : virNWFilterSnoopLeaseFileRefresh:1903 : open("/var/run/libvirt/network/nwfilter.ltmp"): No such file or directory

Comment 15 Leonid Natapov 2013-11-17 16:00:09 UTC
You can easily reproduce it by cleaning the host from vdsm and libvirt yum remove vdsm*,yum remove libvirt* and installing them again from is23.

Comment 16 Leonid Natapov 2013-11-18 10:52:53 UTC
this particluar bug was fixed so I am verifying this BZ. However vdsm still unable to start when starting it after fresh install but the reason is different. Opening another BZ.

Comment 17 errata-xmlrpc 2014-01-21 16:18:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html