Bug 1019360 - vdsm not starts properly after clean installation.
vdsm not starts properly after clean installation.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
unspecified
Unspecified Unspecified
urgent Severity urgent
: ---
: 3.3.0
Assigned To: Yaniv Bronhaim
Leonid Natapov
infra
: Regression, Reopened, Triaged
Depends On:
Blocks: 3.3snap2
  Show dependency treegraph
 
Reported: 2013-10-15 10:48 EDT by Leonid Natapov
Modified: 2016-02-10 14:17 EST (History)
9 users (show)

See Also:
Fixed In Version: is22
Doc Type: Bug Fix
Doc Text:
VDSM did not start properly upon clean installation, which could lead to failure when deploying the hosted engine. This update changes the return method from the vdsmd start function, so VDSM starts properly.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-21 11:18:35 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm log file (32.93 KB, text/plain)
2013-10-15 10:49 EDT, Leonid Natapov
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 20578 None None None Never

  None (edit)
Description Leonid Natapov 2013-10-15 10:48:10 EDT
Description of problem:
vdsm not starts properly after clean installation.
when starting vdsm daemon for the first time getting following exception:
------------

[root@nott-vds1 ~]# service vdsmd start
Starting libvirtd daemon:                                  [  OK  ]
supervdsm start                                            [  OK  ]
vdsm: Running run_init_hooks                                      
vdsm: Running gencerts                                            
vdsm: Running reconfigure_sanlock                                  
vdsm: Running reconfigure_libvirt                                  
Stopping libvirtd daemon: [  OK  ]                                
Reconfiguration of libvirt is done.                                
 
To start working with the new configuration, execute:
'vdsm-tool libvirt-configure-services-restart'      
This will manage restarting of the following services:
libvirtd, supervdsmd                                  
 
diff: /etc/init/libvirtd.conf: No such file or directory
vdsm: Running syslog_available                          
vdsm: Running nwfilter                                  
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
raceback (most recent call last):                                                                            
  File "/usr/bin/vdsm-tool", line 143, in <module>                                                            
    sys.exit(main())                                                                                          
  File "/usr/bin/vdsm-tool", line 140, in main                                                                
    return tool_command[cmd]["command"](*args[1:])                                                            
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/nwfilter.py", line 35, in main                          
    conn = libvirtconnection.get(None, False)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 132, in get
    conn = utils.retry(libvirtOpenAuth, timeout=10, sleep=0.2)
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 903, in retry
    return func()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
vdsm: failed to execute nwfilter, error code 1
Starting up vdsm daemon:
vdsm start                                                 [  OK  ]

---------

As a result /var/run/vdsm/pools directory not created and host can't connect to pool. Restart vdsm solves the problem.

Version-Release number of selected component (if applicable):
vdsm-4.13.0-0.2.beta1.el6ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1.remove vdsm from the host
2.install vdsm
3.start vdsm daemon



Additional info:
Comment 1 Leonid Natapov 2013-10-15 10:48:43 EDT
vdsm.log attached
Comment 2 Leonid Natapov 2013-10-15 10:49:05 EDT
Created attachment 812567 [details]
vdsm log file
Comment 4 Barak 2013-10-16 08:40:25 EDT
If you install & start vdsm manually it explains simply what needs to be done in order to start it.

This applies only for manual work - through host-deploy it is done automatically.

Hence moving to CLOSE NOTABUG
Comment 6 Dan Kenigsberg 2013-10-16 09:23:00 EDT
Frankly, this is a dup of bug 1013371. But for ovirt-3.3.1 we should not attempt to backport the relevant patches from master. We'd better just re-introduce libvirtd restart after an actual configuration thereof.
Comment 8 Yaniv Bronhaim 2013-10-23 11:35:29 EDT
This is the same issue as reported in Bug 1013371. This was fixed and the exception won't appear with http://gerrit.ovirt.org/#/c/19884. 

This fix was not backported to ovirt-3.3, for that please ack rhevm-3.3 flag
Comment 9 Yaniv Bronhaim 2013-10-23 11:37:49 EDT
btw, Dan, I prefer not to re-introduce the libvirt restart. It's simple patch to merge in without any expected regressions that I can think of..
Comment 11 Yaniv Bronhaim 2013-10-27 07:51:47 EDT
Still needs appropriate acks to proceed.
Comment 12 Leonid Natapov 2013-11-17 09:37:33 EST
is23. same error on vdsmd start.

vdsm-4.13.0-0.8.beta1.el6ev.x86_64
libvirt-0.10.2-29.el6.x86_64
Comment 13 Yaniv Bronhaim 2013-11-17 10:18:28 EST
Please share copy paste of output, libvirt log, and status of libvirtd. It means during vdsmd service start you could not connect to libvirt. It can be many issues related to libvirt but this bug talks about failing the start and not move forward with the other pre-start tasks. If not, lets investigate 

please provide the information asap.
Comment 14 Leonid Natapov 2013-11-17 10:57:45 EST
[root@green-vdsc libvirt]# initctl status libvirtd
initctl: Unknown job: libvirtd
[root@green-vdsc libvirt]# service vdsmd status
VDS daemon is not running
[root@green-vdsc ~]# service vdsmd start
Starting libvirtd daemon:                                  [  OK  ]
supervdsm start                                            [  OK  ]
vdsm: Running run_init_hooks
vdsm: Running gencerts
vdsm: Running reconfigure_sanlock
vdsm: Running reconfigure_libvirt
Stopping libvirtd daemon: [  OK  ]
Reconfiguration of libvirt is done.

To start working with the new configuration, execute:
'vdsm-tool libvirt-configure-services-restart'
This will manage restarting of the following services:
libvirtd, supervdsmd

diff: /etc/init/libvirtd.conf: No such file or directory
vdsm: Running prepare_transient_repository
vdsm: Running syslog_available
vdsm: Running nwfilter
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
libvir: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Traceback (most recent call last):
  File "/usr/bin/vdsm-tool", line 143, in <module>
    sys.exit(main())
  File "/usr/bin/vdsm-tool", line 140, in main
    return tool_command[cmd]["command"](*args[1:])
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/nwfilter.py", line 35, in main
    conn = libvirtconnection.get(None, False)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 132, in get
    conn = utils.retry(libvirtOpenAuth, timeout=10, sleep=0.2)
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 903, in retry
    return func()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
vdsm: failed to execute nwfilter, error code 1
vdsm start                                                 [FAILED]
------------------------------------------
after unsuccessful vdsdm start
libvirt log includes only the following:

013-11-17 15:54:48.847+0000: 30726: info : libvirt version: 0.10.2, package: 29.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2013-10-09-06:25:35, x86-026.build.eng.bos.redhat.com)
2013-11-17 15:54:48.847+0000: 30726: error : virNWFilterSnoopLeaseFileRefresh:1903 : open("/var/run/libvirt/network/nwfilter.ltmp"): No such file or directory
Comment 15 Leonid Natapov 2013-11-17 11:00:09 EST
You can easily reproduce it by cleaning the host from vdsm and libvirt yum remove vdsm*,yum remove libvirt* and installing them again from is23.
Comment 16 Leonid Natapov 2013-11-18 05:52:53 EST
this particluar bug was fixed so I am verifying this BZ. However vdsm still unable to start when starting it after fresh install but the reason is different. Opening another BZ.
Comment 17 errata-xmlrpc 2014-01-21 11:18:35 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html

Note You need to log in before you can comment on or make changes to this bug.