Bug 1374978

Summary: Vms are not booting because virtlogd is not started by default after libvirt installation
Product: Red Hat OpenStack Reporter: Alexander Stafeyev <astafeye>
Component: distributionAssignee: Lon Hohberger <lhh>
Status: CLOSED ERRATA QA Contact: Prasanth Anbalagan <panbalag>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: aortega, astafeye, berrange, dasmith, eglynn, jschluet, kchamart, lhh, markmc, mburns, oblaut, panbalag, sasha, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: rcKeywords: TestOnly, Triaged
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:00:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1325503, 1372576    
Bug Blocks:    

Description Alexander Stafeyev 2016-09-11 09:15:11 UTC
Description of problem:
VMs are failing to boot. 
libvirtError: Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory




Version-Release number of selected component (if applicable):
pack stack AIO+compute node deployment. 

openstack-nova-novncproxy-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-scheduler-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-cert-14.0.0-0.20160823074012.21312bf.el7ost.noarch
python-nova-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-compute-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-console-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-conductor-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-common-14.0.0-0.20160823074012.21312bf.el7ost.noarch
puppet-nova-9.1.0-0.20160823051658.5075d8b.el7ost.noarch
openstack-nova-api-14.0.0-0.20160823074012.21312bf.el7ost.noarch
python-novaclient-5.0.0-0.20160802172215.5eb7b65.el7ost.noarch

 rpm -qa | grep packstack 
openstack-packstack-puppet-8.0.0-0.20160818153549.c449b75.1.el7ost.noarch
openstack-packstack-8.0.0-0.20160818153549.c449b75.1.el7ost.noarch

!!!!!rhel 7.3!!!!!


rpm -qa | grep libvirt
libvirt-daemon-driver-nodedev-2.0.0-6.el7.x86_64
libvirt-daemon-driver-storage-2.0.0-6.el7.x86_64
libvirt-daemon-kvm-2.0.0-6.el7.x86_64
libvirt-daemon-driver-nwfilter-2.0.0-6.el7.x86_64
libvirt-client-2.0.0-6.el7.x86_64
libvirt-daemon-driver-interface-2.0.0-6.el7.x86_64
libvirt-daemon-driver-qemu-2.0.0-6.el7.x86_64
libvirt-daemon-config-nwfilter-2.0.0-6.el7.x86_64
libvirt-daemon-driver-secret-2.0.0-6.el7.x86_64
libvirt-daemon-2.0.0-6.el7.x86_64
libvirt-daemon-driver-network-2.0.0-6.el7.x86_64
libvirt-python-2.0.0-2.el7.x86_64

How reproducible:

100%
Steps to Reproduce:
1.delpoy rhos 10 with pack stack 
2.try to boot VMs
3.

Actual results:
Vm is in error state 

Expected results:
VM should be active

Additional info:
solved the issue: 
# systemctl start virtlogd
# systemctl enable  virtlogd



nova logs: 
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [req-7021700c-98aa-4dcf-ba5e-f68783361c15 72799ab7868043c4a4a83bafc489d51b c5b794afc5554b0183ffa6a588082d16 - - -] [instance: c48ef490-7558-426c-881e-62da0e80a897] Instance failed to spawn
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897] Traceback (most recent call last):
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2075, in _build_resources
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     yield resources
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1919, in _build_and_run_instance
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     block_device_info=block_device_info)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2583, in spawn
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     post_xml_callback=gen_confdrive)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4812, in _create_domain_and_network
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     post_xml_callback=post_xml_callback)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4741, in _create_domain
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     guest.launch(pause=pause)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     self._encoded_xml, errors='ignore')
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     self.force_reraise()
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     six.reraise(self.type_, self.value, self.tb)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     return self._domain.createWithFlags(flags)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     rv = execute(f, *args, **kwargs)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     six.reraise(c, e, tb)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     rv = meth(*args, **kwargs)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897] libvirtError: Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
2016-09-11 09:29:32.067 7360 ERROR nova.compute.manager [instance: c48ef490-7558-426c-881e-62da0e80a897]
2016-09-11 09:29:32.068 7360 INFO nova.compute.manager [req-7021700c-98aa-4dcf-ba5e-f68783361c15 72799ab7868043c4a4a83bafc489d51b c5b794afc5554b0183ffa6a588082d16 - - -] [instance: c48ef490-7558-426c-881e-62da0e80a897] Terminating instance



same bug for fedora- https://bugzilla.redhat.com/show_bug.cgi?id=1290357

Comment 2 Daniel Berrangé 2016-09-12 08:34:22 UTC
Fixed by https://bugzilla.redhat.com/show_bug.cgi?id=1325503

Comment 3 Sahid Ferdjaoui 2016-09-12 08:45:12 UTC
We can't really handle that issue in Nova or the packaging, it's related to the installers. Not sure if packstack is officially supported for OSP10 or if it's just osp-director.

Comment 4 Mike Burns 2016-09-14 11:19:22 UTC
Looking at bug 1325503, this is fixed in RHEL.  There was a secondary issue in libvirt filed at bug 1372576 that should resolve the final piece of this issue.  

Marking this testonly and moving to ON_QA, though I certainly wouldn't argue if people want to close it as a duplicate of one of the other bugs.

Comment 10 Alexander Stafeyev 2016-09-27 05:48:56 UTC
do we need to wait until RC-1 to CDN, beta channel, that scheduled to Thu 2016-10-13 or the code already available

Comment 11 Prasanth Anbalagan 2016-10-05 09:15:56 UTC
No issues in RHOS10/RHEL7.3 setup as below. Please feel free to move bug to Verified.

[user@serverX ~]# 
[user@serverX ~]# rpm -qa | grep libvirt
libvirt-daemon-config-nwfilter-2.0.0-9.el7.x86_64
libvirt-daemon-driver-nwfilter-2.0.0-9.el7.x86_64
libvirt-daemon-driver-network-2.0.0-9.el7.x86_64
libvirt-daemon-driver-nodedev-2.0.0-9.el7.x86_64
libvirt-daemon-driver-secret-2.0.0-9.el7.x86_64
libvirt-client-2.0.0-9.el7.x86_64
libvirt-daemon-2.0.0-9.el7.x86_64
libvirt-daemon-driver-interface-2.0.0-9.el7.x86_64
libvirt-daemon-driver-storage-2.0.0-9.el7.x86_64
libvirt-python-2.0.0-2.el7.x86_64
libvirt-daemon-driver-qemu-2.0.0-9.el7.x86_64
libvirt-daemon-kvm-2.0.0-9.el7.x86_64

[user@serverX ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 Beta (Maipo)

[user@serverX ~]# 
[user@serverX ~]# yum list installed | grep openstack-nova
openstack-nova-api.noarch             1:14.0.0-0.20160929203854.59653c6.el7ost
openstack-nova-cert.noarch            1:14.0.0-0.20160929203854.59653c6.el7ost
openstack-nova-common.noarch          1:14.0.0-0.20160929203854.59653c6.el7ost
openstack-nova-compute.noarch         1:14.0.0-0.20160929203854.59653c6.el7ost
openstack-nova-conductor.noarch       1:14.0.0-0.20160929203854.59653c6.el7ost
openstack-nova-console.noarch         1:14.0.0-0.20160929203854.59653c6.el7ost
openstack-nova-novncproxy.noarch      1:14.0.0-0.20160929203854.59653c6.el7ost
openstack-nova-scheduler.noarch       1:14.0.0-0.20160929203854.59653c6.el7ost
[user@serverX ~]# 
[user@serverX ]# nova list --all
+--------------------------------------+-----------------------------+----------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                        | Tenant ID                        | Status | Task State | Power State | Networks            |
+--------------------------------------+-----------------------------+----------------------------------+--------+------------+-------------+---------------------+
| a9b7d5ca-22e4-4b63-9d59-4a4a61aa852d | tempest-instance-1224273146 | 64fa95f6dcf74ee99231d49e6ae2fe15 | ACTIVE | -          | Running     | public=172.24.4.234 |
| 8ef95078-bd90-4f5b-b184-dcafd5eb3881 | tempest-instance-1608873158 | 7c23f669091c4eedbd0b753dd82717d2 | ACTIVE | -          | Running     | public=172.24.4.235 |
| ae57a4c2-93bc-4e0e-a2ef-16b18a973719 | tempest-instance-1961403439 | 0dc6ffaa785c46ecbe0fa59530463abb | ACTIVE | -          | Running     | public=172.24.4.238 |
| be74ec6f-a037-406f-b4c1-b12464546c7b | tempest-instance-1986880685 | f9d934a290be449989f8c8c5f936b712 | ACTIVE | -          | Running     | public=172.24.4.226 |
| 4d3ed454-e9df-465b-8be2-e69d22a9f7b2 | tempest-instance-452607875  | 37202087cfe847a4aa23a69006c9da09 | ACTIVE | -          | Running     | public=172.24.4.237 |
| 279e00e6-1c0b-4f0b-9568-22804b7d1216 | tempest-instance-871610042  | 36d0c89d541f4edb99ea260672f21076 | ACTIVE | -          | Running     | public=172.24.4.232 |
+--------------------------------------+-----------------------------+----------------------------------+--------+------------+-------------+---------------------+
[user@serverX ]#

Comment 14 errata-xmlrpc 2016-12-14 16:00:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html