Bug 1459976 – openstack-nova: repeating errors on compute: Jun 8 05:33:38 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1459976 - openstack-nova: repeating errors on compute: Jun 8 05:33:38 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory

Summary:	openstack-nova: repeating errors on compute: Jun 8 05:33:38 localhost journa...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates (Show other bugs)
Sub Component:
Version:	12.0 (Pike)
Hardware:	Unspecified Unspecified

Priority	medium Severity medium
Target Milestone:	Upstream M2
Target Release:	12.0 (Pike)
Assigned To:	Pradeep Kilambi
QA Contact:	Sasha Smolyak
Docs Contact:

URL:
Whiteboard:
Keywords:	Triaged

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2017-06-08 13:52 EDT by Alexander Chuzhoy
Modified:	2018-02-05 14:07 EST (History)
CC List:	20 users (show)

See Also:
Fixed In Version:	openstack-tripleo-heat-templates-7.0.0-0.20170624014919.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-12-13 16:31:55 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
env files used in deployment (4.20 KB, application/x-gzip) 2017-06-12 14:03 EDT, Alexander Chuzhoy	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	469935	None	None	None	2017-06-28 08:25 EDT
Red Hat Product Errata	RHEA-2017:3462	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-15 20:43:25 EST

Groups: None (edit)

Description Alexander Chuzhoy 2017-06-08 13:52:15 EDT

openstack-nova: repeating errors on compute: Jun  8 05:33:38 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory

Environment:
libvirt-daemon-driver-nwfilter-3.2.0-6.el7.x86_64
libvirt-client-3.2.0-6.el7.x86_64
openstack-nova-novncproxy-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-logical-3.2.0-6.el7.x86_64
libvirt-libs-3.2.0-6.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-6.el7.x86_64
python-nova-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-mpath-3.2.0-6.el7.x86_64
puppet-nova-11.1.0-0.20170520132314.882caca.el7ost.noarch
libvirt-daemon-3.2.0-6.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-6.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-6.el7.x86_64
libvirt-daemon-kvm-3.2.0-6.el7.x86_64
openstack-nova-migration-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-api-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-6.el7.x86_64
python-novaclient-8.0.0-0.20170517113627.e1b9e76.el7ost.noarch
libvirt-daemon-driver-network-3.2.0-6.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-6.el7.x86_64
libvirt-daemon-config-network-3.2.0-6.el7.x86_64
openstack-nova-compute-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-console-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-scsi-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-6.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-6.el7.x86_64
openstack-nova-common-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-gluster-3.2.0-6.el7.x86_64
libvirt-python-3.2.0-2.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-6.el7.x86_64
instack-undercloud-7.0.0-0.20170521005009.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170521033302.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch


Steps to reproduce:
deploy overcloud with containers:
openstack overcloud deploy --templates \
    --libvirt-type kvm \
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/docker-osp12.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml



Check the /var/log/messages on compute.

Result:
repeating error:
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory

Comment 2 Artom Lifshitz 2017-06-08 21:34:19 EDT

I just want to clarify, does the overcloud deploy fail? Or does everything work as expected, except for the presence of those errors in the logs?

Comment 3 Alexander Chuzhoy 2017-06-09 09:15:33 EDT

The overcloud deploy succeeded, but not everything works, here's a list of issues reported upon successful deployment:

https://bugzilla.redhat.com/show_bug.cgi?id=1457358 
https://bugzilla.redhat.com/show_bug.cgi?id=1457421
https://bugzilla.redhat.com/show_bug.cgi?id=1459592

Comment 4 Sven Anderson 2017-06-09 10:20:53 EDT

What are the used environment files in /home/stack/virt/ ?

Comment 5 Alexander Chuzhoy 2017-06-12 14:03 EDT

Created attachment 1287092 [details]
env files used in deployment

Comment 6 Sven Anderson 2017-06-20 13:18:41 EDT

seems to be a ceilometer-agent problem. should be fixed upstream by https://review.openstack.org/#/c/469935/

moving to openstack-ceilometer

Comment 9 Alexander Chuzhoy 2017-07-06 12:15:41 EDT

Failed_QA
Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170628002128.el7ost.noarch

The reported issue still reproduces.

Comment 10 Pradeep Kilambi 2017-07-06 12:50:35 EDT

Is this the same ceilometer-compute-agent container issue thats failing?

Comment 11 Alexander Chuzhoy 2017-07-06 13:20:45 EDT

Yes,
logs on ceilometer_agent_compute:
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused

Comment 12 Pradeep Kilambi 2017-07-06 16:24:43 EDT

I just did a fresh deploy of tripleo with containers and on compute nodes i see compute agent running without errors:

2017-07-06 19:56:18Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE 

Overcloud Endpoint: http://192.168.24.7:5000/v2.0
Overcloud Deployed

(undercloud) [stack@undercloud ~]$ rpm -qa|grep heat-templates
openstack-tripleo-heat-templates-7.0.0-0.20170706121721.25caba5.el7.centos.noarch


[root@overcloud-novacompute-0 ~]# docker ps -a |grep ceilometer
b91b783b48b0        192.168.24.1:8787/tripleoupstream/centos-binary-ceilometer-compute:latest          "kolla_start"       29 minutes ago      Up 29 minutes                                       ceilometer_agent_compute


[root@overcloud-novacompute-0 ~]# docker logs ceilometer_agent_compute
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
Running command: '/usr/bin/ceilometer-polling --polling-namespaces compute'

so something is definitely missing downstream.

Comment 18 errata-xmlrpc 2017-12-13 16:31:55 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.

alifshit
berrange
dansmith
eglynn
jdanjou
jruzicka
kchamart
mburns
pkilambi
rbryant
rhel-osp-director-maint
sasha
sbauza
scorcora
sferdjao
sgordon
srevivo
svanders
tvignaud
vromanso