Bug 1459976 - openstack-nova: repeating errors on compute: Jun 8 05:33:38 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
openstack-nova: repeating errors on compute: Jun 8 05:33:38 localhost journa...
Status: MODIFIED
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
unspecified Severity unspecified
: Upstream M2
: 12.0 (Pike)
Assigned To: Pradeep Kilambi
Sasha Smolyak
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-08 13:52 EDT by Alexander Chuzhoy
Modified: 2017-07-21 08:35 EDT (History)
19 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170624014919.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
env files used in deployment (4.20 KB, application/x-gzip)
2017-06-12 14:03 EDT, Alexander Chuzhoy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 469935 None None None 2017-06-28 08:25 EDT

  None (edit)
Description Alexander Chuzhoy 2017-06-08 13:52:15 EDT
openstack-nova: repeating errors on compute: Jun  8 05:33:38 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory

Environment:
libvirt-daemon-driver-nwfilter-3.2.0-6.el7.x86_64
libvirt-client-3.2.0-6.el7.x86_64
openstack-nova-novncproxy-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-logical-3.2.0-6.el7.x86_64
libvirt-libs-3.2.0-6.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-6.el7.x86_64
python-nova-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-mpath-3.2.0-6.el7.x86_64
puppet-nova-11.1.0-0.20170520132314.882caca.el7ost.noarch
libvirt-daemon-3.2.0-6.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-6.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-6.el7.x86_64
libvirt-daemon-kvm-3.2.0-6.el7.x86_64
openstack-nova-migration-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-api-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-6.el7.x86_64
python-novaclient-8.0.0-0.20170517113627.e1b9e76.el7ost.noarch
libvirt-daemon-driver-network-3.2.0-6.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-6.el7.x86_64
libvirt-daemon-config-network-3.2.0-6.el7.x86_64
openstack-nova-compute-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-console-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-scsi-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-6.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-6.el7.x86_64
openstack-nova-common-16.0.0-0.20170521033533.99bd334.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170521033533.99bd334.el7ost.noarch
libvirt-daemon-driver-storage-gluster-3.2.0-6.el7.x86_64
libvirt-python-3.2.0-2.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-6.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-6.el7.x86_64
instack-undercloud-7.0.0-0.20170521005009.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170521033302.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch


Steps to reproduce:
deploy overcloud with containers:
openstack overcloud deploy --templates \
    --libvirt-type kvm \
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/docker-osp12.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml



Check the /var/log/messages on compute.

Result:
repeating error:
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Jun  8 05:26:23 localhost journal: libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': No such file or directory
Comment 2 Artom Lifshitz 2017-06-08 21:34:19 EDT
I just want to clarify, does the overcloud deploy fail? Or does everything work as expected, except for the presence of those errors in the logs?
Comment 3 Alexander Chuzhoy 2017-06-09 09:15:33 EDT
The overcloud deploy succeeded, but not everything works, here's a list of issues reported upon successful deployment:

https://bugzilla.redhat.com/show_bug.cgi?id=1457358 
https://bugzilla.redhat.com/show_bug.cgi?id=1457421
https://bugzilla.redhat.com/show_bug.cgi?id=1459592
Comment 4 Sven Anderson 2017-06-09 10:20:53 EDT
What are the used environment files in /home/stack/virt/ ?
Comment 5 Alexander Chuzhoy 2017-06-12 14:03 EDT
Created attachment 1287092 [details]
env files used in deployment
Comment 6 Sven Anderson 2017-06-20 13:18:41 EDT
seems to be a ceilometer-agent problem. should be fixed upstream by https://review.openstack.org/#/c/469935/

moving to openstack-ceilometer
Comment 9 Alexander Chuzhoy 2017-07-06 12:15:41 EDT
Failed_QA
Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170628002128.el7ost.noarch

The reported issue still reproduces.
Comment 10 Pradeep Kilambi 2017-07-06 12:50:35 EDT
Is this the same ceilometer-compute-agent container issue thats failing?
Comment 11 Alexander Chuzhoy 2017-07-06 13:20:45 EDT
Yes,
logs on ceilometer_agent_compute:
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused
Comment 12 Pradeep Kilambi 2017-07-06 16:24:43 EDT
I just did a fresh deploy of tripleo with containers and on compute nodes i see compute agent running without errors:

2017-07-06 19:56:18Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE 

Overcloud Endpoint: http://192.168.24.7:5000/v2.0
Overcloud Deployed

(undercloud) [stack@undercloud ~]$ rpm -qa|grep heat-templates
openstack-tripleo-heat-templates-7.0.0-0.20170706121721.25caba5.el7.centos.noarch


[root@overcloud-novacompute-0 ~]# docker ps -a |grep ceilometer
b91b783b48b0        192.168.24.1:8787/tripleoupstream/centos-binary-ceilometer-compute:latest          "kolla_start"       29 minutes ago      Up 29 minutes                                       ceilometer_agent_compute


[root@overcloud-novacompute-0 ~]# docker logs ceilometer_agent_compute
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
Running command: '/usr/bin/ceilometer-polling --polling-namespaces compute'

so something is definitely missing downstream.

Note You need to log in before you can comment on or make changes to this bug.