Bug 1469825 - rhosp-director: nova_libvirt container doesn't start on compute: cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system
Summary: rhosp-director: nova_libvirt container doesn't start on compute: cannot creat...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: 12.0 (Pike)
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-11 22:16 UTC by Alexander Chuzhoy
Modified: 2018-02-05 19:10 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 21:40:04 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 476153 0 None None None 2017-07-17 13:06:31 UTC
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Alexander Chuzhoy 2017-07-11 22:16:38 UTC
rhosp-director: nova_libvirt container doesn't start on compute: cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system



Environment:
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
openstack-nova-placement-api-16.0.0-0.20170627215041.el7ost.noarch
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
openstack-nova-compute-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-console-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-novncproxy-16.0.0-0.20170627215041.el7ost.noarch
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
puppet-nova-11.2.0-0.20170627013550.389d0fb.el7ost.noarch
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
openstack-nova-common-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-migration-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-api-16.0.0-0.20170627215041.el7ost.noarch
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
python-nova-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170627215041.el7ost.noarch
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
openstack-nova-scheduler-16.0.0-0.20170627215041.el7ost.noarch
python-novaclient-9.0.1-0.20170621200422.ddb386b.el7ost.noarch
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64


openstack-tripleo-heat-templates-7.0.0-0.20170628002128.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
instack-undercloud-7.1.1-0.20170623182135.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud and check the status of nova_libvirt container on compute:

Result:
[root@overcloud-novacompute-0 ~]# docker ps -a|grep libvirt
a75c4ca5c2d4        192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-07-02.2                "kolla_start"       33 minutes ago      Exited (139) 33 minutes ago                       nova_libvirt

[root@overcloud-novacompute-0 ~]# docker logs a75c4ca5c2d4
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/nova
INFO:__main__:Setting permission for /var/log/nova/nova-compute.log
Running command: '/usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf'
2017-07-11 21:42:01.693+0000: 14788: info : libvirt version: 3.5.0, package: 1.el7 (Unknown, 2017-06-20-18:23:45, rhel7-next)
2017-07-11 21:42:01.693+0000: 14788: info : hostname: overcloud-novacompute-0.redhat.local
2017-07-11 21:42:01.693+0000: 14788: error : virDriverLoadModuleFile:59 : failed to load module /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so: undefined symbol: rbd_diff_iterate2
2017-07-11 21:42:02.713+0000: 14840: error : virFileRewrite:465 : cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system
2017-07-11 21:42:02.713+0000: 14840: error : virFileRewrite:465 : cannot create file '/etc/libvirt/nwfilter/allow-dhcp-server.xml.new': Read-only file system




The filesystem is mounted r/o.

Comment 1 Alexander Chuzhoy 2017-07-11 22:40:19 UTC
The successfully tested w/a is to mount /etc/libvirt on nova_libvirt without the r/o flag.

Comment 2 Martin André 2017-07-17 13:06:31 UTC
I believe this should be fixed with the latest puddle were we do not bind mount /etc/libvirt anymore. Can you recheck?

This was fixed by commit https://review.openstack.org/#/c/476153/ upstream.

Comment 3 Alexander Chuzhoy 2017-07-17 22:53:42 UTC
The reported issue doesn't reproduce.
Environment:
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
python-docker-py-1.10.6-1.el7.noarch
python-docker-pycreds-1.10.6-1.el7.noarch
docker-common-1.12.6-40.1.gitf55a118.el7.x86_64
docker-rhel-push-plugin-1.12.6-40.1.gitf55a118.el7.x86_64
python-heat-agent-docker-cmd-1.2.1-0.20170628215913.cb2690a.el7ost.noarch
docker-1.12.6-40.1.gitf55a118.el7.x86_64
docker-client-1.12.6-40.1.gitf55a118.el7.x86_64
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch
instack-undercloud-7.1.1-0.20170710151630.el7ost.noarch


No "Fixed in Version". What RPM fixes the issue?

Comment 4 Omri Hochman 2017-07-19 13:26:03 UTC
can you please add the info for which RPM is fixing this issue ?

Comment 5 Alexander Chuzhoy 2017-08-04 14:50:17 UTC
Environment:
instack-undercloud-7.1.1-0.20170714211622.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch


The reported issue doesn't reproduce.

Comment 7 Thierry Vignaud 2017-08-22 13:22:10 UTC
Let's use the version reported in comment #5 ...

Comment 9 Martin André 2017-09-25 14:47:35 UTC
Fixed in openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost

Comment 12 errata-xmlrpc 2017-12-13 21:40:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.