Bug 1469825 - rhosp-director: nova_libvirt container doesn't start on compute: cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system
rhosp-director: nova_libvirt container doesn't start on compute: cannot creat...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity high
: beta
: 12.0 (Pike)
Assigned To: Emilien Macchi
Gurenko Alex
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-11 18:16 EDT by Alexander Chuzhoy
Modified: 2018-02-05 14:10 EST (History)
10 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:40:04 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 476153 None None None 2017-07-17 09:06 EDT

  None (edit)
Description Alexander Chuzhoy 2017-07-11 18:16:38 EDT
rhosp-director: nova_libvirt container doesn't start on compute: cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system



Environment:
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
openstack-nova-placement-api-16.0.0-0.20170627215041.el7ost.noarch
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
openstack-nova-compute-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-console-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-novncproxy-16.0.0-0.20170627215041.el7ost.noarch
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
puppet-nova-11.2.0-0.20170627013550.389d0fb.el7ost.noarch
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
openstack-nova-common-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-migration-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-api-16.0.0-0.20170627215041.el7ost.noarch
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
python-nova-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170627215041.el7ost.noarch
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
openstack-nova-scheduler-16.0.0-0.20170627215041.el7ost.noarch
python-novaclient-9.0.1-0.20170621200422.ddb386b.el7ost.noarch
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64


openstack-tripleo-heat-templates-7.0.0-0.20170628002128.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
instack-undercloud-7.1.1-0.20170623182135.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud and check the status of nova_libvirt container on compute:

Result:
[root@overcloud-novacompute-0 ~]# docker ps -a|grep libvirt
a75c4ca5c2d4        192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-07-02.2                "kolla_start"       33 minutes ago      Exited (139) 33 minutes ago                       nova_libvirt

[root@overcloud-novacompute-0 ~]# docker logs a75c4ca5c2d4
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/nova
INFO:__main__:Setting permission for /var/log/nova/nova-compute.log
Running command: '/usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf'
2017-07-11 21:42:01.693+0000: 14788: info : libvirt version: 3.5.0, package: 1.el7 (Unknown, 2017-06-20-18:23:45, rhel7-next)
2017-07-11 21:42:01.693+0000: 14788: info : hostname: overcloud-novacompute-0.redhat.local
2017-07-11 21:42:01.693+0000: 14788: error : virDriverLoadModuleFile:59 : failed to load module /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so: undefined symbol: rbd_diff_iterate2
2017-07-11 21:42:02.713+0000: 14840: error : virFileRewrite:465 : cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system
2017-07-11 21:42:02.713+0000: 14840: error : virFileRewrite:465 : cannot create file '/etc/libvirt/nwfilter/allow-dhcp-server.xml.new': Read-only file system




The filesystem is mounted r/o.
Comment 1 Alexander Chuzhoy 2017-07-11 18:40:19 EDT
The successfully tested w/a is to mount /etc/libvirt on nova_libvirt without the r/o flag.
Comment 2 Martin André 2017-07-17 09:06:31 EDT
I believe this should be fixed with the latest puddle were we do not bind mount /etc/libvirt anymore. Can you recheck?

This was fixed by commit https://review.openstack.org/#/c/476153/ upstream.
Comment 3 Alexander Chuzhoy 2017-07-17 18:53:42 EDT
The reported issue doesn't reproduce.
Environment:
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
python-docker-py-1.10.6-1.el7.noarch
python-docker-pycreds-1.10.6-1.el7.noarch
docker-common-1.12.6-40.1.gitf55a118.el7.x86_64
docker-rhel-push-plugin-1.12.6-40.1.gitf55a118.el7.x86_64
python-heat-agent-docker-cmd-1.2.1-0.20170628215913.cb2690a.el7ost.noarch
docker-1.12.6-40.1.gitf55a118.el7.x86_64
docker-client-1.12.6-40.1.gitf55a118.el7.x86_64
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch
instack-undercloud-7.1.1-0.20170710151630.el7ost.noarch


No "Fixed in Version". What RPM fixes the issue?
Comment 4 Omri Hochman 2017-07-19 09:26:03 EDT
can you please add the info for which RPM is fixing this issue ?
Comment 5 Alexander Chuzhoy 2017-08-04 10:50:17 EDT
Environment:
instack-undercloud-7.1.1-0.20170714211622.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch


The reported issue doesn't reproduce.
Comment 7 Thierry Vignaud 2017-08-22 09:22:10 EDT
Let's use the version reported in comment #5 ...
Comment 9 Martin André 2017-09-25 10:47:35 EDT
Fixed in openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost
Comment 12 errata-xmlrpc 2017-12-13 16:40:04 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.