Bug 1469825

Summary: rhosp-director: nova_libvirt container doesn't start on compute: cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: dbecker, eglynn, m.andre, mburns, morazi, ohochman, rhel-osp-director-maint, sasha, scorcora, tvignaud
Target Milestone: betaKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:40:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2017-07-11 22:16:38 UTC 
rhosp-director: nova_libvirt container doesn't start on compute: cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system



Environment:
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
openstack-nova-placement-api-16.0.0-0.20170627215041.el7ost.noarch
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
openstack-nova-compute-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-console-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-novncproxy-16.0.0-0.20170627215041.el7ost.noarch
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
puppet-nova-11.2.0-0.20170627013550.389d0fb.el7ost.noarch
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
openstack-nova-common-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-migration-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-api-16.0.0-0.20170627215041.el7ost.noarch
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
python-nova-16.0.0-0.20170627215041.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170627215041.el7ost.noarch
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
openstack-nova-scheduler-16.0.0-0.20170627215041.el7ost.noarch
python-novaclient-9.0.1-0.20170621200422.ddb386b.el7ost.noarch
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64


openstack-tripleo-heat-templates-7.0.0-0.20170628002128.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
instack-undercloud-7.1.1-0.20170623182135.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud and check the status of nova_libvirt container on compute:

Result:
[root@overcloud-novacompute-0 ~]# docker ps -a|grep libvirt
a75c4ca5c2d4        192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-07-02.2                "kolla_start"       33 minutes ago      Exited (139) 33 minutes ago                       nova_libvirt

[root@overcloud-novacompute-0 ~]# docker logs a75c4ca5c2d4
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/nova
INFO:__main__:Setting permission for /var/log/nova/nova-compute.log
Running command: '/usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf'
2017-07-11 21:42:01.693+0000: 14788: info : libvirt version: 3.5.0, package: 1.el7 (Unknown, 2017-06-20-18:23:45, rhel7-next)
2017-07-11 21:42:01.693+0000: 14788: info : hostname: overcloud-novacompute-0.redhat.local
2017-07-11 21:42:01.693+0000: 14788: error : virDriverLoadModuleFile:59 : failed to load module /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so: undefined symbol: rbd_diff_iterate2
2017-07-11 21:42:02.713+0000: 14840: error : virFileRewrite:465 : cannot create file '/etc/libvirt/nwfilter/allow-arp.xml.new': Read-only file system
2017-07-11 21:42:02.713+0000: 14840: error : virFileRewrite:465 : cannot create file '/etc/libvirt/nwfilter/allow-dhcp-server.xml.new': Read-only file system




The filesystem is mounted r/o.
Comment 1 Alexander Chuzhoy 2017-07-11 22:40:19 UTC 
The successfully tested w/a is to mount /etc/libvirt on nova_libvirt without the r/o flag.
Comment 2 Martin André 2017-07-17 13:06:31 UTC 
I believe this should be fixed with the latest puddle were we do not bind mount /etc/libvirt anymore. Can you recheck?

This was fixed by commit https://review.openstack.org/#/c/476153/ upstream.
Comment 3 Alexander Chuzhoy 2017-07-17 22:53:42 UTC 
The reported issue doesn't reproduce.
Environment:
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
python-docker-py-1.10.6-1.el7.noarch
python-docker-pycreds-1.10.6-1.el7.noarch
docker-common-1.12.6-40.1.gitf55a118.el7.x86_64
docker-rhel-push-plugin-1.12.6-40.1.gitf55a118.el7.x86_64
python-heat-agent-docker-cmd-1.2.1-0.20170628215913.cb2690a.el7ost.noarch
docker-1.12.6-40.1.gitf55a118.el7.x86_64
docker-client-1.12.6-40.1.gitf55a118.el7.x86_64
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch
instack-undercloud-7.1.1-0.20170710151630.el7ost.noarch


No "Fixed in Version". What RPM fixes the issue?
Comment 4 Omri Hochman 2017-07-19 13:26:03 UTC 
can you please add the info for which RPM is fixing this issue ?
Comment 5 Alexander Chuzhoy 2017-08-04 14:50:17 UTC 
Environment:
instack-undercloud-7.1.1-0.20170714211622.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch


The reported issue doesn't reproduce.
Comment 7 Thierry Vignaud 2017-08-22 13:22:10 UTC 
Let's use the version reported in comment #5 ...
Comment 9 Martin André 2017-09-25 14:47:35 UTC 
Fixed in openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost
Comment 12 errata-xmlrpc 2017-12-13 21:40:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462