Bug 1464212 - libvirt: Unable to start nova_libvirt container: Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system
libvirt: Unable to start nova_libvirt container: Initialization of secret st...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
medium Severity medium
: beta
: 12.0 (Pike)
Assigned To: Martin André
Alexander Chuzhoy
: AutomationBlocker, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-22 12:27 EDT by Alexander Chuzhoy
Modified: 2018-02-05 14:07 EST (History)
28 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:33:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 476148 None None None 2017-06-27 12:26 EDT
OpenStack gerrit 476153 None None None 2017-06-27 12:26 EDT
OpenStack gerrit 477535 None None None 2017-06-27 12:27 EDT
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-15 20:43:25 EST

  None (edit)
Description Alexander Chuzhoy 2017-06-22 12:27:15 EDT
libvirt: Unable to start nova_libvirt container:  Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system

Environment:
libvirt-daemon-3.5.0-1.el7.x86_64
libvirt-daemon-driver-qemu-3.5.0-1.el7.x86_64
libvirt-daemon-config-nwfilter-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.5.0-1.el7.x86_64
libvirt-3.5.0-1.el7.x86_64
libvirt-libs-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-core-3.5.0-1.el7.x86_64
libvirt-daemon-driver-nwfilter-3.5.0-1.el7.x86_64
libvirt-daemon-driver-interface-3.5.0-1.el7.x86_64
libvirt-daemon-driver-nodedev-3.5.0-1.el7.x86_64
libvirt-daemon-driver-lxc-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-logical-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-disk-3.5.0-1.el7.x86_64
libvirt-client-3.5.0-1.el7.x86_64
libvirt-daemon-kvm-3.5.0-1.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-network-3.5.0-1.el7.x86_64
libvirt-daemon-driver-secret-3.5.0-1.el7.x86_64
libvirt-daemon-config-network-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-3.5.0-1.el7.x86_64



Steps to reproduce:
[root@compute-0 ~]# /bin/docker run --name nova_libvirt -d --pid=host --env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/nova_libvirt/etc/libvirt/:/etc/libvirt/:ro --volume=/lib/modules:/lib/modules:ro --volume=/dev:/dev --volume=/run:/run --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/var/lib/nova:/var/lib/nova --volume=/var/run/libvirt:/var/run/libvirt --volume=/var/lib/libvirt:/var/lib/libvirt --volume=/etc/libvirt/qemu:/etc/libvirt/qemu --volume=/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro --volume=/var/log/containers/nova:/var/log/nova --net=host --privileged=true 192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-06-21.5


Result:
533984034c9dc37c323a85d6431adf640081972c678d05281d704e4d6dc1a541

[root@compute-0 ~]# docker ps -a|grep libvirt
533984034c9d        192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-06-21.5                "kolla_start"       34 seconds ago      Exited (0) 33 seconds ago                           nova_libvirt
[root@compute-0 ~]# 


[root@compute-0 ~]# docker logs nova_libvirt
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/nova
INFO:__main__:Setting permission for /var/log/nova/nova-compute.log
Running command: '/usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf'
2017-06-22 16:24:31.636+0000: 8466: info : libvirt version: 3.5.0, package: 1.el7 (Unknown, 2017-06-20-18:23:45, rhel7-next)
2017-06-22 16:24:31.636+0000: 8466: info : hostname: compute-0.localdomain
2017-06-22 16:24:31.636+0000: 8466: error : secretStateInitialize:469 : cannot create config directory '/etc/libvirt/secrets': Read-only file system
2017-06-22 16:24:31.636+0000: 8466: error : virStateInitialize:775 : Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system
2017-06-22 16:24:31.636+0000: 8466: error : daemonRunStateInit:882 : Driver state initialization failed



Workaround is to run:
mkdir /var/lib/config-data/nova_libvirt/etc/libvirt/secrets

before starting the nova_libvirt container.
Comment 2 Daniel Berrange 2017-06-22 12:35:05 EDT
This is not a libvirt bug. Nova's container deployment is trying to run Libvirt on a read-only root filesystem, which is simply broken.
Comment 3 Michal Privoznik 2017-06-23 03:09:58 EDT
(In reply to Daniel Berrange from comment #2)
> This is not a libvirt bug. Nova's container deployment is trying to run
> Libvirt on a read-only root filesystem, which is simply broken.

Agreed. Libvirt needs /etc/libvirt writable so that it can store its XMLs and other configs there. Switching component to nova.
Comment 4 Omri Hochman 2017-06-23 10:56:33 EDT
> Workaround is to run:
> mkdir /var/lib/config-data/nova_libvirt/etc/libvirt/secrets
> 
> before starting the nova_libvirt container.

Taking it back to container-DFG
Comment 5 Dan Prince 2017-06-26 16:46:26 EDT
I think Martin is on track to fix this upstream with this patch:

https://review.openstack.org/#/c/476153/4/docker/services/nova-compute.yaml
Comment 6 Martin André 2017-06-27 12:24:36 EDT
This will be fixed by a series of patches to enable config files copy in the container instead of direct bind mount. These are on the right track to land, but I expect it takes a bit more time due to upstream inertia. 

Let me know if this is a blocker and I'll work on providing a workaround.
Comment 7 Martin André 2017-06-27 12:41:06 EDT
I think a simple workaround is to add a named volume for '/etc/libvirt/secrets'. The performance of named volumes isn't great, but at least we won't be hitting permissions issues.

In tripleo-heat-templates/docker/services/nova-libvirt.yaml, for nova_libvirt container's volumes [1] you'll need to add a line:

- libvirt_secrets:/etc/libvirt/secrets

[1] https://github.com/openstack/tripleo-heat-templates/blob/af3828437e4ec92a738023afd29f12225161866e/docker/services/nova-libvirt.yaml#L128
Comment 8 Daniel Berrange 2017-06-28 04:59:03 EDT
Libvirt expects the entire of /etc/libvirt  to be writable, not just that one sub directory.
Comment 9 Dan Prince 2017-06-28 09:18:54 EDT
Martin's fixes in https://review.openstack.org/#/c/476153/ upstream will eventually make the entire /etc/libvirt tree writable to Libvirt. So I think we are on track to fix this correctly for the mid/long term.

----

Until all that lands however I think it would be reasonable to work around this with the named volume work around added to the volumes section of the docker/services/nova-libvirt.yaml template as Martin also sugests:

- libvirt_secrets:/etc/libvirt/secrets

Martin: we may also need permissions adjusted on this directory? Could use kolla_config's permissions to do this if so. If it works as is though I think it is probably fine to leave it with the defaults.
Comment 10 Martin André 2017-07-17 11:11:18 EDT
All the fixes merged upstream and should already be available in the latest puddle. Sasha, can you check again?
Comment 11 Alexander Chuzhoy 2017-07-17 18:25:09 EDT
The issues doesn't reproduce.
Environment:
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch
instack-undercloud-7.1.1-0.20170710151630.el7ost.noarch



Will the 'fixed in version' field be updated? What RPM includes the fix?
Comment 12 Thierry Vignaud 2017-07-18 08:38:14 EDT
We got the kolla & THT changes but not yet the puppet-tripleo change (it'll be part of next import)
Comment 14 Omri Hochman 2017-08-04 10:59:04 EDT
it suppose to be ON_QA?
Comment 15 Alexander Chuzhoy 2017-08-04 12:11:51 EDT
Verified:

Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch


The reported issue doesn't reproduce.
Comment 20 errata-xmlrpc 2017-12-13 16:33:29 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.