Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1464212 - libvirt: Unable to start nova_libvirt container: Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system
libvirt: Unable to start nova_libvirt container: Initialization of secret st...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
medium Severity medium
: beta
: 12.0 (Pike)
Assigned To: Martin André
Alexander Chuzhoy
: AutomationBlocker, Triaged
Depends On:
Blocks:
  Show dependency tree / graph
 
Reported: 2017-06-22 12:27 EDT by Alexander Chuzhoy
Modified: 2018-02-05 14:07 EST (History)
28 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:33:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 476148 None None None 2017-06-27 12:26 EDT
OpenStack gerrit 476153 None None None 2017-06-27 12:26 EDT
OpenStack gerrit 477535 None None None 2017-06-27 12:27 EDT
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-15 20:43:25 EST

  None (edit)
Description Alexander Chuzhoy 2017-06-22 12:27:15 EDT 
libvirt: Unable to start nova_libvirt container:  Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system

Environment:
libvirt-daemon-3.5.0-1.el7.x86_64
libvirt-daemon-driver-qemu-3.5.0-1.el7.x86_64
libvirt-daemon-config-nwfilter-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.5.0-1.el7.x86_64
libvirt-3.5.0-1.el7.x86_64
libvirt-libs-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-core-3.5.0-1.el7.x86_64
libvirt-daemon-driver-nwfilter-3.5.0-1.el7.x86_64
libvirt-daemon-driver-interface-3.5.0-1.el7.x86_64
libvirt-daemon-driver-nodedev-3.5.0-1.el7.x86_64
libvirt-daemon-driver-lxc-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-logical-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-disk-3.5.0-1.el7.x86_64
libvirt-client-3.5.0-1.el7.x86_64
libvirt-daemon-kvm-3.5.0-1.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-network-3.5.0-1.el7.x86_64
libvirt-daemon-driver-secret-3.5.0-1.el7.x86_64
libvirt-daemon-config-network-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-3.5.0-1.el7.x86_64



Steps to reproduce:
[root@compute-0 ~]# /bin/docker run --name nova_libvirt -d --pid=host --env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/nova_libvirt/etc/libvirt/:/etc/libvirt/:ro --volume=/lib/modules:/lib/modules:ro --volume=/dev:/dev --volume=/run:/run --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/var/lib/nova:/var/lib/nova --volume=/var/run/libvirt:/var/run/libvirt --volume=/var/lib/libvirt:/var/lib/libvirt --volume=/etc/libvirt/qemu:/etc/libvirt/qemu --volume=/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro --volume=/var/log/containers/nova:/var/log/nova --net=host --privileged=true 192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-06-21.5


Result:
533984034c9dc37c323a85d6431adf640081972c678d05281d704e4d6dc1a541

[root@compute-0 ~]# docker ps -a|grep libvirt
533984034c9d        192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-06-21.5                "kolla_start"       34 seconds ago      Exited (0) 33 seconds ago                           nova_libvirt
[root@compute-0 ~]# 


[root@compute-0 ~]# docker logs nova_libvirt
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/nova
INFO:__main__:Setting permission for /var/log/nova/nova-compute.log
Running command: '/usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf'
2017-06-22 16:24:31.636+0000: 8466: info : libvirt version: 3.5.0, package: 1.el7 (Unknown, 2017-06-20-18:23:45, rhel7-next)
2017-06-22 16:24:31.636+0000: 8466: info : hostname: compute-0.localdomain
2017-06-22 16:24:31.636+0000: 8466: error : secretStateInitialize:469 : cannot create config directory '/etc/libvirt/secrets': Read-only file system
2017-06-22 16:24:31.636+0000: 8466: error : virStateInitialize:775 : Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system
2017-06-22 16:24:31.636+0000: 8466: error : daemonRunStateInit:882 : Driver state initialization failed



Workaround is to run:
mkdir /var/lib/config-data/nova_libvirt/etc/libvirt/secrets

before starting the nova_libvirt container.
Comment 2 Daniel Berrange 2017-06-22 12:35:05 EDT 
This is not a libvirt bug. Nova's container deployment is trying to run Libvirt on a read-only root filesystem, which is simply broken.
Comment 3 Michal Privoznik 2017-06-23 03:09:58 EDT 
(In reply to Daniel Berrange from comment #2)
> This is not a libvirt bug. Nova's container deployment is trying to run
> Libvirt on a read-only root filesystem, which is simply broken.

Agreed. Libvirt needs /etc/libvirt writable so that it can store its XMLs and other configs there. Switching component to nova.
Comment 4 Omri Hochman 2017-06-23 10:56:33 EDT 
> Workaround is to run:
> mkdir /var/lib/config-data/nova_libvirt/etc/libvirt/secrets
> 
> before starting the nova_libvirt container.

Taking it back to container-DFG
Comment 5 Dan Prince 2017-06-26 16:46:26 EDT 
I think Martin is on track to fix this upstream with this patch:

https://review.openstack.org/#/c/476153/4/docker/services/nova-compute.yaml
Comment 6 Martin André 2017-06-27 12:24:36 EDT 
This will be fixed by a series of patches to enable config files copy in the container instead of direct bind mount. These are on the right track to land, but I expect it takes a bit more time due to upstream inertia. 

Let me know if this is a blocker and I'll work on providing a workaround.
Comment 7 Martin André 2017-06-27 12:41:06 EDT 
I think a simple workaround is to add a named volume for '/etc/libvirt/secrets'. The performance of named volumes isn't great, but at least we won't be hitting permissions issues.

In tripleo-heat-templates/docker/services/nova-libvirt.yaml, for nova_libvirt container's volumes [1] you'll need to add a line:

- libvirt_secrets:/etc/libvirt/secrets

[1] https://github.com/openstack/tripleo-heat-templates/blob/af3828437e4ec92a738023afd29f12225161866e/docker/services/nova-libvirt.yaml#L128
Comment 8 Daniel Berrange 2017-06-28 04:59:03 EDT 
Libvirt expects the entire of /etc/libvirt  to be writable, not just that one sub directory.
Comment 9 Dan Prince 2017-06-28 09:18:54 EDT 
Martin's fixes in https://review.openstack.org/#/c/476153/ upstream will eventually make the entire /etc/libvirt tree writable to Libvirt. So I think we are on track to fix this correctly for the mid/long term.

----

Until all that lands however I think it would be reasonable to work around this with the named volume work around added to the volumes section of the docker/services/nova-libvirt.yaml template as Martin also sugests:

- libvirt_secrets:/etc/libvirt/secrets

Martin: we may also need permissions adjusted on this directory? Could use kolla_config's permissions to do this if so. If it works as is though I think it is probably fine to leave it with the defaults.
Comment 10 Martin André 2017-07-17 11:11:18 EDT 
All the fixes merged upstream and should already be available in the latest puddle. Sasha, can you check again?
Comment 11 Alexander Chuzhoy 2017-07-17 18:25:09 EDT 
The issues doesn't reproduce.
Environment:
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch
instack-undercloud-7.1.1-0.20170710151630.el7ost.noarch



Will the 'fixed in version' field be updated? What RPM includes the fix?
Comment 12 Thierry Vignaud 2017-07-18 08:38:14 EDT 
We got the kolla & THT changes but not yet the puppet-tripleo change (it'll be part of next import)
Comment 14 Omri Hochman 2017-08-04 10:59:04 EDT 
it suppose to be ON_QA?
Comment 15 Alexander Chuzhoy 2017-08-04 12:11:51 EDT 
Verified:

Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch


The reported issue doesn't reproduce.
Comment 20 errata-xmlrpc 2017-12-13 16:33:29 EST 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.