Hide Forgot
libvirt: Unable to start nova_libvirt container: Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system Environment: libvirt-daemon-3.5.0-1.el7.x86_64 libvirt-daemon-driver-qemu-3.5.0-1.el7.x86_64 libvirt-daemon-config-nwfilter-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-iscsi-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-gluster-3.5.0-1.el7.x86_64 libvirt-3.5.0-1.el7.x86_64 libvirt-libs-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-core-3.5.0-1.el7.x86_64 libvirt-daemon-driver-nwfilter-3.5.0-1.el7.x86_64 libvirt-daemon-driver-interface-3.5.0-1.el7.x86_64 libvirt-daemon-driver-nodedev-3.5.0-1.el7.x86_64 libvirt-daemon-driver-lxc-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-logical-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-mpath-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-rbd-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-disk-3.5.0-1.el7.x86_64 libvirt-client-3.5.0-1.el7.x86_64 libvirt-daemon-kvm-3.5.0-1.el7.x86_64 libvirt-python-3.2.0-3.el7.x86_64 libvirt-daemon-driver-network-3.5.0-1.el7.x86_64 libvirt-daemon-driver-secret-3.5.0-1.el7.x86_64 libvirt-daemon-config-network-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-scsi-3.5.0-1.el7.x86_64 libvirt-daemon-driver-storage-3.5.0-1.el7.x86_64 Steps to reproduce: [root@compute-0 ~]# /bin/docker run --name nova_libvirt -d --pid=host --env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/nova_libvirt/etc/libvirt/:/etc/libvirt/:ro --volume=/lib/modules:/lib/modules:ro --volume=/dev:/dev --volume=/run:/run --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/var/lib/nova:/var/lib/nova --volume=/var/run/libvirt:/var/run/libvirt --volume=/var/lib/libvirt:/var/lib/libvirt --volume=/etc/libvirt/qemu:/etc/libvirt/qemu --volume=/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro --volume=/var/log/containers/nova:/var/log/nova --net=host --privileged=true 192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-06-21.5 Result: 533984034c9dc37c323a85d6431adf640081972c678d05281d704e4d6dc1a541 [root@compute-0 ~]# docker ps -a|grep libvirt 533984034c9d 192.168.24.1:8787/rhosp12/openstack-nova-libvirt-docker:2017-06-21.5 "kolla_start" 34 seconds ago Exited (0) 33 seconds ago nova_libvirt [root@compute-0 ~]# [root@compute-0 ~]# docker logs nova_libvirt INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/log/nova INFO:__main__:Setting permission for /var/log/nova/nova-compute.log Running command: '/usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf' 2017-06-22 16:24:31.636+0000: 8466: info : libvirt version: 3.5.0, package: 1.el7 (Unknown, 2017-06-20-18:23:45, rhel7-next) 2017-06-22 16:24:31.636+0000: 8466: info : hostname: compute-0.localdomain 2017-06-22 16:24:31.636+0000: 8466: error : secretStateInitialize:469 : cannot create config directory '/etc/libvirt/secrets': Read-only file system 2017-06-22 16:24:31.636+0000: 8466: error : virStateInitialize:775 : Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system 2017-06-22 16:24:31.636+0000: 8466: error : daemonRunStateInit:882 : Driver state initialization failed Workaround is to run: mkdir /var/lib/config-data/nova_libvirt/etc/libvirt/secrets before starting the nova_libvirt container.
This is not a libvirt bug. Nova's container deployment is trying to run Libvirt on a read-only root filesystem, which is simply broken.
(In reply to Daniel Berrange from comment #2) > This is not a libvirt bug. Nova's container deployment is trying to run > Libvirt on a read-only root filesystem, which is simply broken. Agreed. Libvirt needs /etc/libvirt writable so that it can store its XMLs and other configs there. Switching component to nova.
> Workaround is to run: > mkdir /var/lib/config-data/nova_libvirt/etc/libvirt/secrets > > before starting the nova_libvirt container. Taking it back to container-DFG
I think Martin is on track to fix this upstream with this patch: https://review.openstack.org/#/c/476153/4/docker/services/nova-compute.yaml
This will be fixed by a series of patches to enable config files copy in the container instead of direct bind mount. These are on the right track to land, but I expect it takes a bit more time due to upstream inertia. Let me know if this is a blocker and I'll work on providing a workaround.
I think a simple workaround is to add a named volume for '/etc/libvirt/secrets'. The performance of named volumes isn't great, but at least we won't be hitting permissions issues. In tripleo-heat-templates/docker/services/nova-libvirt.yaml, for nova_libvirt container's volumes [1] you'll need to add a line: - libvirt_secrets:/etc/libvirt/secrets [1] https://github.com/openstack/tripleo-heat-templates/blob/af3828437e4ec92a738023afd29f12225161866e/docker/services/nova-libvirt.yaml#L128
Libvirt expects the entire of /etc/libvirt to be writable, not just that one sub directory.
Martin's fixes in https://review.openstack.org/#/c/476153/ upstream will eventually make the entire /etc/libvirt tree writable to Libvirt. So I think we are on track to fix this correctly for the mid/long term. ---- Until all that lands however I think it would be reasonable to work around this with the named volume work around added to the volumes section of the docker/services/nova-libvirt.yaml template as Martin also sugests: - libvirt_secrets:/etc/libvirt/secrets Martin: we may also need permissions adjusted on this directory? Could use kolla_config's permissions to do this if so. If it works as is though I think it is probably fine to leave it with the defaults.
All the fixes merged upstream and should already be available in the latest puddle. Sasha, can you check again?
The issues doesn't reproduce. Environment: libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64 libvirt-daemon-driver-network-3.2.0-14.el7.x86_64 libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64 libvirt-libs-3.2.0-14.el7.x86_64 libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64 libvirt-daemon-config-network-3.2.0-14.el7.x86_64 libvirt-client-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64 libvirt-daemon-kvm-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64 libvirt-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64 libvirt-python-3.2.0-3.el7.x86_64 libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64 libvirt-daemon-3.2.0-14.el7.x86_64 libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64 openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch instack-undercloud-7.1.1-0.20170710151630.el7ost.noarch Will the 'fixed in version' field be updated? What RPM includes the fix?
We got the kolla & THT changes but not yet the puppet-tripleo change (it'll be part of next import)
it suppose to be ON_QA?
Verified: Environment: openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch The reported issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462