Bug 1464212
| Summary: | libvirt: Unable to start nova_libvirt container: Initialization of secret state driver failed: cannot create config directory '/etc/libvirt/secrets': Read-only file system | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Martin André <maandre> |
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 12.0 (Pike) | CC: | afazekas, berrange, chhu, dasmith, dprince, dyuan, eglynn, jishao, jjoyce, jschluet, kchamart, maandre, m.andre, mburns, mprivozn, ohochman, rbalakri, rhel-osp-director-maint, sasha, sbauza, scorcora, sferdjao, sgordon, srevivo, tvignaud, vromanso, xuzhang |
| Target Milestone: | beta | Keywords: | AutomationBlocker, Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 21:33:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Alexander Chuzhoy
2017-06-22 16:27:15 UTC
This is not a libvirt bug. Nova's container deployment is trying to run Libvirt on a read-only root filesystem, which is simply broken. (In reply to Daniel Berrange from comment #2) > This is not a libvirt bug. Nova's container deployment is trying to run > Libvirt on a read-only root filesystem, which is simply broken. Agreed. Libvirt needs /etc/libvirt writable so that it can store its XMLs and other configs there. Switching component to nova.
> Workaround is to run:
> mkdir /var/lib/config-data/nova_libvirt/etc/libvirt/secrets
>
> before starting the nova_libvirt container.
Taking it back to container-DFG
I think Martin is on track to fix this upstream with this patch: https://review.openstack.org/#/c/476153/4/docker/services/nova-compute.yaml This will be fixed by a series of patches to enable config files copy in the container instead of direct bind mount. These are on the right track to land, but I expect it takes a bit more time due to upstream inertia. Let me know if this is a blocker and I'll work on providing a workaround. I think a simple workaround is to add a named volume for '/etc/libvirt/secrets'. The performance of named volumes isn't great, but at least we won't be hitting permissions issues. In tripleo-heat-templates/docker/services/nova-libvirt.yaml, for nova_libvirt container's volumes [1] you'll need to add a line: - libvirt_secrets:/etc/libvirt/secrets [1] https://github.com/openstack/tripleo-heat-templates/blob/af3828437e4ec92a738023afd29f12225161866e/docker/services/nova-libvirt.yaml#L128 Libvirt expects the entire of /etc/libvirt to be writable, not just that one sub directory. Martin's fixes in https://review.openstack.org/#/c/476153/ upstream will eventually make the entire /etc/libvirt tree writable to Libvirt. So I think we are on track to fix this correctly for the mid/long term. ---- Until all that lands however I think it would be reasonable to work around this with the named volume work around added to the volumes section of the docker/services/nova-libvirt.yaml template as Martin also sugests: - libvirt_secrets:/etc/libvirt/secrets Martin: we may also need permissions adjusted on this directory? Could use kolla_config's permissions to do this if so. If it works as is though I think it is probably fine to leave it with the defaults. All the fixes merged upstream and should already be available in the latest puddle. Sasha, can you check again? The issues doesn't reproduce. Environment: libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64 libvirt-daemon-driver-network-3.2.0-14.el7.x86_64 libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64 libvirt-libs-3.2.0-14.el7.x86_64 libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64 libvirt-daemon-config-network-3.2.0-14.el7.x86_64 libvirt-client-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64 libvirt-daemon-kvm-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64 libvirt-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64 libvirt-python-3.2.0-3.el7.x86_64 libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64 libvirt-daemon-3.2.0-14.el7.x86_64 libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64 openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch instack-undercloud-7.1.1-0.20170710151630.el7ost.noarch Will the 'fixed in version' field be updated? What RPM includes the fix? We got the kolla & THT changes but not yet the puppet-tripleo change (it'll be part of next import) it suppose to be ON_QA? Verified: Environment: openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch The reported issue doesn't reproduce. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |