QE, I've included some notes below regarding limitations, deployment and testing. Ultimately I'd like to see a job defined that exercises both workarounds executing the compute, encrypted volume and live migration tests. Happy to help with this. [ Limitations of the these workarounds ] - Resizing of attached volumes is not supported. - Live migration between mixed computes with these workarounds both enabled and disabled is not supported. - Pre-existing ceph volumes must have the object-map fast-diff deep-flatten features disabled before use with this workaround: $ rbd --id openstack feature disable volumes/volume-${volume_uuid} object-map fast-diff deep-flatten - ceph.conf should be updated on the controllers hosting cinder-volume to ensure new volumes are also created without these features enabled allowing them to be used with this workaround, these can be re-enabled later: [global] default features = 3 [ Deployment Notes ] I used the following job to deploy an OSP 13 Ceph based env with Barbican: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/All/job/DFG-all-unified-13_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-ceph-native-default/430/ Note that the default Compute RAM with this job is only 6GB, I manually updated the VMs after deployment but with future runs you will want to override these ir deployment options as has already been done for the controllers. The required openstack-nova and python-os-brick builds aren't part of a puddle yet, for now please find them below: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1179489 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1179529 To deploy these on a compute do the following: 1. Install the required updates: $ sudo docker exec -ti -u root nova_compute bash > yum update http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/python-nova-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/openstack-nova-common-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/openstack-nova-compute-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/openstack-nova-migration-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/python-os-brick/2.3.9/2.el7ost/noarch/python2-os-brick-2.3.9-2.el7ost.noarch.rpm -y > exit 2. Enable both the RBD and LUKSv1 workarounds in nova.conf: $ sudo crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf workarounds disable_native_luksv1 true $ sudo crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf workarounds rbd_volume_local_attach true 3. Restart nova_compute to ensure these are picked up by the compute container: $ sudo docker restart nova_compute 4. Verify that they are set within the container: $ sudo docker exec -ti -u root nova_compute crudini --get /etc/nova/nova.conf workarounds disable_native_luksv1 $ sudo docker exec -ti -u root nova_compute crudini --get /etc/nova/nova.conf workarounds rbd_volume_local_attach Alternatively once the packages are provided in a puddle we can enable these at deployment with something like: parameter_defaults: ComputeExtraConfig: nova::config::nova_config: workarounds/disable_native_luksv1: value: True workarounds/rbd_volume_local_attach: value: True Additionally we also need to set the following option in the /etc/ceph/ceph.conf of the controller running cinder-volume: [global] default features = 3 Then restart the openstack-cinder-volume-docker-0 to ensure it is picked up by the container *before* creating any volumes: $ sudo docker restart openstack-cinder-volume-docker-0 Alternatively you can enable this during the deployment with this extra config: parameter_defaults: CephConfigOverrides: global: default features: 3 [ Testing Notes ] Use the following commands to verify if an instance is correctly using both workarounds on a given compute. This example uses an instance with a UUID of 0c860d72-8dd8-488b-9cc1-86eca61c2f2f that has an ephemeral disk (controlled by Nova) and encrypted volume (controlled by Cinder) RBD volumes attached: - Show all block devices connected to the instance: $ sudo virsh domblklist 0c860d72-8dd8-488b-9cc1-86eca61c2f2f Target Source ------------------------------------------------ vda vms/0c860d72-8dd8-488b-9cc1-86eca61c2f2f_disk vdb /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 - Ensure the host block device path for the encrypted RBD volume provided by cinder points to a dm-crypt device: $ ll /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 lrwxrwxrwx. 1 root root 61 Apr 30 18:56 /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 -> /dev/mapper/crypt-volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 - Ensure the dm-crypt device in turn is using a host rbd block device: $ sudo cryptsetup status crypt-volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 /dev/mapper/crypt-volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 is active and is in use. type: LUKS1 cipher: aes-xts-plain64 keysize: 512 bits key location: dm-crypt device: /dev/rbd0 sector size: 512 offset: 4040 sectors size: 2097152 sectors mode: read/write - The /dev/rbd0 device should appear encrypted while the attached /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 is decrypted: $ sudo qemu-img info /dev/rbd0 image: /dev/rbd0 file format: luks virtual size: 1.0G (1073741824 bytes) disk size: 0 encrypted: yes Format specific information: ivgen alg: plain64 hash alg: sha256 cipher alg: aes-256 uuid: e2a33fad-1973-41d9-a928-bfdb04811dcd cipher mode: xts slots: [..] $ sudo qemu-img info /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 image: /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 file format: raw virtual size: 1.0G (1073741824 bytes) disk size: 0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2725