Bug 1824120

QE, I've included some notes below regarding limitations, deployment and testing. Ultimately I'd like to see a job defined that exercises both workarounds executing the compute, encrypted volume and live migration tests. Happy to help with this.

[ Limitations of the these workarounds ]

- Resizing of attached volumes is not supported.

- Live migration between mixed computes with these workarounds both enabled and disabled is not supported.

- Pre-existing ceph volumes must have the object-map fast-diff deep-flatten features disabled before use with this workaround:

$ rbd --id openstack feature disable volumes/volume-${volume_uuid} object-map fast-diff deep-flatten

- ceph.conf should be updated on the controllers hosting cinder-volume to ensure new volumes are also created without these features enabled allowing them to be used with this workaround, these can be re-enabled later:

[global]
default features = 3


[ Deployment Notes ]

I used the following job to deploy an OSP 13  Ceph based env with Barbican:

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/All/job/DFG-all-unified-13_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-ceph-native-default/430/

Note that the default Compute RAM with this job is only 6GB, I manually updated the VMs after deployment but with future runs you will want to override these ir deployment options as has already been done for the controllers.

The required openstack-nova and python-os-brick builds aren't part of a puddle yet, for now please find them below:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1179489

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1179529

To deploy these on a compute do the following:

1. Install the required updates:

$ sudo docker exec -ti -u root nova_compute bash
> yum update http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/python-nova-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/openstack-nova-common-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/openstack-nova-compute-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/openstack-nova/17.0.13/6.el7ost/noarch/openstack-nova-migration-17.0.13-6.el7ost.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/python-os-brick/2.3.9/2.el7ost/noarch/python2-os-brick-2.3.9-2.el7ost.noarch.rpm -y
> exit

2. Enable both the RBD and LUKSv1 workarounds in nova.conf:

$ sudo crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf workarounds disable_native_luksv1 true
$ sudo crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf workarounds  rbd_volume_local_attach true

3. Restart nova_compute to ensure these are picked up by the compute container:

$ sudo docker restart nova_compute

4. Verify that they are set within the container:

$ sudo docker exec -ti -u root nova_compute crudini --get /etc/nova/nova.conf workarounds disable_native_luksv1
$ sudo docker exec -ti -u root nova_compute crudini --get /etc/nova/nova.conf workarounds rbd_volume_local_attach

Alternatively once the packages are provided in a puddle we can enable these at deployment with something like:

parameter_defaults:
  ComputeExtraConfig:
    nova::config::nova_config:
      workarounds/disable_native_luksv1:
        value: True
      workarounds/rbd_volume_local_attach:
        value: True

Additionally we also need to set the following option in the /etc/ceph/ceph.conf of the controller running cinder-volume:

[global]
default features = 3

Then restart the openstack-cinder-volume-docker-0 to ensure it is picked up by the container *before* creating any volumes:

$ sudo docker restart openstack-cinder-volume-docker-0

Alternatively you can enable this during the deployment with this extra config:

parameter_defaults:
  CephConfigOverrides:
    global:
      default features: 3

[ Testing Notes ]

Use the following commands to verify if an instance is correctly using both workarounds on a given compute.

This example uses an instance with a UUID of 0c860d72-8dd8-488b-9cc1-86eca61c2f2f that has an ephemeral disk (controlled by Nova) and encrypted volume (controlled by Cinder) RBD volumes attached:

- Show all block devices connected to the instance:

$ sudo virsh domblklist 0c860d72-8dd8-488b-9cc1-86eca61c2f2f
Target     Source
------------------------------------------------
vda        vms/0c860d72-8dd8-488b-9cc1-86eca61c2f2f_disk
vdb        /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3

- Ensure the host block device path for the encrypted RBD volume provided by cinder points to a dm-crypt device:

$ ll /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3
lrwxrwxrwx. 1 root root 61 Apr 30 18:56 /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 -> /dev/mapper/crypt-volume-07e8b5e7-7430-42a9-93fd-ad605096abf3

- Ensure the dm-crypt device in turn is using a host rbd block device:

$ sudo cryptsetup status crypt-volume-07e8b5e7-7430-42a9-93fd-ad605096abf3
/dev/mapper/crypt-volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/rbd0
  sector size:  512
  offset:  4040 sectors
  size:    2097152 sectors
  mode:    read/write

- The /dev/rbd0 device should appear encrypted while the attached /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3 is decrypted:

$ sudo qemu-img info /dev/rbd0
image: /dev/rbd0
file format: luks
virtual size: 1.0G (1073741824 bytes)
disk size: 0
encrypted: yes
Format specific information:
    ivgen alg: plain64
    hash alg: sha256
    cipher alg: aes-256
    uuid: e2a33fad-1973-41d9-a928-bfdb04811dcd
    cipher mode: xts
    slots:
[..]

$ sudo qemu-img info /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3
image: /dev/rbd/volumes/volume-07e8b5e7-7430-42a9-93fd-ad605096abf3
file format: raw
virtual size: 1.0G (1073741824 bytes)
disk size: 0

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2725