Bug 1434968

Summary: OSP10 volume data lost at detach volume encrypted over nfs or vm deletion
Product: Red Hat OpenStack Reporter: Cyril Lopez <cylopez>
Component: openstack-novaAssignee: Matthew Booth <mbooth>
Status: CLOSED CURRENTRELEASE QA Contact: Prasanth Anbalagan <panbalag>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: dasmith, eglynn, eharney, kchamart, lyarwood, mbooth, sbauza, sgordon, srevivo, vromanso
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-16 11:43:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Cyril Lopez 2017-03-22 18:57:11 UTC 
Description of problem:
When cinder backend is nfs and volume are encrypted, add a volume to a vm is working good but at detach or deletion of vm, the volume data is lost.

Version-Release number of selected component (if applicable):
python-nova-14.0.3-8.el7ost.noarch
python-novaclient-6.0.0-1.el7ost.noarch
openstack-nova-conductor-14.0.3-8.el7ost.noarch
openstack-nova-scheduler-14.0.3-8.el7ost.noarch
openstack-nova-api-14.0.3-8.el7ost.noarch
openstack-nova-common-14.0.3-8.el7ost.noarch
openstack-nova-compute-14.0.3-8.el7ost.noarch
openstack-cinder-9.1.1-3.el7ost.noarch
python-cinderclient-1.9.0-5.el7ost.noarch
python-cinder-9.1.1-3.el7ost.noarch
libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64
libvirt-client-2.0.0-10.el7_3.5.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-2.0.0-10.el7_3.5.x86_64
cryptsetup-1.7.2-1.el7.x86_64
cryptsetup-libs-1.7.2-1.el7.x86_64

How reproducible:
Deploy OSP10 with nfs backend, setup the keymgr/fixed_key in nova and cinder

Steps to Reproduce:
1. deploy with director
2. cinder type-create LUKS
3. cinder type-key LUKS set volume_backend_name=nfs
4. cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
5. openstack volume create --size 1 --type LUKS chiffre
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2017-03-22T18:26:51.471440           |
| description         | None                                 |
| encrypted           | True                                 |
| id                  | e753ff22-3ea3-4455-a3af-7d7bfd6880da |
| multiattach         | False                                |
| name                | chiffre                              |
| properties          |                                      |
| replication_status  | disabled                             |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | LUKS                                 |
| updated_at          | None                                 |
| user_id             | 239cd8dc013042d7a4b8138942e31b0d     |
+---------------------+--------------------------------------+

On NFS server :
ls /storage/ -lha
-rw-rw-rw-.  1 nfsnobody nfsnobody 1.0G Mar 22 18:26 volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da

6. openstack server add volume chiffre chiffre

7.[root@chiffre ~]# fdisk -l /dev/vdb
Disk /dev/vdb: 1022 MiB, 1071644672 bytes, 2093056 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

on NFS server :
[root@overcloud-objectstorage-0 ~]# ls /storage/ -lha
total 260K
drwxrwxrwx.  2 root      root       195 Mar 22 18:28 .
drwxr-xr-x. 18 root      root       257 Mar 22 10:27 ..
-rw-rw-rw-.  1 nfsnobody nfsnobody 1.0G Mar 22 18:28 .nfs000000000000040400000003
lrwxrwxrwx.  1 nfsnobody nfsnobody   61 Mar 22 18:28 volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da -> /dev/mapper/crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da
cryptsetup status crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da
/dev/mapper/crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da is active.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/loop0
  loop:    /var/lib/nova/mnt/415db1b1714269aabb5f696a22f79d88/.nfs000000000000040400000006
  offset:  4096 sectors
  size:    2093056 sectors
  mode:    read/write

8. openstack server remove volume chiffre chiffre

on NFS server:
[root@overcloud-objectstorage-0 ~]# ls /storage/ -lha
lrwxrwxrwx.  1 nfsnobody nfsnobody   61 Mar 22 18:28 volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da -> /dev/mapper/crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da


Actual results:
We loose the data of the volume.

Expected results:
It's suppose to have a cryptsetup luksClose on the file to recover it.

Nova log of this example:
2017-03-22 18:27:54.477 161191 INFO nova.compute.manager [req-0a805ba4-8940-401c-b062-7605ca0276ce 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] [instance: 3c85cff2-dedb-4f72-bcc8-46a83d59d703] Attaching volume e753ff22-3ea3-4455-a3af-7d7bfd6880da to /dev/vdb
2017-03-22 18:27:56.211 161191 WARNING nova.volume.encryptors.luks [req-0a805ba4-8940-401c-b062-7605ca0276ce 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] isLuks exited abnormally (status 1): Device /var/lib/nova/mnt/415db1b1714269aabb5f696a22f79d88/volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da is not a valid LUKS device.
2017-03-22 18:27:56.212 161191 INFO nova.volume.encryptors.luks [req-0a805ba4-8940-401c-b062-7605ca0276ce 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] /var/lib/nova/mnt/415db1b1714269aabb5f696a22f79d88/volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da is not a valid LUKS device; formatting device for first use
2017-03-22 18:29:44.478 161191 INFO nova.compute.manager [req-5d53bd2a-51e8-4147-8ccd-3ba72456b258 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] [instance: 3c85cff2-dedb-4f72-bcc8-46a83d59d703] Detach volume e753ff22-3ea3-4455-a3af-7d7bfd6880da from mountpoint /dev/vdb
Comment 1 Matthew Booth 2019-05-16 11:43:32 UTC 
This was fixed in OSP13, but unfortunately can't be backported to OSP10.