Bug 1434968 - OSP10 volume data lost at detach volume encrypted over nfs or vm deletion
Summary: OSP10 volume data lost at detach volume encrypted over nfs or vm deletion
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 10.0 (Newton)
Assignee: Matthew Booth
QA Contact: Prasanth Anbalagan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-22 18:57 UTC by Cyril Lopez
Modified: 2019-09-09 14:00 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-16 11:43:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 523958 None None None 2018-01-31 13:36:06 UTC
Red Hat Bugzilla 1305022 medium ON_DEV [RFE][cinder] Support volume encryption on NFS backends 2020-10-14 00:28:05 UTC

Description Cyril Lopez 2017-03-22 18:57:11 UTC
Description of problem:
When cinder backend is nfs and volume are encrypted, add a volume to a vm is working good but at detach or deletion of vm, the volume data is lost.

Version-Release number of selected component (if applicable):
python-nova-14.0.3-8.el7ost.noarch
python-novaclient-6.0.0-1.el7ost.noarch
openstack-nova-conductor-14.0.3-8.el7ost.noarch
openstack-nova-scheduler-14.0.3-8.el7ost.noarch
openstack-nova-api-14.0.3-8.el7ost.noarch
openstack-nova-common-14.0.3-8.el7ost.noarch
openstack-nova-compute-14.0.3-8.el7ost.noarch
openstack-cinder-9.1.1-3.el7ost.noarch
python-cinderclient-1.9.0-5.el7ost.noarch
python-cinder-9.1.1-3.el7ost.noarch
libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64
libvirt-client-2.0.0-10.el7_3.5.x86_64
libvirt-2.0.0-10.el7_3.5.x86_64
libvirt-daemon-2.0.0-10.el7_3.5.x86_64
cryptsetup-1.7.2-1.el7.x86_64
cryptsetup-libs-1.7.2-1.el7.x86_64

How reproducible:
Deploy OSP10 with nfs backend, setup the keymgr/fixed_key in nova and cinder

Steps to Reproduce:
1. deploy with director
2. cinder type-create LUKS
3. cinder type-key LUKS set volume_backend_name=nfs
4. cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
5. openstack volume create --size 1 --type LUKS chiffre
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2017-03-22T18:26:51.471440           |
| description         | None                                 |
| encrypted           | True                                 |
| id                  | e753ff22-3ea3-4455-a3af-7d7bfd6880da |
| multiattach         | False                                |
| name                | chiffre                              |
| properties          |                                      |
| replication_status  | disabled                             |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | LUKS                                 |
| updated_at          | None                                 |
| user_id             | 239cd8dc013042d7a4b8138942e31b0d     |
+---------------------+--------------------------------------+

On NFS server :
ls /storage/ -lha
-rw-rw-rw-.  1 nfsnobody nfsnobody 1.0G Mar 22 18:26 volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da

6. openstack server add volume chiffre chiffre

7.[root@chiffre ~]# fdisk -l /dev/vdb
Disk /dev/vdb: 1022 MiB, 1071644672 bytes, 2093056 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

on NFS server :
[root@overcloud-objectstorage-0 ~]# ls /storage/ -lha
total 260K
drwxrwxrwx.  2 root      root       195 Mar 22 18:28 .
drwxr-xr-x. 18 root      root       257 Mar 22 10:27 ..
-rw-rw-rw-.  1 nfsnobody nfsnobody 1.0G Mar 22 18:28 .nfs000000000000040400000003
lrwxrwxrwx.  1 nfsnobody nfsnobody   61 Mar 22 18:28 volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da -> /dev/mapper/crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da
cryptsetup status crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da
/dev/mapper/crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da is active.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/loop0
  loop:    /var/lib/nova/mnt/415db1b1714269aabb5f696a22f79d88/.nfs000000000000040400000006
  offset:  4096 sectors
  size:    2093056 sectors
  mode:    read/write

8. openstack server remove volume chiffre chiffre

on NFS server:
[root@overcloud-objectstorage-0 ~]# ls /storage/ -lha
lrwxrwxrwx.  1 nfsnobody nfsnobody   61 Mar 22 18:28 volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da -> /dev/mapper/crypt-volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da


Actual results:
We loose the data of the volume.

Expected results:
It's suppose to have a cryptsetup luksClose on the file to recover it.

Nova log of this example:
2017-03-22 18:27:54.477 161191 INFO nova.compute.manager [req-0a805ba4-8940-401c-b062-7605ca0276ce 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] [instance: 3c85cff2-dedb-4f72-bcc8-46a83d59d703] Attaching volume e753ff22-3ea3-4455-a3af-7d7bfd6880da to /dev/vdb
2017-03-22 18:27:56.211 161191 WARNING nova.volume.encryptors.luks [req-0a805ba4-8940-401c-b062-7605ca0276ce 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] isLuks exited abnormally (status 1): Device /var/lib/nova/mnt/415db1b1714269aabb5f696a22f79d88/volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da is not a valid LUKS device.
2017-03-22 18:27:56.212 161191 INFO nova.volume.encryptors.luks [req-0a805ba4-8940-401c-b062-7605ca0276ce 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] /var/lib/nova/mnt/415db1b1714269aabb5f696a22f79d88/volume-e753ff22-3ea3-4455-a3af-7d7bfd6880da is not a valid LUKS device; formatting device for first use
2017-03-22 18:29:44.478 161191 INFO nova.compute.manager [req-5d53bd2a-51e8-4147-8ccd-3ba72456b258 239cd8dc013042d7a4b8138942e31b0d 95519e705c3441fda040de9583f2c01a - - -] [instance: 3c85cff2-dedb-4f72-bcc8-46a83d59d703] Detach volume e753ff22-3ea3-4455-a3af-7d7bfd6880da from mountpoint /dev/vdb

Comment 1 Matthew Booth 2019-05-16 11:43:32 UTC
This was fixed in OSP13, but unfortunately can't be backported to OSP10.


Note You need to log in before you can comment on or make changes to this bug.