Bug 1487920

Can we get DEBUG nova-compute logs from the environment?

Note that the test here isn't correct for testing RHBZ#1285089 that is associated with _booting_ an instance from a volume that is _already_ encrypted. We should be creating an encrypted volume and directly booting from it.

This appears to either be an issue with c-vol encrypting the original volume _or_ the n-cpu container not having access to the required underlying host block devices. Either way n-cpu DEBUG logs should show us what is at fault here.

(In reply to Lee Yarwood from comment #2)
> Can we get DEBUG nova-compute logs from the environment?
> 
> Note that the test here isn't correct for testing RHBZ#1285089 that is
> associated with _booting_ an instance from a volume that is _already_
> encrypted. We should be creating an encrypted volume and directly booting
> from it.
> 
> This appears to either be an issue with c-vol encrypting the original volume
> _or_ the n-cpu container not having access to the required underlying host
> block devices. Either way n-cpu DEBUG logs should show us what is at fault
> here.

This Issue been revealed as part of test run for the following RFE on containerize environment. 
https://bugzilla.redhat.com/show_bug.cgi?id=1406802
I will setup debug, run it again and send the output.

(In reply to Avi Avraham from comment #3)
> This Issue been revealed as part of test run for the following RFE on
> containerize environment. 
> https://bugzilla.redhat.com/show_bug.cgi?id=1406802
> I will setup debug, run it again and send the output.

Thanks, adding the needinfo back in until we get logs here. 

Please flip it back to me once we have these.

Closing this out with INSUFFICIENT_DATA, please reopen if this reproduces with DEBUG nova-compute logs and I'll be happy to take another look at this.

I wonder if this issue is related bug #1484467, with dealt with an IQN mismatch between services running on bare metal and containers in the same host.

I reproduce the setup heating the same issue again 
Adding log files + I got a working environment to test it 
Please update if someone can take a look on the setup

Created attachment 1343211 [details]
nova-compute debug log

The immediate reason this error is occurring is that luksOpen hangs on the compute. When we try to attach a fresh encrypted volume to a nova instance nova first attaches the iscsi volume, which succeeds, then attaches the luks encryptor. The encryptor does the following sequence of calls:

1. cryptsetup luksOpen: this fails fast, because the the volume isn't luks formatted yet
2. cryptsetup isLuks: this reveals that it's not luks
3. cryptsetup luksFormat: this formats the volume as luks
4. cryptsetup luksOpen: this hangs indefinitely

Because it hangs rather than fails, Nova's cleanup never runs. It never detaches the iscsi volume from the compute, and therefore the volume remains in the attaching state. I don't see that there is anything either Nova or Cinder can do about this.

Note that while luksOpen is hung and never returns, the encrypted block device is created on the compute and appears to be functional:

========
()[root@compute-0 /]# ls /dev/mapper/
control  crypt-scsi-360014059815b9b9aaf94198b915113ee
()[root@compute-0 /]# mke2fs /dev/mapper/crypt-scsi-360014059815b9b9aaf94198b915113ee
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=16 blocks, Stripe width=16 blocks
65408 inodes, 261632 blocks
13081 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8176 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
========

I replaced /usr/sbin/cryptsetup with a wrapper which invokes it via strace. I've attached the full log of that to the bug, but the pertinent part seems to be:

========
ioctl(5, DM_DEV_CREATE, 0x243da70)      = 0
ioctl(5, DM_TABLE_LOAD, 0x243da70)      = 0
ioctl(5, DM_DEV_SUSPEND, 0x24399b0)     = 0
semget(0xd4d9dce, 1, 0)                 = 0
semctl(0, 0, GETVAL, 0xffffffff)        = 2
semop(0, [{0, -1, IPC_NOWAIT}], 1)      = 0
semop(0, [{0, 0, 0}], 1
========

A search revealed this post, which seems to describe a very similar/identical issue:

https://github.com/moby/moby/issues/22025

I'm going to change the component here to os-brick as that's the immediate high-level component causing the issue. However, I suspect the actual issue is at a lower layer.

Created attachment 1343802 [details]
strace of cryptsetup which hangs indefinitely

*** Bug 1478161 has been marked as a duplicate of this bug. ***

Created attachment 1346673 [details]
cryptsetup luksOpen --debug output

Output of a cryptsetup run that hangs with --debug.

Launching the nova compute container with the docker "--ipc=host" option seems to resolve this issue.

Both the manual cryptsetup luksOpen test from comment #14 succeeded, as did an attach of an encrypted Cinder volume.

The bug was verified as cherry pick 
RPM was manually installed 
Polarion test run 
https://polarion.engineering.redhat.com/polarion/#/project/RHELOpenStackPlatform/testrun?id=20170813-1217
waiting for the package to be part of rhos12 puddle

Verified on:
openstack-tripleo-heat-templates-7.0.3-0.20171024200825.el7ost.noarch

Test on a containerized system, works as expected.
1. Created and successfully attached two Cinder volumes encrypted and none encrypted to an instance 

(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+
| ID                                   | Status | Name             | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+
| 6c33b84b-14c2-4eb8-a9a4-12b7457c4310 | in-use | -                | 1    | -           | false    | 056a61f4-53b0-4dd3-8e78-77ad42901d4a |
| 985a2bc6-8f13-4d35-bd0b-1252e2269355 | in-use | encrypted volume | 1    | LUKS        | false    | 056a61f4-53b0-4dd3-8e78-77ad42901d4a |
+--------------------------------------+--------+------------------+------+-------------+----------+--------------------------------------+

2. $ nova list
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                          |
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+
| 056a61f4-53b0-4dd3-8e78-77ad42901d4a | inst1 | ACTIVE | -          | Running     | internal=192.168.0.16, 10.0.0.212 |
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+


3. Wrote txt data to both volumes.

4. Synced volumes 

5. On Cinder volume grep-ed for txt data, only none encrypted data was returned.

# strings /dev/cinder-volumes/* | grep txt
tshefi.txt

# strings /dev/cinder-volumes/* | grep kuku  


Encrypted data kuku.txt isn't returned as expected. 


Encrypted volume is successfully attached to an instance on a containerized nova compute deployment.

Created attachment 1351494 [details]
config and logs of verification.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462