This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1274684 - cinder volume ends up attached multiple times according to DB
cinder volume ends up attached multiple times according to DB
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: Eoghan Glynn
nlevinki
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-23 07:03 EDT by David Juran
Modified: 2015-11-24 11:04 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-24 11:04:22 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (879.08 KB, application/x-xz)
2015-11-12 09:41 EST, David Juran
no flags Details

  None (edit)
Description David Juran 2015-10-23 07:03:13 EDT
Description of problem:
On my horribly overloaded home lab, I attached a cinder volume to an instance. Then inspecting the block_device_mapping table in the nova DB, the device seem to be attached three times:

MariaDB [nova]> select instance_uuid,device_name,volume_id,connection_info from block_device_mapping WHERE volume_id='33a22444-e243-4c8f-abc8-52eff4d1492a';
+--------------------------------------+-------------+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| instance_uuid                        | device_name | volume_id                            | connection_info                                                                                                                                                                                                                                                                                                                                                                |
+--------------------------------------+-------------+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| bf22c858-6e0b-40b2-983a-9522c02ccb78 | /dev/vdb    | 33a22444-e243-4c8f-abc8-52eff4d1492a | NULL                                                                                                                                                                                                                                                                                                                                                                           |
| bf22c858-6e0b-40b2-983a-9522c02ccb78 | /dev/vdc    | 33a22444-e243-4c8f-abc8-52eff4d1492a | NULL                                                                                                                                                                                                                                                                                                                                                                           |
| bf22c858-6e0b-40b2-983a-9522c02ccb78 | /dev/vdd    | 33a22444-e243-4c8f-abc8-52eff4d1492a | {"driver_volume_type": "rbd", "serial": "33a22444-e243-4c8f-abc8-52eff4d1492a", "data": {"secret_type": "ceph", "name": "volumes/volume-33a22444-e243-4c8f-abc8-52eff4d1492a", "secret_uuid": "9f304ada-7725-11e5-9d0e-525400e60c17", "qos_specs": null, "hosts": ["192.0.2.14"], "auth_enabled": true, "access_mode": "rw", "auth_username": "openstack", "ports": ["6789"]}} |
+--------------------------------------+-------------+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00 sec)


Looking in the cinder DB, it's only there once:
MariaDB [cinder]> select * from volume_attachment;
+---------------------+---------------------+------------+---------+--------------------------------------+--------------------------------------+---------------+--------------------------------------+------------+---------------------+-------------+-------------+---------------+
| created_at          | updated_at          | deleted_at | deleted | id                                   | volume_id                            | attached_host | instance_uuid                        | mountpoint | attach_time         | detach_time | attach_mode | attach_status |
+---------------------+---------------------+------------+---------+--------------------------------------+--------------------------------------+---------------+--------------------------------------+------------+---------------------+-------------+-------------+---------------+
| 2015-10-22 15:40:37 | 2015-10-22 15:40:38 | NULL       |       0 | f23530a3-2b8d-4e9c-b32a-24239e72378f | 33a22444-e243-4c8f-abc8-52eff4d1492a | NULL          | bf22c858-6e0b-40b2-983a-9522c02ccb78 | /dev/vdd   | 2015-10-22 15:40:38 | NULL        | rw          | attached      |
+---------------------+---------------------+------------+---------+--------------------------------------+--------------------------------------+---------------+--------------------------------------+------------+---------------------+-------------+-------------+---------------+
1 row in set (0.00 sec)




And of course, in reality, on the instance itself, the volume is only attached once and fully functional. According to libvirt on the compute node, as /dev/vdd.

When then trying to detach the volume, the process fails since connection_info is empty fro those bogus connections

Version-Release number of selected component (if applicable):
openstack-nova-2015.1.1-3.el7ost
openstack-cinder-2015.1.1-2.el7ost
Comment 2 Dan Smith 2015-11-06 10:25:00 EST
Can you include the deleted column in the block-device table dump?

Also, please include full system logs around the time of the failure, as well as the commands you're running to do the detach, and the related output from them.
Comment 3 David Juran 2015-11-12 09:32:27 EST
Do note I've done some cleanup since this bug was reported, notably I deleted the bogus entries from the DB, so not sure how much this is worth:

MariaDB [nova]> select deleted,instance_uuid,device_name,volume_id,connection_info from block_device_mapping WHERE volume_id='33a22444-e243-4c8f-abc8-52eff4d1492a';
+---------+--------------------------------------+-------------+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| deleted | instance_uuid                        | device_name | volume_id                            | connection_info                                                                                                                                                                                                                                                                                                                                                                |
+---------+--------------------------------------+-------------+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|       5 | bf22c858-6e0b-40b2-983a-9522c02ccb78 | /dev/vdd    | 33a22444-e243-4c8f-abc8-52eff4d1492a | {"driver_volume_type": "rbd", "serial": "33a22444-e243-4c8f-abc8-52eff4d1492a", "data": {"secret_type": "ceph", "name": "volumes/volume-33a22444-e243-4c8f-abc8-52eff4d1492a", "secret_uuid": "9f304ada-7725-11e5-9d0e-525400e60c17", "qos_specs": null, "hosts": ["192.0.2.14"], "auth_enabled": true, "access_mode": "rw", "auth_username": "openstack", "ports": ["6789"]}} |
+---------+--------------------------------------+-------------+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


Attachning the logs from what I _think_ is when this error occurred, though I'm not entirely sure of the point-in-time.
Comment 4 David Juran 2015-11-12 09:41 EST
Created attachment 1093318 [details]
logs
Comment 5 Eoghan Glynn 2015-11-24 11:04:22 EST
Please re-open when/if the issue is reproduced without cleaning DB and removing logs.

Note You need to log in before you can comment on or make changes to this bug.