Bug 1181220 - Nova's Block device mappings can become invalid/inconsistent if errors are encountered while calling for Cinder to attach a volume.
Summary: Nova's Block device mappings can become invalid/inconsistent if errors are en...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 5.0 (RHEL 7)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: z6
: 5.0 (RHEL 7)
Assignee: Michal Pryc
QA Contact: Prasanth Anbalagan
URL:
Whiteboard:
Depends On:
Blocks: 1293365
TreeView+ depends on / blocked
 
Reported: 2015-01-12 15:55 UTC by Lee Yarwood
Modified: 2023-02-22 23:02 UTC (History)
17 users (show)

Fixed In Version: openstack-nova-2014.1.5-13.el7ost
Doc Type: Bug Fix
Doc Text:
With this fix the block device information is being persisted to DB even if exception occurs. This wasn't the case for the methods wrapped with the @update_db decorator, which was causing Nova to not save data about volumes.
Clone Of:
: 1293365 (view as bug list)
Environment:
Last Closed: 2016-03-08 17:07:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1439282 0 None None None Never
OpenStack gerrit 170087 0 None None None Never
OpenStack gerrit 170088 0 None None None Never
Red Hat Bugzilla 1293365 0 medium CLOSED Nova's Block device mappings can become invalid/inconsistent if errors are encountered while calling for Cinder to attac... 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2016:0361 0 normal SHIPPED_LIVE openstack-nova bug fix advisory 2016-03-08 22:06:33 UTC

Internal Links: 1293365

Comment 12 Prasanth Anbalagan 2016-03-01 19:57:14 UTC
Verified that volumes are correctly mapped to instances and removed along with the instance. The failure scenario described in the bug was never observed since cinder attach of volume was successful in all tries.

*************
Version
*************

[root@rhos-compute-node-02 ~(keystone_admin)]# yum list installed | grep openstack-nova
openstack-nova-api.noarch        2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-cert.noarch       2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-common.noarch     2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-compute.noarch    2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-conductor.noarch  2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-console.noarch    2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-novncproxy.noarch 2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-scheduler.noarch  2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
[root@rhos-compute-node-02 ~(keystone_admin)]# 

********
Logs
********

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova boot --flavor 2 --block-device source=image,id=a73e61d5-633e-4f3e-a728-c1c6b4dc946f,dest=volume,size=10,shutdown=preserve,bootindex=0 vm01
+--------------------------------------+-------------------------------------------------+
| Property                             | Value                                           |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                          |
| OS-EXT-AZ:availability_zone          | nova                                            |
| OS-EXT-SRV-ATTR:host                 | -                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                               |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000005                               |
| OS-EXT-STS:power_state               | 0                                               |
| OS-EXT-STS:task_state                | scheduling                                      |
| OS-EXT-STS:vm_state                  | building                                        |
| OS-SRV-USG:launched_at               | -                                               |
| OS-SRV-USG:terminated_at             | -                                               |
| accessIPv4                           |                                                 |
| accessIPv6                           |                                                 |
| adminPass                            | Bo82VrUFtx5F                                    |
| config_drive                         |                                                 |
| created                              | 2016-03-01T18:32:37Z                            |
| flavor                               | m1.small (2)                                    |
| hostId                               |                                                 |
| id                                   | f8f7d003-60f7-418f-aeab-1be34a3ccdc9            |
| image                                | Attempt to boot from volume - no image supplied |
| key_name                             | -                                               |
| metadata                             | {}                                              |
| name                                 | vm01                                            |
| os-extended-volumes:volumes_attached | []                                              |
| progress                             | 0                                               |
| security_groups                      | default                                         |
| status                               | BUILD                                           |
| tenant_id                            | aa8e7fc282f849fc9099e4b3910fd291                |
| updated                              | 2016-03-01T18:32:37Z                            |
| user_id                              | d45021949119414b97b55a94f7737ea2                |
+--------------------------------------+-------------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| f8f7d003-60f7-418f-aeab-1be34a3ccdc9 | vm01 | ACTIVE | -          | Running     | public=172.24.4.231 |
+--------------------------------------+------+--------+------------+-------------+---------------------+

[root@rhos-compute-node-02 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 3bc99f74-cbbe-4460-b1a7-1483236725c7 | in-use |              |  10  |     None    |   true   | f8f7d003-60f7-418f-aeab-1be34a3ccdc9 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+


MariaDB [nova]> 
MariaDB [nova]> select * from block_device_mapping where instance_uuid = 'f8f7d003-60f7-418f-aeab-1be34a3ccdc9' \G;
*************************** 1. row ***************************
           created_at: 2016-03-01 18:32:37
           updated_at: 2016-03-01 18:32:41
           deleted_at: NULL
                   id: 5
          device_name: /dev/vda
delete_on_termination: 0
          snapshot_id: NULL
            volume_id: 3bc99f74-cbbe-4460-b1a7-1483236725c7
          volume_size: 10
            no_device: NULL
      connection_info: {"driver_volume_type": "iscsi", "serial": "3bc99f74-cbbe-4460-b1a7-1483236725c7", "data": {"access_mode": "rw", "target_discovered": false, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7", "target_portal": "10.8.0.50:3260", "volume_id": "3bc99f74-cbbe-4460-b1a7-1483236725c7", "target_lun": 0, "device_path": "/dev/disk/by-path/ip-10.8.0.50:3260-iscsi-iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7-lun-0", "auth_password": "xKeK9PAsPQh3ChmzwpZ7", "auth_username": "XJdjVNpoHgYLQhD3LL2Y", "auth_method": "CHAP"}}
        instance_uuid: f8f7d003-60f7-418f-aeab-1be34a3ccdc9
              deleted: 0
          source_type: image
     destination_type: volume
         guest_format: NULL
          device_type: disk
             disk_bus: virtio
           boot_index: 0
             image_id: a73e61d5-633e-4f3e-a728-c1c6b4dc946f
1 row in set (0.00 sec)


MariaDB [nova]> use cinder;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [cinder]> 
MariaDB [cinder]> select * from volumes where id='3bc99f74-cbbe-4460-b1a7-1483236725c7' \G;
*************************** 1. row ***************************
         created_at: 2016-03-01 18:32:38
         updated_at: 2016-03-01 18:32:40
         deleted_at: NULL
            deleted: 0
                 id: 3bc99f74-cbbe-4460-b1a7-1483236725c7
             ec2_id: NULL
            user_id: d45021949119414b97b55a94f7737ea2
         project_id: aa8e7fc282f849fc9099e4b3910fd291
               host: rhos-compute-node-02.lab.eng.rdu2.redhat.com
               size: 10
  availability_zone: nova
      instance_uuid: f8f7d003-60f7-418f-aeab-1be34a3ccdc9
         mountpoint: /dev/vda
        attach_time: 2016-03-01T18:32:40.720727
             status: in-use
      attach_status: attached
       scheduled_at: 2016-03-01 18:32:38
        launched_at: 2016-03-01 18:32:39
      terminated_at: NULL
       display_name: 
display_description: 
  provider_location: 10.8.0.50:3260,iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7 iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7 0
      provider_auth: CHAP XJdjVNpoHgYLQhD3LL2Y xKeK9PAsPQh3ChmzwpZ7
        snapshot_id: NULL
     volume_type_id: NULL
       source_volid: NULL
           bootable: 1
      attached_host: NULL
  provider_geometry: NULL
           _name_id: NULL
  encryption_key_id: NULL
   migration_status: NULL
1 row in set (0.00 sec)




*****************************************
Deleteing volume upon removing instance
*****************************************

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova boot --flavor 2 --block-device-mapping vda=d5b65335-d32c-41fd-8c96-0bc161ff9d23:::1 vm02+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          | nova                                             |
| OS-EXT-SRV-ATTR:host                 | -                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000008                                |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | mLZawx47QYbE                                     |
| config_drive                         |                                                  |
| created                              | 2016-03-01T19:49:29Z                             |
| flavor                               | m1.small (2)                                     |
| hostId                               |                                                  |
| id                                   | a6b2d9a8-a673-445a-80c9-02b9dabfbc35             |
| image                                | Attempt to boot from volume - no image supplied  |
| key_name                             | -                                                |
| metadata                             | {}                                               |
| name                                 | vm02                                             |
| os-extended-volumes:volumes_attached | [{"id": "d5b65335-d32c-41fd-8c96-0bc161ff9d23"}] |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | aa8e7fc282f849fc9099e4b3910fd291                 |
| updated                              | 2016-03-01T19:49:29Z                             |
| user_id                              | d45021949119414b97b55a94f7737ea2                 |
+--------------------------------------+--------------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| a6b2d9a8-a673-445a-80c9-02b9dabfbc35 | vm02 | ACTIVE | -          | Running     | public=172.24.4.234 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| d5b65335-d32c-41fd-8c96-0bc161ff9d23 | in-use |   bootvol    |  2   |     None    |   true   | a6b2d9a8-a673-445a-80c9-02b9dabfbc35 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova delete vm02
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# cinder list
+----+--------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+----+--------+--------------+------+-------------+----------+-------------+
+----+--------+--------------+------+-------------+----------+-------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]#

Comment 14 errata-xmlrpc 2016-03-08 17:07:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0361.html


Note You need to log in before you can comment on or make changes to this bug.