Bug 1181220

Summary: Nova's Block device mappings can become invalid/inconsistent if errors are encountered while calling for Cinder to attach a volume.
Product: Red Hat OpenStack Reporter: Lee Yarwood <lyarwood>
Component: openstack-novaAssignee: Michal Pryc <mpryc>
Status: CLOSED ERRATA QA Contact: Prasanth Anbalagan <panbalag>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.0 (RHEL 7)CC: berrange, dasmith, dmaley, eglynn, jschluet, kchamart, lyarwood, mpryc, ndipanov, panbalag, pbandark, rdopiera, sbauza, sferdjao, sgordon, vromanso, yeylon
Target Milestone: z6Keywords: Triaged, ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-2014.1.5-13.el7ost Doc Type: Bug Fix
Doc Text:
With this fix the block device information is being persisted to DB even if exception occurs. This wasn't the case for the methods wrapped with the @update_db decorator, which was causing Nova to not save data about volumes.
Story Points: ---
Clone Of:
: 1293365 (view as bug list) Environment:
Last Closed: 2016-03-08 17:07:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1293365    

Comment 12 Prasanth Anbalagan 2016-03-01 19:57:14 UTC
Verified that volumes are correctly mapped to instances and removed along with the instance. The failure scenario described in the bug was never observed since cinder attach of volume was successful in all tries.

*************
Version
*************

[root@rhos-compute-node-02 ~(keystone_admin)]# yum list installed | grep openstack-nova
openstack-nova-api.noarch        2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-cert.noarch       2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-common.noarch     2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-compute.noarch    2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-conductor.noarch  2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-console.noarch    2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-novncproxy.noarch 2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
openstack-nova-scheduler.noarch  2014.1.5-27.el7ost      @rhelosp-5.0-el7-puddle
[root@rhos-compute-node-02 ~(keystone_admin)]# 

********
Logs
********

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova boot --flavor 2 --block-device source=image,id=a73e61d5-633e-4f3e-a728-c1c6b4dc946f,dest=volume,size=10,shutdown=preserve,bootindex=0 vm01
+--------------------------------------+-------------------------------------------------+
| Property                             | Value                                           |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                          |
| OS-EXT-AZ:availability_zone          | nova                                            |
| OS-EXT-SRV-ATTR:host                 | -                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                               |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000005                               |
| OS-EXT-STS:power_state               | 0                                               |
| OS-EXT-STS:task_state                | scheduling                                      |
| OS-EXT-STS:vm_state                  | building                                        |
| OS-SRV-USG:launched_at               | -                                               |
| OS-SRV-USG:terminated_at             | -                                               |
| accessIPv4                           |                                                 |
| accessIPv6                           |                                                 |
| adminPass                            | Bo82VrUFtx5F                                    |
| config_drive                         |                                                 |
| created                              | 2016-03-01T18:32:37Z                            |
| flavor                               | m1.small (2)                                    |
| hostId                               |                                                 |
| id                                   | f8f7d003-60f7-418f-aeab-1be34a3ccdc9            |
| image                                | Attempt to boot from volume - no image supplied |
| key_name                             | -                                               |
| metadata                             | {}                                              |
| name                                 | vm01                                            |
| os-extended-volumes:volumes_attached | []                                              |
| progress                             | 0                                               |
| security_groups                      | default                                         |
| status                               | BUILD                                           |
| tenant_id                            | aa8e7fc282f849fc9099e4b3910fd291                |
| updated                              | 2016-03-01T18:32:37Z                            |
| user_id                              | d45021949119414b97b55a94f7737ea2                |
+--------------------------------------+-------------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| f8f7d003-60f7-418f-aeab-1be34a3ccdc9 | vm01 | ACTIVE | -          | Running     | public=172.24.4.231 |
+--------------------------------------+------+--------+------------+-------------+---------------------+

[root@rhos-compute-node-02 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 3bc99f74-cbbe-4460-b1a7-1483236725c7 | in-use |              |  10  |     None    |   true   | f8f7d003-60f7-418f-aeab-1be34a3ccdc9 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+


MariaDB [nova]> 
MariaDB [nova]> select * from block_device_mapping where instance_uuid = 'f8f7d003-60f7-418f-aeab-1be34a3ccdc9' \G;
*************************** 1. row ***************************
           created_at: 2016-03-01 18:32:37
           updated_at: 2016-03-01 18:32:41
           deleted_at: NULL
                   id: 5
          device_name: /dev/vda
delete_on_termination: 0
          snapshot_id: NULL
            volume_id: 3bc99f74-cbbe-4460-b1a7-1483236725c7
          volume_size: 10
            no_device: NULL
      connection_info: {"driver_volume_type": "iscsi", "serial": "3bc99f74-cbbe-4460-b1a7-1483236725c7", "data": {"access_mode": "rw", "target_discovered": false, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7", "target_portal": "10.8.0.50:3260", "volume_id": "3bc99f74-cbbe-4460-b1a7-1483236725c7", "target_lun": 0, "device_path": "/dev/disk/by-path/ip-10.8.0.50:3260-iscsi-iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7-lun-0", "auth_password": "xKeK9PAsPQh3ChmzwpZ7", "auth_username": "XJdjVNpoHgYLQhD3LL2Y", "auth_method": "CHAP"}}
        instance_uuid: f8f7d003-60f7-418f-aeab-1be34a3ccdc9
              deleted: 0
          source_type: image
     destination_type: volume
         guest_format: NULL
          device_type: disk
             disk_bus: virtio
           boot_index: 0
             image_id: a73e61d5-633e-4f3e-a728-c1c6b4dc946f
1 row in set (0.00 sec)


MariaDB [nova]> use cinder;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [cinder]> 
MariaDB [cinder]> select * from volumes where id='3bc99f74-cbbe-4460-b1a7-1483236725c7' \G;
*************************** 1. row ***************************
         created_at: 2016-03-01 18:32:38
         updated_at: 2016-03-01 18:32:40
         deleted_at: NULL
            deleted: 0
                 id: 3bc99f74-cbbe-4460-b1a7-1483236725c7
             ec2_id: NULL
            user_id: d45021949119414b97b55a94f7737ea2
         project_id: aa8e7fc282f849fc9099e4b3910fd291
               host: rhos-compute-node-02.lab.eng.rdu2.redhat.com
               size: 10
  availability_zone: nova
      instance_uuid: f8f7d003-60f7-418f-aeab-1be34a3ccdc9
         mountpoint: /dev/vda
        attach_time: 2016-03-01T18:32:40.720727
             status: in-use
      attach_status: attached
       scheduled_at: 2016-03-01 18:32:38
        launched_at: 2016-03-01 18:32:39
      terminated_at: NULL
       display_name: 
display_description: 
  provider_location: 10.8.0.50:3260,iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7 iqn.2010-10.org.openstack:volume-3bc99f74-cbbe-4460-b1a7-1483236725c7 0
      provider_auth: CHAP XJdjVNpoHgYLQhD3LL2Y xKeK9PAsPQh3ChmzwpZ7
        snapshot_id: NULL
     volume_type_id: NULL
       source_volid: NULL
           bootable: 1
      attached_host: NULL
  provider_geometry: NULL
           _name_id: NULL
  encryption_key_id: NULL
   migration_status: NULL
1 row in set (0.00 sec)




*****************************************
Deleteing volume upon removing instance
*****************************************

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova boot --flavor 2 --block-device-mapping vda=d5b65335-d32c-41fd-8c96-0bc161ff9d23:::1 vm02+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          | nova                                             |
| OS-EXT-SRV-ATTR:host                 | -                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000008                                |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | mLZawx47QYbE                                     |
| config_drive                         |                                                  |
| created                              | 2016-03-01T19:49:29Z                             |
| flavor                               | m1.small (2)                                     |
| hostId                               |                                                  |
| id                                   | a6b2d9a8-a673-445a-80c9-02b9dabfbc35             |
| image                                | Attempt to boot from volume - no image supplied  |
| key_name                             | -                                                |
| metadata                             | {}                                               |
| name                                 | vm02                                             |
| os-extended-volumes:volumes_attached | [{"id": "d5b65335-d32c-41fd-8c96-0bc161ff9d23"}] |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | aa8e7fc282f849fc9099e4b3910fd291                 |
| updated                              | 2016-03-01T19:49:29Z                             |
| user_id                              | d45021949119414b97b55a94f7737ea2                 |
+--------------------------------------+--------------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| a6b2d9a8-a673-445a-80c9-02b9dabfbc35 | vm02 | ACTIVE | -          | Running     | public=172.24.4.234 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| d5b65335-d32c-41fd-8c96-0bc161ff9d23 | in-use |   bootvol    |  2   |     None    |   true   | a6b2d9a8-a673-445a-80c9-02b9dabfbc35 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova delete vm02
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# cinder list
+----+--------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+----+--------+--------------+------+-------------+----------+-------------+
+----+--------+--------------+------+-------------+----------+-------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]#

Comment 14 errata-xmlrpc 2016-03-08 17:07:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0361.html