Bug 1304525

Summary: Volume appears as attached although it doesn't show up in the guest
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED WORKSFORME QA Contact: nlevinki <nlevinki>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: athomas, berrange, dasmith, eglynn, eharney, kchamart, lyarwood, ndipanov, ohochman, sasha, sbauza, sferdjao, sgordon, sgotliv, vromanso, yeylon
Target Milestone: z5Keywords: ZStream
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-23 11:04:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
cinder and nova logs from a controller
none
cinder and nova logs from a controller
none
nova-compute log crom a compute none

Description Alexander Chuzhoy 2016-02-03 22:32:56 UTC 
openstack-cinder:  cinder volume isn't attached to the instance on first attempt although shows as attached.

Environment:
python-cinder-7.0.0-2.el7ost.noarch
openstack-puppet-modules-7.0.1-2.el7ost.noarch
python-cinderclient-1.4.0-1.el7ost.noarch
openstack-cinder-7.0.0-2.el7ost.noarch



Step to reproduce:
create a cinder volume and attach to the instance.

Result:

[stack@instack ~]$ cinder list --all-tenant
+--------------------------------------+----------------------------------+--------+------------------+------+------+-------------+----------+-------------+--------------------------------------+
|                  ID                  |            Tenant ID             | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach |             Attached to              |
+--------------------------------------+----------------------------------+--------+------------------+------+------+-------------+----------+-------------+--------------------------------------+
| 7f5c921c-2363-4b63-94de-c5767d949519 | fc27286b9beb4742b2cd26b9b1437429 | in-use |        -         |  -   |  1   |      -      |  false   |    False    | ec916830-ac0b-4783-a8a6-e6aa00bae814 |
+--------------------------------------+----------------------------------+--------+------------------+------+------+-------------+----------+-------------+--------------------------------------+
[stack@instack ~]$ ssh cirros.200.101                             
# fdisk -l
Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065     2088449     1036192+  83  Linux
# $ Connection to 192.168.200.101 closed.


#Note ^^, no vdb

]$
[stack@instack ~]$ nova volume-detach ec916830-ac0b-4783-a8a6-e6aa00bae814 7f5c921c-2363-4b63-94de-c5767d949519

[stack@instack ~]$ nova volume-attach ec916830-ac0b-4783-a8a6-e6aa00bae814 7f5c921c-2363-4b63-94de-c5767d949519

+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 7f5c921c-2363-4b63-94de-c5767d949519 |
| serverId | ec916830-ac0b-4783-a8a6-e6aa00bae814 |
| volumeId | 7f5c921c-2363-4b63-94de-c5767d949519 |
+----------+--------------------------------------+
[stack@instack ~]$ ssh cirros.200.101
cirros.200.101's password:
$ sudo fdisk -l

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065     2088449     1036192+  83  Linux

Disk /dev/vdb: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesn't contain a valid partition table


# Note ^^ the vdb appeared.



Expected result:
The attached volume should be seen on the first attempt.
Comment 2 Sergey Gotliv 2016-02-04 09:04:57 UTC 
Note: your Cinder version is a bit outdated, we are going to GA with 	
openstack-cinder-7.0.1..... yours is openstack-cinder-7.0.0-2.el7ost.noarch.

We need both Nova and Cinder logs to understand what happened. Did Cinder fail to create a volume or provide a connection information or maybe Nova failed to attach this volume to the running instance.
Comment 3 Alexander Chuzhoy 2016-02-04 20:17:29 UTC 
Created attachment 1121199 [details]
cinder and nova logs from a controller
Comment 4 Alexander Chuzhoy 2016-02-04 20:24:19 UTC 
Created attachment 1121202 [details]
cinder and nova logs from a controller
Comment 5 Alexander Chuzhoy 2016-02-04 20:42:07 UTC 
Created attachment 1121203 [details]
nova-compute log crom a compute
Comment 6 Alexander Chuzhoy 2016-02-05 22:14:07 UTC 
Additional info:

Note the VM ID:
[stack@instack ~]$ nova list --all-tenant
+--------------------------------------+--------+----------------------------------+--------+------------+-------------+--------------------------------------+
| ID                                   | Name   | Tenant ID                        | Status | Task State | Power State | Networks                             |
+--------------------------------------+--------+----------------------------------+--------+------------+-------------+--------------------------------------+
| c8345b30-d0e3-43f3-8845-4a50f67e8aa7 | nisim1 | ef92657639184ad589a20aa0d239846f | ACTIVE | -          | Running     | tenant=192.168.32.5, 192.168.200.101 |
+--------------------------------------+--------+----------------------------------+--------+------------+-------------+--------------------------------------+



Note that the cinder volume appears as attached to that VM:

[stack@instack ~]$ cinder list --all-tenant
+--------------------------------------+----------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |            Tenant ID             | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+----------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 1f588c45-b8b9-4588-8da0-d36cd3b0715d | ef92657639184ad589a20aa0d239846f | in-use |      -       |  1   |      -      |  false   | c8345b30-d0e3-43f3-8845-4a50f67e8aa7 |
+--------------------------------------+----------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+

Yet - when we check the disks on that VM - it has only one.

[stack@instack ~]$ ssh cirros.200.101 "sudo /sbin/fdisk -l"

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065     2088449     1036192+  83  Linux
Comment 11 Alexander Chuzhoy 2016-02-11 20:56:52 UTC 
Reproduced:
Environment:
python-nova-2015.1.0-16.el7ost.noarch
openstack-nova-console-2015.1.0-16.el7ost.noarch
python-cinderclient-1.2.1-1.el7ost.noarch
openstack-nova-compute-2015.1.0-16.el7ost.noarch
openstack-nova-novncproxy-2015.1.0-16.el7ost.noarch
openstack-nova-scheduler-2015.1.0-16.el7ost.noarch
python-cinder-2015.1.0-3.el7ost.noarch
openstack-nova-cert-2015.1.0-16.el7ost.noarch
openstack-cinder-2015.1.0-3.el7ost.noarch
openstack-nova-api-2015.1.0-16.el7ost.noarch
openstack-nova-common-2015.1.0-16.el7ost.noarch
openstack-nova-conductor-2015.1.0-16.el7ost.noarch
python-novaclient-2.23.0-1.el7ost.noarch




[stack@instack ~]$ nova show e4d77817-5fdc-437d-ba47-b4562cb184de                                                                                              
+--------------------------------------+----------------------------------------------------------+                                                            
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | overcloud-compute-0.localdomain                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | overcloud-compute-0.localdomain                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000002                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2016-02-11T20:34:14.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2016-02-11T20:33:36Z                                     |
| flavor                               | m1.tiny (1)                                              |
| hostId                               | 21f62e0206f093421eae52d951d6470242ab4e8f62ed5387673b06e0 |
| id                                   | e4d77817-5fdc-437d-ba47-b4562cb184de                     |
| image                                | cirros (e85261ed-7188-49f5-b29f-109d796564ac)            |
| key_name                             | oskey                                                    |
| metadata                             | {}                                                       |
| name                                 | nisim1                                                   |
| os-extended-volumes:volumes_attached | [{"id": "06902a7b-b904-4709-b3e0-25974ae29624"}]         |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tenant network                       | 192.168.32.5, 192.168.200.101                            |
| tenant_id                            | f3bccd5571df4d508736dd4754813079                         |
| updated                              | 2016-02-11T20:34:14Z                                     |
| user_id                              | 981300fa16014e488d28dcfab5fa666a                         |
+--------------------------------------+----------------------------------------------------------+




[stack@instack ~]$ cinder list --all-tenant
+--------------------------------------+----------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |            Tenant ID             | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+----------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 06902a7b-b904-4709-b3e0-25974ae29624 | f3bccd5571df4d508736dd4754813079 | in-use |      -       |  1   |      -      |  false   | e4d77817-5fdc-437d-ba47-b4562cb184de |



nothing on the instance.