Bug 929194 - Cannot boot an instance from volume
Summary: Cannot boot an instance from volume
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 2.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 2.1
Assignee: Nikola Dipanov
QA Contact: Kashyap Chamarthy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-03-29 13:08 UTC by Nikola Dipanov
Modified: 2020-08-13 08:07 UTC (History)
7 users (show)

Fixed In Version: openstack-nova-2012.2.3-9.el6ost.src.rpm
Doc Type: Bug Fix
Doc Text:
Updates to the libvirt driver for Nova resulted in an invalid key error in the device mapping code. This error prevented the booting of instances from volume storage. The device mapping code has been updated to prevent the key error from occurring. Booting instances from volume storage now works correctly.
Clone Of:
Environment:
Last Closed: 2013-04-08 23:42:48 UTC
Target Upstream Version:


Attachments (Terms of Use)
test using boot from volume (7.20 KB, text/plain)
2013-04-01 11:22 UTC, Kashyap Chamarthy
no flags Details
instance boot from volume -- successful (15.24 KB, text/plain)
2013-04-04 12:36 UTC, Kashyap Chamarthy
no flags Details
succesful investigative trace of networking inside cirros image (10.12 KB, text/plain)
2013-04-05 15:28 UTC, Kashyap Chamarthy
no flags Details


Links
System ID Priority Status Summary Last Updated
Launchpad 1118829 None None None Never
Red Hat Product Errata RHBA-2013:0718 normal SHIPPED_LIVE openstack-nova bug fix advisory 2013-04-09 03:41:21 UTC

Description Nikola Dipanov 2013-03-29 13:08:02 UTC
Description of problem:
when trying to boot an instance from volume as such (I have prepared a bootable volume in advance as per https://fedoraproject.org/wiki/QA:Testcase_Nova_Create_Bootable_Volume):

[root@rhel-test ~(keystone_admin)]$ nova boot --flavor 1 --image 1973f799-1c02-455c-a056-ffc0e8878c95 
--block_device_mapping vda=4ce6844d-18e3-45ea-9f61-2119dc8b000e:::0 --key_name nova_key volume_backed 

The instance fails to boot and the following trace is found in the compute logs:

<snip/>
2013-03-29 12:35:53 ERROR nova.compute.manager [req-d9a42882-218a-4dc9-9ff6-c370af670695 24c24e095e4b4ec0951ffe3c58cc23b2 e312a7a4cc6542ab9ff8dab6155c68ca] [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585] Instance failed to spawn
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585] Traceback (most recent call last):
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 756, in _spawn
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]     block_device_info)
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]     temp_level, payload)
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]     self.gen.next()
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]     return f(*args, **kw)
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1098, in spawn
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]     block_device_info=block_device_info)
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1856, in to_xml
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]     disk_info, rescue, block_device_info)
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1708, in get_guest_config
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]     {'root_device_name': '/dev/' + disk_mapping['disk']['dev']})
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585] KeyError: 'disk'
2013-03-29 12:35:53 15950 TRACE nova.compute.manager [instance: a771b7dd-3fb6-4c13-87b0-c0af37fcc585]


Version-Release number of selected component (if applicable):


How reproducible:

Allways

Steps to Reproduce:

Described above
  
Actual results:

Fails

Expected results:

Instance should boot just fine

Additional info:

There seems to be an upstream fix for the bug: https://review.openstack.org/#/c/21473/

Comment 3 Kashyap Chamarthy 2013-04-01 11:22:29 UTC
Created attachment 730248 [details]
test using boot from volume

Note, although, I seem to get the IP address for the builder instance, I can't ping it, consequently, can't ssh.

This is reproducible with the below two types of nova networking schemes.

(1)
====
.
.
fixed_range=10.65.207.48/29
floating_range=10.65.207.56/29
flat_interface = eth2
public_interface = eth2
flat_network_bridge=demonetbr0
.
.
====


(2) 
====
.
.
fixed_range=192.168.32.0/22
floating_range=10.3.4.0/22
flat_interface=eth1
public_interface=eth2
flat_network_bridge=br100
.
.
====

Comment 4 Kashyap Chamarthy 2013-04-01 13:26:02 UTC
Some more debugging info, after discussion with Bob Kukura.

-> Get the console-log for the 'builder' cirros image:
#----------#
[tuser1@interceptor export(keystone_user1)]$ nova console-log builder > LOG.txt
#----------#
[tuser1@interceptor export(keystone_user1)]$ egrep -i 'dhcp|lease' LOG.txt 
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.228969] pci 0000:00:01.0: PIIX3: Enabling Passive Release
udhcpc (v1.18.5) started
Lease of 10.65.207.136 obtained, lease time 86400
#----------#


-> List security groups info:
#----------#
[tuser1@interceptor ~(keystone_admin)]$ nova secgroup-list
+---------+-------------+
| Name    | Description |
+---------+-------------+
| default | default     |
+---------+-------------+

[tuser1@interceptor ~(keystone_admin)]$  nova secgroup-list-rules default

[tuser1@interceptor ~(keystone_admin)]$ 
#----------#


-> info about the cirros instance:
#----------#
$ nova show builder
+------------------------+----------------------------------------------------------+
| Property               | Value                                                    |
+------------------------+----------------------------------------------------------+
| status                 | ACTIVE                                                   |
| updated                | 2013-04-01T03:09:14Z                                     |
| OS-EXT-STS:task_state  | None                                                     |
| key_name               | oskey3                                                   |
| image                  | cirros-ami (f29ecbe5-5d02-4cc8-9768-cd5a5fb0d28f)        |
| hostId                 | ea9a88283bae81eb0a5621a827995c8485a5fa60e13ca4be47ce90d6 |
| OS-EXT-STS:vm_state    | active                                                   |
| flavor                 | m1.tiny (1)                                              |
| id                     | ec77afe2-2a66-45aa-87c7-0330b7e06865                     |
| security_groups        | [{u'name': u'default'}]                                  |
| user_id                | 320ce46de7e24a75a7ff8906d7355ff7                         |
| name                   | builder                                                  |
| created                | 2013-04-01T03:08:53Z                                     |
| tenant_id              | 57ff99aae24b4035b52177a722c4091f                         |
| OS-DCF:diskConfig      | MANUAL                                                   |
| accessIPv4             |                                                          |
| accessIPv6             |                                                          |
| net1 network           | 10.65.207.54                                             |
| progress               | 0                                                        |
| OS-EXT-STS:power_state | 1                                                        |
| metadata               | {}                                                       |
| config_drive           |                                                          |
+------------------------+----------------------------------------------------------+
#----------#

(NOTE: this on the first networking scheme mentioned in comment #3)

Comment 6 Kashyap Chamarthy 2013-04-04 12:29:37 UTC
Verification Info:

1] Version info:
#-------------#
$ cat /etc/redhat-release ; arch
Red Hat Enterprise Linux Server release 6.4 (Santiago)
x86_64

$ rpm -q openstack-nova
openstack-nova-2012.2.3-9.el6ost.noarch

$ rpm -q openstack-nova --changelog | grep -i 929194
- Fixes boot from volume libvirt driver bug #929194
#-------------#


2] Ensure the fix is in:
#-------------#
$ grep root_device_name /usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py
            root_device_name = "/dev/" + disk_mapping['root']['dev']
            root_device_name = None
        if root_device_name:
                {'root_device_name': root_device_name})
            guest.os_root = root_device_name
                guest.os_root = root_device_name
                                            root_device_name)
                                        root_device_name)
#-------------#

Comment 7 Kashyap Chamarthy 2013-04-04 12:36:50 UTC
Created attachment 731578 [details]
instance boot from volume -- successful

Just noting the boot from volume detail.

[tuser1@interceptor ~(keystone_user1)]$  nova boot --flavor 1 --image 1e6292f9-82bd-4cdb-969e-c863cb1c6692 --block_device_mapping vda=$VOLUME_ID:::0 --key_name oskey volume_backed1
+------------------------+--------------------------------------+
| Property               | Value                                |
+------------------------+--------------------------------------+
| status                 | BUILD                                |
| updated                | 2013-04-04T12:18:26Z                 |
| OS-EXT-STS:task_state  | scheduling                           |
| key_name               | oskey                                |
| image                  | fedora-17                            |
| hostId                 |                                      |
| OS-EXT-STS:vm_state    | building                             |
| flavor                 | m1.tiny                              |
| id                     | 47e6585a-ed92-4084-88f2-50153cf10218 |
| security_groups        | [{u'name': u'default'}]              |
| user_id                | 320ce46de7e24a75a7ff8906d7355ff7     |
| name                   | volume_backed1                       |
| adminPass              | qve3t5pH484k                         |
| tenant_id              | 57ff99aae24b4035b52177a722c4091f     |
| created                | 2013-04-04T12:18:26Z                 |
| OS-DCF:diskConfig      | MANUAL                               |
| accessIPv4             |                                      |
| accessIPv6             |                                      |
| progress               | 0                                    |
| OS-EXT-STS:power_state | 0                                    |
| metadata               | {}                                   |
| config_drive           |                                      |
+------------------------+--------------------------------------+
[tuser1@interceptor ~(keystone_user1)]$

Comment 8 Kashyap Chamarthy 2013-04-04 17:46:47 UTC
Another test, with a different setup

Short: 
  - The instance booted from the volume isn't reachable via ping (icmp)
  - And for some reason, the device file -- (created by cinder volume) -- doesn't appear to 'exist' 
=========================
[root@puma06 ~(keystone_admin)]$ file /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1
/dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1: symbolic link to `../../sdb'
=========================
[root@puma06 ~(keystone_admin)]$ file ../../sdb
../../sdb: cannot open `../../sdb' (No such file or directory)
[root@puma06 ~(keystone_admin)]$
=========================


sequence of commands invoked for the test:
=========================
[root@puma06 ~(keystone_admin)]$ cinder create --display_name=bootable_volume 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|      created_at     |      2013-04-04T13:55:41.764308      |
| display_description |                 None                 |
|     display_name    |           bootable_volume            |
|          id         | 2c370395-7f59-4c89-b312-ba35dbb986c0 |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |   Display Name  | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | available | bootable_volume |  1   |     None    |          |             |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
[root@puma06 ~(keystone_admin)]$ 

=========================
[root@puma06 ~(keystone_admin)]$ VOLUME_ID=2c370395-7f59-4c89-b312-ba35dbb986c0
[root@puma06 ~(keystone_admin)]$ echo $VOLUME_ID
2c370395-7f59-4c89-b312-ba35dbb986c0
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ glance image-list | grep fedora-17
| e1b71961-d66d-4315-8e83-32aa1bd44f3f | fedora-17 | qcow2       | bare             | 251985920 | active |
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ nova boot --image e1b71961-d66d-4315-8e83-32aa1bd44f3f --flavor 1 --key_name oskey f17-builder
+-------------------------------------+--------------------------------------+
| Property                            | Value                                |
+-------------------------------------+--------------------------------------+
| status                              | BUILD                                |
| updated                             | 2013-04-04T14:17:52Z                 |
| OS-EXT-STS:task_state               | scheduling                           |
| OS-EXT-SRV-ATTR:host                | None                                 |
| key_name                            | oskey                                |
| image                               | fedora-17                            |
| hostId                              |                                      |
| OS-EXT-STS:vm_state                 | building                             |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000005                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                 |
| flavor                              | m1.tiny                              |
| id                                  | 220d6e57-f7ad-4256-a861-7855ffc6788f |
| security_groups                     | [{u'name': u'default'}]              |
| user_id                             | b385f4065f8c40f59fcd69d67c403ae6     |
| name                                | f17-builder                          |
| adminPass                           | ihbmLpLrvp3n                         |
| tenant_id                           | d2218b6611e9423fb0027acb1c389317     |
| created                             | 2013-04-04T14:17:52Z                 |
| OS-DCF:diskConfig                   | MANUAL                               |
| accessIPv4                          |                                      |
| accessIPv6                          |                                      |
| progress                            | 0                                    |
| OS-EXT-STS:power_state              | 0                                    |
| metadata                            | {}                                   |
| config_drive                        |                                      |
+-------------------------------------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17-builder
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder | ACTIVE | novanetwork=192.168.32.4 |
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ ssh -i oskey.priv 192.168.32.4
The authenticity of host '192.168.32.4 (192.168.32.4)' can't be established.
RSA key fingerprint is cd:14:a4:c3:e4:a8:15:5a:1c:80:15:15:52:8b:e0:87.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.32.4' (RSA) to the list of known hosts.
reverse mapping checking getaddrinfo for porkchop.test.redhat.com [192.168.32.4] failed - POSSIBLE BREAK-IN ATTEMPT!
[root@f17-builder ~]# 
=========================
[root@f17-builder ~]# yum install wget qemu-img -y
=========================
[root@f17-builder ~]# qemu-img convert cirros-0.3.0-x86_64-disk.img  -O raw cirros-0.3.0-x86_64-disk.img.raw
[root@f17-builder ~]#
=========================
[root@f17-builder ~]# file cirros-0.3.0-x86_64-disk.img
cirros-0.3.0-x86_64-disk.img: QEMU QCOW Image (v2), 41126400 bytes
=========================
[root@f17-builder ~]# file cirros-0.3.0-x86_64-disk.img.raw
cirros-0.3.0-x86_64-disk.img.raw: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, stage2 address 0x2000, stage2 segment 0x200; partition 1: ID=0x83, active, starthead 0, startsector 16065, 64260 sectors, code offset 0x48
[root@f17-builder ~]#
=========================
[root@f17-builder ~]# logout
Connection to 192.168.32.4 closed.
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17-builder
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder | ACTIVE | novanetwork=192.168.32.4 |
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |   Display Name  | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | available | bootable_volume |  1   |     None    |          |             |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ nova volume-attach f17-builder 2c370395-7f59-4c89-b312-ba35dbb986c0 /dev/vdb
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| serverId | 220d6e57-f7ad-4256-a861-7855ffc6788f |
| id       | 2c370395-7f59-4c89-b312-ba35dbb986c0 |
| volumeId | 2c370395-7f59-4c89-b312-ba35dbb986c0 |
+----------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status |   Display Name  | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | in-use | bootable_volume |  1   |     None    |          | 220d6e57-f7ad-4256-a861-7855ffc6788f |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ ssh -i oskey.priv 192.168.32.4
reverse mapping checking getaddrinfo for porkchop.test.redhat.com [192.168.32.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Last login: Thu Apr  4 10:31:11 2013 from 192.168.32.1
[root@f17-builder ~]# cat /proc/partitions 
major minor  #blocks  name

 252        0   10241024 vda
 252        1        960 vda1
 252        2    9765625 vda2
 252       16    1048576 vdb
[root@f17-builder ~]# 
=========================
[root@f17-builder ~]# logout
Connection to 192.168.32.4 closed.
=========================
[root@puma06 ~(keystone_admin)]$ nova volume-detach f17-builder 2c370395-7f59-4c89-b312-ba35dbb986c0
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |   Display Name  | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | available | bootable_volume |  1   |     None    |          |             |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ nova boot --flavor 1 --image e1b71961-d66d-4315-8e83-32aa1bd44f3f --block_device_mapping vda=2c370395-7f59-4c89-b312-ba35dbb986c0:::0 --key_name oskey f17_volume_backed
+-------------------------------------+--------------------------------------+
| Property                            | Value                                |
+-------------------------------------+--------------------------------------+
| status                              | BUILD                                |
| updated                             | 2013-04-04T15:01:40Z                 |
| OS-EXT-STS:task_state               | scheduling                           |
| OS-EXT-SRV-ATTR:host                | None                                 |
| key_name                            | oskey                                |
| image                               | fedora-17                            |
| hostId                              |                                      |
| OS-EXT-STS:vm_state                 | building                             |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000006                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                 |
| flavor                              | m1.tiny                              |
| id                                  | d4d42ec6-8a10-4ddb-8962-d6fa5875f8c5 |
| security_groups                     | [{u'name': u'default'}]              |
| user_id                             | b385f4065f8c40f59fcd69d67c403ae6     |
| name                                | f17_volume_backed                    |
| adminPass                           | U4cUAR3nnMQ4                         |
| tenant_id                           | d2218b6611e9423fb0027acb1c389317     |
| created                             | 2013-04-04T15:01:40Z                 |
| OS-DCF:diskConfig                   | MANUAL                               |
| accessIPv4                          |                                      |
| accessIPv6                          |                                      |
| progress                            | 0                                    |
| OS-EXT-STS:power_state              | 0                                    |
| metadata                            | {}                                   |
| config_drive                        |                                      |
+-------------------------------------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder       | ACTIVE | novanetwork=192.168.32.4 |
| d4d42ec6-8a10-4ddb-8962-d6fa5875f8c5 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.5 |
[root@puma06 ~(keystone_admin)]$ 
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder       | ACTIVE | novanetwork=192.168.32.4 |
| d4d42ec6-8a10-4ddb-8962-d6fa5875f8c5 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.5 |
[root@puma06 ~(keystone_admin)]$ ping 192.168.32.5
PING 192.168.32.5 (192.168.32.5) 56(84) bytes of data.
From 192.168.32.1 icmp_seq=1 Destination Host Unreachable
From 192.168.32.1 icmp_seq=2 Destination Host Unreachable
From 192.168.32.1 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.32.5 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4607ms
pipe 3
=========================
[root@puma06 ~(keystone_admin)]$ file /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1
/dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1: symbolic link to `../../sdb'
[root@puma06 ~(keystone_admin)]$ file ../../sdb
../../sdb: cannot open `../../sdb' (No such file or directory)
[root@puma06 ~(keystone_admin)]$
=========================

Comment 9 Daniel Berrangé 2013-04-05 08:58:47 UTC
(In reply to comment #8)
> Another test, with a different setup
> 
> Short: 
>   - The instance booted from the volume isn't reachable via ping (icmp)
>   - And for some reason, the device file -- (created by cinder volume) --
> doesn't appear to 'exist' 
> =========================
> [root@puma06 ~(keystone_admin)]$ file
> /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:
> volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1
> /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:
> volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1: symbolic link to
> `../../sdb'
> =========================
> [root@puma06 ~(keystone_admin)]$ file ../../sdb
> ../../sdb: cannot open `../../sdb' (No such file or directory)
> [root@puma06 ~(keystone_admin)]$
> =========================

Your test scenario here is broken. The symlink is relative to the original file's directory, but you are checking it from your home directory, so of course you can't locate it from there. You need todo

 $ cd dev/disk/by-path/
 $ file ../../sdb

Comment 10 Kashyap Chamarthy 2013-04-05 13:28:38 UTC
> 
> Your test scenario here is broken. The symlink is relative to the original
> file's directory, but you are checking it from your home directory, so of
> course you can't locate it from there. You need todo

Indeed. That's a human error on my side. However, the issue - the volume booted instance doesn't appear to get network. Trying to drill this down further.
> 
>  $ cd dev/disk/by-path/
>  $ file ../../sdb

[root@puma06 dev(keystone_admin)]$ cd /dev/disk/by-path/
[root@puma06 by-path(keystone_admin)]$
[root@puma06 by-path(keystone_admin)]$ file ../../sdb
../../sdb: block special

Thanks for your comment.

Comment 11 Kashyap Chamarthy 2013-04-05 15:09:10 UTC
VERIFIED.

(verification procedure is noted in the previous comments)

** Short **

-> Add this iptables rule, so that we get dhcp leases:
==================
[root@puma06 ~(keystone_admin)]$ iptables -A POSTROUTING -t mangle -p udp --dport 68 -j CHECKSUM --checksum-fill
[root@puma06 ~(keystone_admin)]$ 
==================

-> list the nova instances
==================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder       | ACTIVE  | novanetwork=192.168.32.4 |
| f1169841-5e67-45d9-85c4-0d44bd3bec90 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.6 |
[root@puma06 ~(keystone_admin)]$ 
==================

-> ssh into the volume_backed instance
==================
[root@puma06 ~(keystone_admin)]$ ssh -i oskey.priv -o UserKnownHostsFile=/dev/null cirros@192.168.32.6
The authenticity of host '192.168.32.6 (192.168.32.6)' can't be established.
RSA key fingerprint is a1:bc:0b:27:fb:29:ca:04:ac:ac:1c:06:c5:ac:b4:9a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.32.6' (RSA) to the list of known hosts.
$ pwd
/home/cirros
$ ls
$ ifconfig
eth0      Link encap:Ethernet  HWaddr FA:16:3E:0E:4A:23  
          inet addr:192.168.32.6  Bcast:192.168.35.255  Mask:255.255.252.0
          inet6 addr: fe80::f816:3eff:fe0e:4a23/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:202 errors:0 dropped:0 overruns:0 frame:0
          TX packets:154 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:28482 (27.8 KiB)  TX bytes:21801 (21.2 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

$ 
==================

The above confirms IP address for the cirros image.

Comment 14 errata-xmlrpc 2013-04-08 23:42:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0718.html


Note You need to log in before you can comment on or make changes to this bug.