Bug 929194
| Summary: | Cannot boot an instance from volume | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Nikola Dipanov <ndipanov> | ||||||||
| Component: | openstack-nova | Assignee: | Nikola Dipanov <ndipanov> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Kashyap Chamarthy <kchamart> | ||||||||
| Severity: | urgent | Docs Contact: | |||||||||
| Priority: | urgent | ||||||||||
| Version: | 2.1 | CC: | eglynn, jhenner, kchamart, ndipanov, sgordon, ykaul | ||||||||
| Target Milestone: | rc | ||||||||||
| Target Release: | 2.1 | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | openstack-nova-2012.2.3-9.el6ost.src.rpm | Doc Type: | Bug Fix | ||||||||
| Doc Text: |
Updates to the libvirt driver for Nova resulted in an invalid key error in the device mapping code. This error prevented the booting of instances from volume storage. The device mapping code has been updated to prevent the key error from occurring. Booting instances from volume storage now works correctly.
|
Story Points: | --- | ||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2013-04-08 23:42:48 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Nikola Dipanov
2013-03-29 13:08:02 UTC
Created attachment 730248 [details]
test using boot from volume
Note, although, I seem to get the IP address for the builder instance, I can't ping it, consequently, can't ssh.
This is reproducible with the below two types of nova networking schemes.
(1)
====
.
.
fixed_range=10.65.207.48/29
floating_range=10.65.207.56/29
flat_interface = eth2
public_interface = eth2
flat_network_bridge=demonetbr0
.
.
====
(2)
====
.
.
fixed_range=192.168.32.0/22
floating_range=10.3.4.0/22
flat_interface=eth1
public_interface=eth2
flat_network_bridge=br100
.
.
====
Some more debugging info, after discussion with Bob Kukura.
-> Get the console-log for the 'builder' cirros image:
#----------#
[tuser1@interceptor export(keystone_user1)]$ nova console-log builder > LOG.txt
#----------#
[tuser1@interceptor export(keystone_user1)]$ egrep -i 'dhcp|lease' LOG.txt
[ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[ 0.228969] pci 0000:00:01.0: PIIX3: Enabling Passive Release
udhcpc (v1.18.5) started
Lease of 10.65.207.136 obtained, lease time 86400
#----------#
-> List security groups info:
#----------#
[tuser1@interceptor ~(keystone_admin)]$ nova secgroup-list
+---------+-------------+
| Name | Description |
+---------+-------------+
| default | default |
+---------+-------------+
[tuser1@interceptor ~(keystone_admin)]$ nova secgroup-list-rules default
[tuser1@interceptor ~(keystone_admin)]$
#----------#
-> info about the cirros instance:
#----------#
$ nova show builder
+------------------------+----------------------------------------------------------+
| Property | Value |
+------------------------+----------------------------------------------------------+
| status | ACTIVE |
| updated | 2013-04-01T03:09:14Z |
| OS-EXT-STS:task_state | None |
| key_name | oskey3 |
| image | cirros-ami (f29ecbe5-5d02-4cc8-9768-cd5a5fb0d28f) |
| hostId | ea9a88283bae81eb0a5621a827995c8485a5fa60e13ca4be47ce90d6 |
| OS-EXT-STS:vm_state | active |
| flavor | m1.tiny (1) |
| id | ec77afe2-2a66-45aa-87c7-0330b7e06865 |
| security_groups | [{u'name': u'default'}] |
| user_id | 320ce46de7e24a75a7ff8906d7355ff7 |
| name | builder |
| created | 2013-04-01T03:08:53Z |
| tenant_id | 57ff99aae24b4035b52177a722c4091f |
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| net1 network | 10.65.207.54 |
| progress | 0 |
| OS-EXT-STS:power_state | 1 |
| metadata | {} |
| config_drive | |
+------------------------+----------------------------------------------------------+
#----------#
(NOTE: this on the first networking scheme mentioned in comment #3)
Verification Info: 1] Version info: #-------------# $ cat /etc/redhat-release ; arch Red Hat Enterprise Linux Server release 6.4 (Santiago) x86_64 $ rpm -q openstack-nova openstack-nova-2012.2.3-9.el6ost.noarch $ rpm -q openstack-nova --changelog | grep -i 929194 - Fixes boot from volume libvirt driver bug #929194 #-------------# 2] Ensure the fix is in: #-------------# $ grep root_device_name /usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py root_device_name = "/dev/" + disk_mapping['root']['dev'] root_device_name = None if root_device_name: {'root_device_name': root_device_name}) guest.os_root = root_device_name guest.os_root = root_device_name root_device_name) root_device_name) #-------------# Created attachment 731578 [details]
instance boot from volume -- successful
Just noting the boot from volume detail.
[tuser1@interceptor ~(keystone_user1)]$ nova boot --flavor 1 --image 1e6292f9-82bd-4cdb-969e-c863cb1c6692 --block_device_mapping vda=$VOLUME_ID:::0 --key_name oskey volume_backed1
+------------------------+--------------------------------------+
| Property | Value |
+------------------------+--------------------------------------+
| status | BUILD |
| updated | 2013-04-04T12:18:26Z |
| OS-EXT-STS:task_state | scheduling |
| key_name | oskey |
| image | fedora-17 |
| hostId | |
| OS-EXT-STS:vm_state | building |
| flavor | m1.tiny |
| id | 47e6585a-ed92-4084-88f2-50153cf10218 |
| security_groups | [{u'name': u'default'}] |
| user_id | 320ce46de7e24a75a7ff8906d7355ff7 |
| name | volume_backed1 |
| adminPass | qve3t5pH484k |
| tenant_id | 57ff99aae24b4035b52177a722c4091f |
| created | 2013-04-04T12:18:26Z |
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| metadata | {} |
| config_drive | |
+------------------------+--------------------------------------+
[tuser1@interceptor ~(keystone_user1)]$
Another test, with a different setup
Short:
- The instance booted from the volume isn't reachable via ping (icmp)
- And for some reason, the device file -- (created by cinder volume) -- doesn't appear to 'exist'
=========================
[root@puma06 ~(keystone_admin)]$ file /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1
/dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1: symbolic link to `../../sdb'
=========================
[root@puma06 ~(keystone_admin)]$ file ../../sdb
../../sdb: cannot open `../../sdb' (No such file or directory)
[root@puma06 ~(keystone_admin)]$
=========================
sequence of commands invoked for the test:
=========================
[root@puma06 ~(keystone_admin)]$ cinder create --display_name=bootable_volume 1
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| created_at | 2013-04-04T13:55:41.764308 |
| display_description | None |
| display_name | bootable_volume |
| id | 2c370395-7f59-4c89-b312-ba35dbb986c0 |
| metadata | {} |
| size | 1 |
| snapshot_id | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | available | bootable_volume | 1 | None | | |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ VOLUME_ID=2c370395-7f59-4c89-b312-ba35dbb986c0
[root@puma06 ~(keystone_admin)]$ echo $VOLUME_ID
2c370395-7f59-4c89-b312-ba35dbb986c0
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ glance image-list | grep fedora-17
| e1b71961-d66d-4315-8e83-32aa1bd44f3f | fedora-17 | qcow2 | bare | 251985920 | active |
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ nova boot --image e1b71961-d66d-4315-8e83-32aa1bd44f3f --flavor 1 --key_name oskey f17-builder
+-------------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------+
| status | BUILD |
| updated | 2013-04-04T14:17:52Z |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-SRV-ATTR:host | None |
| key_name | oskey |
| image | fedora-17 |
| hostId | |
| OS-EXT-STS:vm_state | building |
| OS-EXT-SRV-ATTR:instance_name | instance-00000005 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| flavor | m1.tiny |
| id | 220d6e57-f7ad-4256-a861-7855ffc6788f |
| security_groups | [{u'name': u'default'}] |
| user_id | b385f4065f8c40f59fcd69d67c403ae6 |
| name | f17-builder |
| adminPass | ihbmLpLrvp3n |
| tenant_id | d2218b6611e9423fb0027acb1c389317 |
| created | 2013-04-04T14:17:52Z |
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| metadata | {} |
| config_drive | |
+-------------------------------------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17-builder
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder | ACTIVE | novanetwork=192.168.32.4 |
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ ssh -i oskey.priv 192.168.32.4
The authenticity of host '192.168.32.4 (192.168.32.4)' can't be established.
RSA key fingerprint is cd:14:a4:c3:e4:a8:15:5a:1c:80:15:15:52:8b:e0:87.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.32.4' (RSA) to the list of known hosts.
reverse mapping checking getaddrinfo for porkchop.test.redhat.com [192.168.32.4] failed - POSSIBLE BREAK-IN ATTEMPT!
[root@f17-builder ~]#
=========================
[root@f17-builder ~]# yum install wget qemu-img -y
=========================
[root@f17-builder ~]# qemu-img convert cirros-0.3.0-x86_64-disk.img -O raw cirros-0.3.0-x86_64-disk.img.raw
[root@f17-builder ~]#
=========================
[root@f17-builder ~]# file cirros-0.3.0-x86_64-disk.img
cirros-0.3.0-x86_64-disk.img: QEMU QCOW Image (v2), 41126400 bytes
=========================
[root@f17-builder ~]# file cirros-0.3.0-x86_64-disk.img.raw
cirros-0.3.0-x86_64-disk.img.raw: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, stage2 address 0x2000, stage2 segment 0x200; partition 1: ID=0x83, active, starthead 0, startsector 16065, 64260 sectors, code offset 0x48
[root@f17-builder ~]#
=========================
[root@f17-builder ~]# logout
Connection to 192.168.32.4 closed.
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17-builder
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder | ACTIVE | novanetwork=192.168.32.4 |
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | available | bootable_volume | 1 | None | | |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ nova volume-attach f17-builder 2c370395-7f59-4c89-b312-ba35dbb986c0 /dev/vdb
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| serverId | 220d6e57-f7ad-4256-a861-7855ffc6788f |
| id | 2c370395-7f59-4c89-b312-ba35dbb986c0 |
| volumeId | 2c370395-7f59-4c89-b312-ba35dbb986c0 |
+----------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | in-use | bootable_volume | 1 | None | | 220d6e57-f7ad-4256-a861-7855ffc6788f |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ ssh -i oskey.priv 192.168.32.4
reverse mapping checking getaddrinfo for porkchop.test.redhat.com [192.168.32.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Last login: Thu Apr 4 10:31:11 2013 from 192.168.32.1
[root@f17-builder ~]# cat /proc/partitions
major minor #blocks name
252 0 10241024 vda
252 1 960 vda1
252 2 9765625 vda2
252 16 1048576 vdb
[root@f17-builder ~]#
=========================
[root@f17-builder ~]# logout
Connection to 192.168.32.4 closed.
=========================
[root@puma06 ~(keystone_admin)]$ nova volume-detach f17-builder 2c370395-7f59-4c89-b312-ba35dbb986c0
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| 2c370395-7f59-4c89-b312-ba35dbb986c0 | available | bootable_volume | 1 | None | | |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ nova boot --flavor 1 --image e1b71961-d66d-4315-8e83-32aa1bd44f3f --block_device_mapping vda=2c370395-7f59-4c89-b312-ba35dbb986c0:::0 --key_name oskey f17_volume_backed
+-------------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------+
| status | BUILD |
| updated | 2013-04-04T15:01:40Z |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-SRV-ATTR:host | None |
| key_name | oskey |
| image | fedora-17 |
| hostId | |
| OS-EXT-STS:vm_state | building |
| OS-EXT-SRV-ATTR:instance_name | instance-00000006 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| flavor | m1.tiny |
| id | d4d42ec6-8a10-4ddb-8962-d6fa5875f8c5 |
| security_groups | [{u'name': u'default'}] |
| user_id | b385f4065f8c40f59fcd69d67c403ae6 |
| name | f17_volume_backed |
| adminPass | U4cUAR3nnMQ4 |
| tenant_id | d2218b6611e9423fb0027acb1c389317 |
| created | 2013-04-04T15:01:40Z |
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| metadata | {} |
| config_drive | |
+-------------------------------------+--------------------------------------+
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder | ACTIVE | novanetwork=192.168.32.4 |
| d4d42ec6-8a10-4ddb-8962-d6fa5875f8c5 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.5 |
[root@puma06 ~(keystone_admin)]$
=========================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder | ACTIVE | novanetwork=192.168.32.4 |
| d4d42ec6-8a10-4ddb-8962-d6fa5875f8c5 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.5 |
[root@puma06 ~(keystone_admin)]$ ping 192.168.32.5
PING 192.168.32.5 (192.168.32.5) 56(84) bytes of data.
From 192.168.32.1 icmp_seq=1 Destination Host Unreachable
From 192.168.32.1 icmp_seq=2 Destination Host Unreachable
From 192.168.32.1 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.32.5 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4607ms
pipe 3
=========================
[root@puma06 ~(keystone_admin)]$ file /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1
/dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack:volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1: symbolic link to `../../sdb'
[root@puma06 ~(keystone_admin)]$ file ../../sdb
../../sdb: cannot open `../../sdb' (No such file or directory)
[root@puma06 ~(keystone_admin)]$
=========================
(In reply to comment #8) > Another test, with a different setup > > Short: > - The instance booted from the volume isn't reachable via ping (icmp) > - And for some reason, the device file -- (created by cinder volume) -- > doesn't appear to 'exist' > ========================= > [root@puma06 ~(keystone_admin)]$ file > /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack: > volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1 > /dev/disk/by-path/ip-10.35.160.21:3260-iscsi-iqn.2010-10.org.openstack: > volume-2c370395-7f59-4c89-b312-ba35dbb986c0-lun-1: symbolic link to > `../../sdb' > ========================= > [root@puma06 ~(keystone_admin)]$ file ../../sdb > ../../sdb: cannot open `../../sdb' (No such file or directory) > [root@puma06 ~(keystone_admin)]$ > ========================= Your test scenario here is broken. The symlink is relative to the original file's directory, but you are checking it from your home directory, so of course you can't locate it from there. You need todo $ cd dev/disk/by-path/ $ file ../../sdb > > Your test scenario here is broken. The symlink is relative to the original > file's directory, but you are checking it from your home directory, so of > course you can't locate it from there. You need todo Indeed. That's a human error on my side. However, the issue - the volume booted instance doesn't appear to get network. Trying to drill this down further. > > $ cd dev/disk/by-path/ > $ file ../../sdb [root@puma06 dev(keystone_admin)]$ cd /dev/disk/by-path/ [root@puma06 by-path(keystone_admin)]$ [root@puma06 by-path(keystone_admin)]$ file ../../sdb ../../sdb: block special Thanks for your comment. VERIFIED.
(verification procedure is noted in the previous comments)
** Short **
-> Add this iptables rule, so that we get dhcp leases:
==================
[root@puma06 ~(keystone_admin)]$ iptables -A POSTROUTING -t mangle -p udp --dport 68 -j CHECKSUM --checksum-fill
[root@puma06 ~(keystone_admin)]$
==================
-> list the nova instances
==================
[root@puma06 ~(keystone_admin)]$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder | ACTIVE | novanetwork=192.168.32.4 |
| f1169841-5e67-45d9-85c4-0d44bd3bec90 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.6 |
[root@puma06 ~(keystone_admin)]$
==================
-> ssh into the volume_backed instance
==================
[root@puma06 ~(keystone_admin)]$ ssh -i oskey.priv -o UserKnownHostsFile=/dev/null cirros.32.6
The authenticity of host '192.168.32.6 (192.168.32.6)' can't be established.
RSA key fingerprint is a1:bc:0b:27:fb:29:ca:04:ac:ac:1c:06:c5:ac:b4:9a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.32.6' (RSA) to the list of known hosts.
$ pwd
/home/cirros
$ ls
$ ifconfig
eth0 Link encap:Ethernet HWaddr FA:16:3E:0E:4A:23
inet addr:192.168.32.6 Bcast:192.168.35.255 Mask:255.255.252.0
inet6 addr: fe80::f816:3eff:fe0e:4a23/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:202 errors:0 dropped:0 overruns:0 frame:0
TX packets:154 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:28482 (27.8 KiB) TX bytes:21801 (21.2 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
$
==================
The above confirms IP address for the cirros image.
Created attachment 731927 [details] succesful investigative trace of networking inside cirros image Additional info: [1] Related external issue -- https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/issues/14 [2] https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch11s02.html Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0718.html |