Description of problem: unable to attach a cinder volume to an instance in juno release of openstack. Version-Release number of selected component (if applicable): RDO openstack juno on F21 Server How reproducible: 100% Steps to Reproduce: 1. create an instance 2. create a volume and attach to the instance. It will fail with the below error in nova-compute.log ==--== 2015-03-31 18:48:59.381 2162 INFO nova.compute.resource_tracker [-] Compute_service record updated for localhost.localdomain:localhost.localdomain 2015-03-31 18:49:06.909 2162 AUDIT nova.compute.manager [req-282ee583-ca53-43f5-ab70-80a1c5632246 None] [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] Attaching volume 7dd8b1d4-5936-4645-b71c-eb7fbbca96cc to /dev/vdb 2015-03-31 18:49:06.910 2162 WARNING nova.volume.cinder [req-282ee583-ca53-43f5-ab70-80a1c5632246 None] Cinder V1 API is deprecated as of the Juno release, and Nova is still configured to use it. Enable the V2 API in Cinder and set cinder_catalog_info in nova.conf to use it. 2015-03-31 18:50:01.194 2162 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-03-31 18:50:01.386 2162 AUDIT nova.compute.resource_tracker [-] Total physical ram (MB): 3952, total allocated virtual ram (MB): 1024 2015-03-31 18:50:01.387 2162 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 77 2015-03-31 18:50:01.387 2162 AUDIT nova.compute.resource_tracker [-] Total usable vcpus: 3, total allocated vcpus: 1 2015-03-31 18:50:01.388 2162 AUDIT nova.compute.resource_tracker [-] PCI stats: [] 2015-03-31 18:50:01.388 2162 INFO nova.compute.resource_tracker [-] Compute_service record updated for localhost.localdomain:localhost.localdomain 2015-03-31 18:50:51.194 2162 INFO nova.compute.manager [-] Updating bandwidth usage cache 2015-03-31 18:50:51.231 2162 WARNING nova.compute.manager [-] Bandwidth usage not supported by hypervisor. 2015-03-31 18:51:01.508 2162 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources 2015-03-31 18:51:01.730 2162 AUDIT nova.compute.resource_tracker [-] Total physical ram (MB): 3952, total allocated virtual ram (MB): 1024 2015-03-31 18:51:01.730 2162 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 77 2015-03-31 18:51:01.730 2162 AUDIT nova.compute.resource_tracker [-] Total usable vcpus: 3, total allocated vcpus: 1 2015-03-31 18:51:01.730 2162 AUDIT nova.compute.resource_tracker [-] PCI stats: [] 2015-03-31 18:51:01.731 2162 INFO nova.compute.resource_tracker [-] Compute_service record updated for localhost.localdomain:localhost.localdomain 2015-03-31 18:51:30.321 2162 ERROR nova.virt.block_device [req-282ee583-ca53-43f5-ab70-80a1c5632246 None] [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] Driver failed to attach volume 7dd8b1d4-5936-4645-b71c-eb7fbbca96cc at /dev/vdb 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] Traceback (most recent call last): 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] device_type=self['device_type'], encryption=encryption) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1381, in attach_volume 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] conf = self._connect_volume(connection_info, disk_info) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1328, in _connect_volume 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] return driver.connect_volume(connection_info, disk_info) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] return f(*args, **kwargs) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 318, in connect_volume 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] self._run_iscsiadm(iscsi_properties, ("--rescan",)) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 236, in _run_iscsiadm 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] check_exit_code=check_exit_code) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 163, in execute 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] return processutils.execute(*cmd, **kwargs) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 203, in execute 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] cmd=sanitized_cmd) 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] ProcessExecutionError: Unexpected error while running command. 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7dd8b1d4-5936-4645-b71c-eb7fbbca96cc -p 192.168.122.70:3260 --rescan 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] Exit code: 21 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] Stdout: u'' 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] Stderr: u'iscsiadm: No session found.\n' 2015-03-31 18:51:30.321 2162 TRACE nova.virt.block_device [instance: 7a0d5c72-e409-4f48-b500-892fb91e504b] ==--== Actual results: failed to attach the volume. Expected results: volume should attach without any issue. Additional info: manually started the iscsi service but still the issue is there and its unable to login the iscsi: ==--== # systemctl status iscsi.service ● iscsi.service - Login and scanning of iSCSI devices Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled) Active: active (exited) since Tue 2015-03-31 18:54:14 IST; 8min ago Docs: man:iscsid(8) man:iscsiadm(8) Process: 5385 ExecStart=/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=11) Process: 5381 ExecStart=/usr/libexec/iscsi-mark-root-nodes (code=exited, status=0/SUCCESS) Main PID: 5385 (code=exited, status=11) CGroup: /system.slice/iscsi.service Mar 31 18:52:00 localhost.localdomain iscsi-mark-root-nodes[5381]: iscsiadm: No active sessions. Mar 31 18:54:14 localhost.localdomain iscsiadm[5385]: iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-7dd8b1d4-5936-4645-b71c-eb7fbbca96cc, portal: 192.168.122.70,3260]. Mar 31 18:54:14 localhost.localdomain iscsiadm[5385]: iscsiadm: initiator reported error (11 - iSCSI PDU timed out) Mar 31 18:54:14 localhost.localdomain iscsiadm[5385]: iscsiadm: Could not log into all portals Mar 31 18:54:14 localhost.localdomain iscsiadm[5385]: Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-7dd8b1d4-5936-4645-b71c-eb7fbbca96cc, portal: 192.168.122.70,3260] (multiple) ==--==
Could you also please attach the cinder volume log?
The new instance creation is OK. From command line also its not working: [root@localhost ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 7dd8b1d4-5936-4645-b71c-eb7fbbca96cc | available | vol | 1 | iscsi | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@localhost ~(keystone_admin)]# nova list +--------------------------------------+-------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+---------------------+ | 7a0d5c72-e409-4f48-b500-892fb91e504b | first | ACTIVE | - | Running | public=172.24.4.228 | +--------------------------------------+-------+--------+------------+-------------+---------------------+ [root@localhost ~(keystone_admin)]# nova volume-attach 7a0d5c72-e409-4f48-b500-892fb91e504b 7dd8b1d4-5936-4645-b71c-eb7fbbca96cc auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 7dd8b1d4-5936-4645-b71c-eb7fbbca96cc | | serverId | 7a0d5c72-e409-4f48-b500-892fb91e504b | | volumeId | 7dd8b1d4-5936-4645-b71c-eb7fbbca96cc | +----------+--------------------------------------+ [root@localhost ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 7dd8b1d4-5936-4645-b71c-eb7fbbca96cc | attaching | vol | 1 | iscsi | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
Created attachment 1009071 [details] Cinder volume log
Eric, Here is the error: ==--== 2015-03-31 18:49:10.864 1868 INFO cinder.brick.iscsi.iscsi [req-b432129a-cbd6-4153-aab7-0f1444fe9a96 c8907d964a6241ff886896b19c9be242 41dc9394e65f46b187e4021f8f7881dd - - -] Creating iscsi_target fo r volume: volume-7dd8b1d4-5936-4645-b71c-eb7fbbca96cc 2015-03-31 18:49:12.147 1868 ERROR cinder.brick.iscsi.iscsi [req-b432129a-cbd6-4153-aab7-0f1444fe9a96 c8907d964a6241ff886896b19c9be242 41dc9394e65f46b187e4021f8f7881dd - - -] Failed to create iscsi target for volume id:volume-7dd8b1d4-5936-4645-b71c-eb7fbbca96cc. 2015-03-31 18:49:12.149 1868 ERROR oslo.messaging.rpc.dispatcher [req-b432129a-cbd6-4153-aab7-0f1444fe9a96 c8907d964a6241ff886896b19c9be242 41dc9394e65f46b187e4021f8f7881dd - - -] Exception during m essage handling: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. ==--== More errors are there in the attached volume.log file. Thanks, Uday
Eric, Did you get a chance to look into the attached logs. Manually also the below command is not working. # sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-7dd8b1d4-5936-4645-b71c-eb7fbbca96cc -p 192.168.122.70:3260 --rescan Thanks, Uday
This is caused by puppet-cinder starting tgtd even though it should be using lioadm.
Fedora 20 and 21 should now use lioadm
Thanks Lukas, Currently do you have any workaround for this issue ?
I would grab opm rpm from here http://koji.fedoraproject.org/koji/buildinfo?buildID=625675 and rerun packstack. After rerun stop tgtd daemon.
https://review.gerrithub.io/230643