Created attachment 931779 [details] Cinder and glance logs Description of problem: Fail to create cinder volume from Glance image, with Cinder Dell EQL storage driver/backend. Empty volumes are created successfully ERROR cinder.volume.flows.manager.create_volume [req-67a1a22e-f7b7-49e4-a365-f66e58026904 3421d84131ca4385af185e04e2cb3d11 b5678896173d4959a3190f7ca17d886a - - -] Failed to copy image 58a8d212-7ecf-4d8a-b07c-94d879fadb52 to volume: b03d7bba-5dcb-4735-af42-cb8783f8da33, error: Volume device not found at /dev/disk/by-path/ip-10.16.45.61:3260-iscsi-iqn.2001-05.com.equallogic:0-8a0906-4d06b4906-0030073ce8853fee-volume-b03d7bba-5dcb-4735-af42-cb8783f8da33-lun-0. 2014-08-28 11:14:43.076 29860 DEBUG cinder.volume.flows.common [req-67a1a22e-f7b7-49e4-a365-f66e58026904 3421d84131ca4385af185e04e2cb3d11 b5678896173d4959a3190f7ca17d886a - - -] Updating volume: b03d7bba-5dcb-4735-af42-cb8783f8da33 with {'status': 'error'} due to: ??? error_out_volume /usr/lib/python2.7/site-packages/cinder/volume/flows/common.py:87 2014-08-28 11:14:43.114 29860 ERROR cinder.volume.flows.manager.create_volume [req-67a1a22e-f7b7-49e4-a365-f66e58026904 3421d84131ca4385af185e04e2cb3d11 b5678896173d4959a3190f7ca17d886a - - -] Volume b03d7bba-5dcb-4735-af42-cb8783f8da33: create failed ... 2014-08-28 11:14:43.115 29860 TRACE oslo.messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex) 2014-08-28 11:14:43.115 29860 TRACE oslo.messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: Volume device not found at /dev/disk/by-path/ip-10.16.45.61:3260-iscsi-iqn.2001-05.com.equallogic:0-8a0906-4d06b4906-0030073ce8853fee-volume-b03d7bba-5dcb-4735-af42-cb8783f8da33-lun-0. 2014-08-28 11:14:43.115 29860 TRACE oslo.messaging.rpc.dispatcher Version-Release number of selected component (if applicable): RHEL7 python-cinder-2014.1.2-2.el7ost.noarch openstack-cinder-2014.1.2-2.el7ost.noarch python-cinderclient-1.0.9-1.el7ost.noarch openstack-glance-2014.1.2-1.el7ost.noarch python-glance-2014.1.2-1.el7ost.noarch python-glanceclient-0.13.1-1.el7ost.noarch Dell EQL PS6000 storage How reproducible: Every time Steps to Reproduce: 1. Setup AIO, (my setup also included Nova live migration over NFS share, not related just mentioning the fact) 2. Configure Cinder to use EQL driver 3. Try to create a new volume from Glance image, fails status = error, Volume on log -> b03d7bba-5dcb-4735-af42-cb8783f8da33). 4. Create an empty Cinder volume, successfully created. Actual results: Failed to create volume from Glance image, volumes status=error Expected results: Should successfully create a Cinder volume from Glance image, as happens with Cinder LVM.
Created attachment 931973 [details] OPS5 KB article on botting from volume, image from EQL array
This looks very familiar what we had seen for OSP4. Enclosed is updated article for OSP5 we published on it. It had not been published yet. Will check with EQL team when it will be published. Original OSP4 one is at https://eqlsupport.dell.com/support/resources.aspx?id=6442455902&terms=openstack Rajini will take a quick look at it to see if this is the same problem.
Yes, the KB article is correct. Because of the reported size 1.01GB by EQL, there is a problem. See below for steps to reproduce, glance for me actually reported 1.01GB size ( step3) and cinder volume create with 1GB failed with a proper error message. It is different from the Error Message reported in this bug. Steps to reproduce: 1. Create a 1GB volume in OpenStack using Horizon or the Cinder CLI, backed with the Cinder EQL driver. 2. The volume is created and stored in the EQL array. The actual size is 1.01GB, but OpenStack reports it as 1GB. 3. Using the Cinder CLI, upload the 1.01GB volume to Glance. Once uploaded, Glance will report the image size as 1GB. 4. Using the Cinder CLI, create a new 1GB image with the Glance image as the source. The volume creation will fail as the target volume size is too small. 5. Change the target volume size to 2GB, and then repeat step 4. The volume will be created. stack@devstack-juno-precise:/opt/stack/logs/screen$ glance image-list +--------------------------------------+---------------------------------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------------------------+-------------+------------------+------------+--------+ | 844a2c3b-bcc7-40c9-bde6-156a6f0f07a2 | bug1124770 | raw | bare | 1085276160 | active | stack@devstack-juno-precise:/opt/stack/logs/screen$ cinder create --image-id 844a2c3b-bcc7-40c9-bde6-156a6f0f07a2 --display-name Glance2Volume000_1 1 ERROR: Invalid input received: Size of specified image 2GB is larger than volume size 1GB. (HTTP 400) (Request-ID: req-e6c35cd9-abe9-4cf5-83d0-34f671a3ed85)
I can successfully create a cinder volume from glance image on the latest devstack-juno distribution on our environment when I specify 2GB size. See below. Can you send us the configuration information and architeture of glance, cinder and nova? Seems like some access or configuration issue. stack@devstack-juno-precise:/opt/stack/logs/screen$ cinder create --image-id 844a2c3b-bcc7-40c9-bde6-156a6f0f07a2 --display-name Glance2Volume000_1 2 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-08-28T16:28:53.000000 | | description | None | | encrypted | False | | id | b56cc4b3-3a61-4a20-8e45-2deb0511b6aa | | metadata | {} | | name | Glance2Volume000_1 | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 9d506fdd361b41899e4bec36cef788f7 | | size | 2 | | snapshot_id | None | | source_volid | None | | status | creating | | user_id | 7392adef8a6d4a58a0c61b162a760911 | | volume_type | None | +--------------------------------+--------------------------------------+
In summary, this bug is not related to the KB article. Seems like some access or configuration issue. Can you send us the configuration information and architeture of glance, cinder and nova?
I couldn't reproduce in Juno/master. I will try it on IceHouse
Attaching Cinder and Glance config files. Glance uses local /var/lib/glance/images folder. The only input I can add think of is maybe network delay is causing this. As my AIO+Compute nodes are in Israel and Dell storage is at US site. Yes I'd rather test this all on same local LAN but don't have a local Dell storage :( Any way if problem is caused by latency or WAN problem, shouldn't we cover this by driver, meaning giving a better error notice?
Created attachment 933120 [details] Glance and cinder config files
Please, configure: num_volume_device_scan_tries=5 in your cinder.conf, it will work. It's default value is 3 which is not enough in your particular case. Your environment is too slow, it takes time for new device to appear under /dev/disk/by-path/... meantime after 3 retries system declares that destination images doesn't exist and stops operation.