This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1308329 - ceph: uploading volume to image fails
ceph: uploading volume to image fails
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: ga
: 9.0 (Mitaka)
Assigned To: Eric Harney
lkuchlan
: Automation, AutomationBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-14 05:04 EST by bkopilov
Modified: 2016-08-11 08:14 EDT (History)
7 users (show)

See Also:
Fixed In Version: openstack-cinder-8.0.0-4.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-08-11 08:14:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
cinder_conf (24.22 KB, application/x-gzip)
2016-02-14 05:22 EST, bkopilov
no flags Details
cinder_log (5.81 KB, application/x-gzip)
2016-02-14 05:22 EST, bkopilov
no flags Details
glance_conf (13.36 KB, application/x-gzip)
2016-02-14 05:25 EST, bkopilov
no flags Details
glance_log (10.79 KB, application/x-gzip)
2016-02-14 05:25 EST, bkopilov
no flags Details
messages_log (573 bytes, application/x-gzip)
2016-02-14 05:26 EST, bkopilov
no flags Details
nova_llogs (9.69 KB, application/x-gzip)
2016-02-14 05:26 EST, bkopilov
no flags Details
glance_cinder_logs (806.16 KB, application/x-gzip)
2016-02-16 02:09 EST, bkopilov
no flags Details

  None (edit)
Description bkopilov 2016-02-14 05:04:46 EST
Description of problem:
uploading volume to image fails



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
#1 create volume
cinder --os-volume-api-version 1 create 1
#2 upload-volume-to-image (raw image)

# 4c7c6bc9-b684-409f-bdc8-169c2a3d6542 | cirros-0.3.4-x86_64-disk.img | raw
| bare | 1073741824 | active
#3 cinder --os-volume-api-version 1 upload-to-image
 e51bbdf2-3037-445a-8dd3-1d9e86cd1733 4c7c6bc9-b684-409f-bdc8-169c2a3d6542

Actual results:


Expected results:


Additional info:
Comment 1 bkopilov 2016-02-14 05:22 EST
Created attachment 1127005 [details]
cinder_conf
Comment 2 bkopilov 2016-02-14 05:22 EST
Created attachment 1127006 [details]
cinder_log
Comment 3 bkopilov 2016-02-14 05:25 EST
Created attachment 1127007 [details]
glance_conf
Comment 4 bkopilov 2016-02-14 05:25 EST
Created attachment 1127008 [details]
glance_log
Comment 5 bkopilov 2016-02-14 05:26 EST
Created attachment 1127009 [details]
messages_log
Comment 6 bkopilov 2016-02-14 05:26 EST
Created attachment 1127010 [details]
nova_llogs
Comment 7 bkopilov 2016-02-14 06:00:59 EST
2016-02-12 14:19:50.057 13664 ERROR cinder.volume.manager [req-cad6c847-c508-4994-9b2d-4c0117586190 473ce88f3353485a95902485b624c51e 346db09dcb1447aaacf1258b628cfb4c - - -] Cannot delete volume 3a983909-cc06-41a7-8c25-bedd7be9ac71: volume is busy
2016-02-12 14:32:14.158 13664 WARNING requests.packages.urllib3.connectionpool [req-eb6b93ae-aeb7-48c6-be36-a7e466e1a597 66d18bf575b9474392835f8f0c786337 2c47cfe2ea7845eaa87348dae4261d0d - - -] Connection pool is full, discarding connection: 172.17.3.10
2016-02-12 14:32:14.261 13664 ERROR cinder.volume.manager [req-eb6b93ae-aeb7-48c6-be36-a7e466e1a597 66d18bf575b9474392835f8f0c786337 2c47cfe2ea7845eaa87348dae4261d0d - - -] Error occurred while uploading volume 36b7fccb-a2c0-421a-95e4-8b2855497379 to image 347128a7-5b59-4db8-b246-f213fa1f4d4a.
2016-02-12 14:32:14.329 13664 ERROR oslo_messaging.rpc.dispatcher [req-eb6b93ae-aeb7-48c6-be36-a7e466e1a597 66d18bf575b9474392835f8f0c786337 2c47cfe2ea7845eaa87348dae4261d0d - - -] Exception during message handling: 'NoneType' object has no attribute 'keys'
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 972, in copy_volume_to_image
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     payload['message'] = six.text_type(error)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 958, in copy_volume_to_image
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     image_meta)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 904, in copy_volume_to_image
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     image_meta, tmp_file)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 337, in upload_volume
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     image_service.update(context, image_id, {}, image_file)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 333, in update
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     return self._translate_from_glance(context, image_meta)
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 366, in _translate_from_glance
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher     for key in image.keys()
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'NoneType' object has no attribute 'keys'
2016-02-12 14:32:14.329 13664 TRACE oslo_messaging.rpc.dispatcher 
2016-02-12 14:33:28.605 13664 WARNING cinder.volume.drivers.rbd [req-da9724ae-540a-49ca-a8cb-c8c06c0ef73c 2ecdd7b7e33643efb888e021b0c00e09 2d702d8179fd43ac9c9ba344c8037a7f - - -] ImageBusy error raised while deleting rbd volume. This may have been caused by a connection from a client that has crashed and, if so, may be resolved by retrying the delete after 30 seconds has elapsed.
2016-02-12 14:33:28.609 13664 ERROR cinder.volume.manager [req-da9724ae-540a-49ca-a8cb-c8c06c0ef73c 2ecdd7b7e33643efb888e021b0c00e09 2d702d8179fd43ac9c9ba344c8037a7f - - -] Cannot delete volume f734dc6d-0e14-41ab-b0b2-7410f78a9977: volume is busy
2016-02-12 14:38:47.062 13664 WARNING cinder.quota [req-61955db0-4c50-4d98-8343-b8b160a098b8 - - - - -] Deprecated: Default quota for resource: gigabytes_scenario-type-luks-1457371063 is set by the default quota flag: quota_gigabytes_scenario-type-luks-1457371063, it is now deprecated. Please use the default quota class for default quota.
2016-02-12 14:38:47.065 13664 WARNING cinder.quota [req-61955db0-4c50-4d98-8343-b8b160a098b8 - - - - -] Deprecated: Default quota for resource: volumes_scenario-type-luks-1457371063 is set by the default quota flag: quota_volumes_scenario-type-luks-1457371063, it is now deprecated. Please use the default quota class for default quota.
2016-02-12 14:47:18.290 13664 WARNING requests.packages.urllib3.connectionpool [req-86ffb8c9-464c-44d7-987f-a934f3bef797 f99ed8a44b634bbe8b2e69852048cff5 0ebda79750a64ceb96eb74d60f789f6a - - -] Connection pool is full, discarding connection: 172.17.3.10
2016-02-12 14:47:18.390 13664 ERROR cinder.volume.manager [req-86ffb8c9-464c-44d7-987f-a934f3bef797 f99ed8a44b634bbe8b2e69852048cff5 0ebda79750a64ceb96eb74d60f789f6a - - -] Error occurred while uploading volume 85d4226e-805c-4cdd-996e-f3fad75878fe to image 15c64524-8497-43e7-859f-5577949e9c75.
2016-02-12 14:47:18.459 13664 ERROR oslo_messaging.rpc.dispatcher [req-86ffb8c9-464c-44d7-987f-a934f3bef797 f99ed8a44b634bbe8b2e69852048cff5 0ebda79750a64ceb96eb74d60f789f6a - - -] Exception during message handling: 'NoneType' object has no attribute 'keys'
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 972, in copy_volume_to_image
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     payload['message'] = six.text_type(error)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 958, in copy_volume_to_image
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     image_meta)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 904, in copy_volume_to_image
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     image_meta, tmp_file)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 337, in upload_volume
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     image_service.update(context, image_id, {}, image_file)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 333, in update
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     return self._translate_from_glance(context, image_meta)
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 366, in _translate_from_glance
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher     for key in image.keys()
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'NoneType' object has no attribute 'keys'
2016-02-12 14:47:18.459 13664 TRACE oslo_messaging.rpc.dispatcher 
2016-02-12 14:48:45.907 13664 WARNING cinder.volume.drivers.rbd [req-e21366ed-0c6e-4fe0-8683-1ceecb1c2bba 2f7d7cdf21224dceb467818669a3bf4a 0e95ca969b6948f181b3bb356cdd4dc8 - - -] ImageBusy error raised while deleting rbd volume. This may have been caused by a connection from a client that has crashed and, if so, may be resolved by retrying the delete after 30 seconds has elapsed.
Comment 10 Sergey Gotliv 2016-02-15 17:15:01 EST
How did you create that environment? Glance registry log is completely empty, I don't see any evidence that this environment is properly booted. Can you, please, start all services in a debug mode and re-run that test again?
Comment 11 bkopilov 2016-02-16 01:16:29 EST
Hi , 
It was deployed with OSP-D , by out automation.
Comment 12 bkopilov 2016-02-16 02:09 EST
Created attachment 1127498 [details]
glance_cinder_logs
Comment 13 Sergey Gotliv 2016-05-23 17:10:02 EDT
Benny,

Can you, please, retest it with openstack-cinder-8.0.0-4.el7ost or later versions.
Comment 15 lkuchlan 2016-06-06 04:26:06 EDT
Tested using:
openstack-cinder-8.0.0-4.el7ost.noarch
python-cinderclient-1.6.0-1.el7ost.noarch
python-cinder-8.0.0-4.el7ost.noarch

Verification flow:

[stack@undercloud ~]$ cinder --os-volume-api-version 1 create 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2016-06-06T08:20:50.417335      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | 90dc4d12-5d4b-4af9-946d-43c922ff97f3 |
|       metadata      |                  {}                  |
|     multiattach     |                false                 |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[stack@undercloud ~]$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 90dc4d12-5d4b-4af9-946d-43c922ff97f3 | available |  -   |  1   |      -      |  false   |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
[stack@undercloud ~]$ cinder --os-volume-api-version 1 upload-to-image 90dc4d12-5d4b-4af9-946d-43c922ff97f3 image1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|   container_format  |                 bare                 |
|     disk_format     |                 raw                  |
| display_description |                 None                 |
|          id         | 90dc4d12-5d4b-4af9-946d-43c922ff97f3 |
|       image_id      | 8f8e15c4-3f38-4c16-9611-4ce6c9953193 |
|      image_name     |                image1                |
|         size        |                  1                   |
|        status       |              uploading               |
|      updated_at     |      2016-06-06T08:20:53.000000      |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[stack@undercloud ~]$ glance image-list
+--------------------------------------+--------+
| ID                                   | Name   |
+--------------------------------------+--------+
| 8f8e15c4-3f38-4c16-9611-4ce6c9953193 | image1 |
+--------------------------------------+--------+
Comment 17 errata-xmlrpc 2016-08-11 08:14:14 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html

Note You need to log in before you can comment on or make changes to this bug.