Bug 1661356 - [OSP14] cinder-volume service will become down when creating image from volume
Summary: [OSP14] cinder-volume service will become down when creating image from volume
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 14.0 (Rocky)
Assignee: Sofia Enriquez
QA Contact: Tzach Shefi
Laura Marsh
URL:
Whiteboard:
Depends On:
Blocks: 1646940
TreeView+ depends on / blocked
 
Reported: 2018-12-21 00:56 UTC by Sofia Enriquez
Modified: 2019-03-18 12:56 UTC (History)
2 users (show)

Fixed In Version: openstack-cinder-13.0.3-0.20190118014305.44c5314.el7ost
Doc Type: Bug Fix
Doc Text:
Cause: Previously, when performing image operations in Cinder, users experienced errors on RabbitMQ and DB connections because file I/O operations blocked greenthreads, preventing switching to another greenthread on I/O Consequence: As a result, the cinder-volume service was `down` when creating an image from a volume. Fix: With this update, image operations that can prevent greenthread switching are executed in native threads. Result: As a result, the cinder-volume service no longer appears `down` to the scheduler.
Clone Of: 1646940
Environment:
Last Closed: 2019-03-18 12:56:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1801958 0 None None None 2019-01-04 18:35:25 UTC
OpenStack gerrit 624497 0 None MERGED Ensure image utils don't block greenthreads 2020-06-15 04:11:10 UTC
Red Hat Product Errata RHBA-2019:0586 0 None None None 2019-03-18 12:56:29 UTC

Description Sofia Enriquez 2018-12-21 00:56:06 UTC
The problem is caused by the I/O operations performed by the glance client when uploading the volume to the image in Glance, as they seem to be blocking the greenthread switching, as the issues happen after we have attached the volume to the Cinder node, confirmed that we can do I/O on it using dd, and right when the code opens the device and calls the Glance client with the file descriptor.

Comment 1 Sofia Enriquez 2018-12-21 15:19:42 UTC
Duplication of the original bug. Not needed.

Comment 2 Sofia Enriquez 2019-01-04 18:30:11 UTC
Reopen this bug to track a fix for bug #1661356 in OPS14.

Comment 5 Tzach Shefi 2019-03-13 09:56:45 UTC
Verified on:
openstack-cinder-13.0.3-0.20190118014305.44c5314.el7ost


1. Uploaded a rhel (~500mb) image to glance. 
glance image-create --disk-format qcow2 --container-format bare --file rhel7.3_root_qum5net.qcow2 --name rhel

Created a volume from this image:
#cinder create 10 --image rhel


Upload volume to a new image:
openstack image create --volume 2eada5c4-a970-4551-b01f-1cedef03f38c  imageFromVol

While doing this I watched -d -n 2 openstack volume service list
+------------------+-------------------------+------+---------+-------+----------------------------+
| Binary           | Host                    | Zone | Status  | State | Updated At                 |
+------------------+-------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | controller-2            | nova | enabled | up    | 2019-02-28T15:38:25.000000 |
| cinder-scheduler | controller-1            | nova | enabled | up    | 2019-02-28T15:38:28.000000 |
| cinder-scheduler | controller-0            | nova | enabled | up    | 2019-02-28T15:38:29.000000 |
| cinder-volume    | hostgroup@tripleo_iscsi | nova | enabled | up    | 2019-02-28T15:38:30.000000 |
+------------------+-------------------------+------+---------+-------+----------------------------+

Cinder volume remains up the whole time, 
tried a few time same "up" status remains, never it go down. 

Image is available:
#glance image-list
+--------------------------------------+--------------+
| ID                                   | Name         |
+--------------------------------------+--------------+
| 8e33e4de-c7ef-49d7-b61b-e6ce150c2a1d | cirros       |
| 5e4d538d-57b5-4a82-aab9-4588208d9d3b | imageFromVol |
| 0188e8aa-e05f-48d0-b1f9-5d81a79a03be | rhel         |
+--------------------------------------+--------------+

Looks good to verify.

Comment 7 errata-xmlrpc 2019-03-18 12:56:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0586


Note You need to log in before you can comment on or make changes to this bug.