Bug 1851051

Summary: copy-image can start multiple importing threads due to race condition
Product: Red Hat OpenStack Reporter: Abhishek Kekane <akekane>
Component: openstack-glanceAssignee: Abhishek Kekane <akekane>
Status: CLOSED ERRATA QA Contact: Mike Abrams <mabrams>
Severity: high Docs Contact: RHOS Documentation Team <rhos-docs>
Priority: high    
Version: 16.1 (Train)CC: athomas, cyril, eglynn, gcharot, gfidente, gregraka, lmarsh, pgrist
Target Milestone: betaKeywords: Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-glance-19.0.4-2.20210216215005.5bbd356.el8ost Doc Type: Bug Fix
Doc Text:
Before this update, RBD performance was degraded when multiple instances were launched simultaneously. This was due to the Image service (glance) starting multiple threads to perform the same copying operation. This update resolves the issue.
Story Points: ---
Clone Of:
: 1852496 (view as bug list) Environment:
Last Closed: 2021-09-15 07:08:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1866592    
Bug Blocks: 1852496    

Description Abhishek Kekane 2020-06-25 13:55:31 UTC
Two subsequent copy-image API calls for same image to different set of stores or same set of stores will cause in race condition and can provide different results than expected.

Especially in a situation where glance is running on multiple control plane nodes (i.e. any real-world situation), I believe there is a race condition whereby two closely-timed requests to copy an image to a store will result in two copy operations in glance proceeding in parallel.

Glance will not realize that a thread is already running to complete the initial task and will start another. In a situation where a user spawns a thousand new instances to a thousand compute nodes in a single operation where the image needs copying first, it's highly plausible to have _many_ duplicate glance operations going, impacting write performance on the rbd cluster at the very least.

Comment 1 Cyril Roelandt 2020-06-26 14:17:54 UTC
Is there some kind of workaround available right now? Or should the admin just be careful? Should we release an update to the documentation?

Comment 2 Abhishek Kekane 2020-06-29 05:17:37 UTC
(In reply to Cyril Roelandt from comment #1)
> Is there some kind of workaround available right now? Or should the admin
> just be careful? Should we release an update to the documentation?

At the moment admin should be just careful to avoid this scenario. We should mention this in documentation as well.

Comment 3 Cyril Roelandt 2020-06-29 18:32:55 UTC
@Laura: can this be added as a warning in our documentation?

Comment 4 Laura Marsh 2020-06-30 14:32:45 UTC
yes, working on the wording now

Comment 8 Cyril Roelandt 2020-09-02 17:16:32 UTC
And also https://review.opendev.org/744024

Comment 9 Cyril Roelandt 2020-09-02 17:18:40 UTC
Disregard #8

Comment 11 Abhishek Kekane 2020-09-04 05:24:00 UTC
(In reply to Cyril Roelandt from comment #10)
> @Abhishek: the patches in the "Links" box are still required, right? These
> ones:
> 
> https://review.opendev.org/737868/
> https://review.opendev.org/743426/
> https://review.opendev.org/743427/
> https://review.opendev.org/743593/
> https://review.opendev.org/743595/
> https://review.opendev.org/743596/
> https://review.opendev.org/743597/
> https://review.opendev.org/743839/

Hi Cyril,

Patches mentioned in comment #6 and #7 includes these patches.
Actually patches in links box are from master branch and patches in comment #6 and #7 are upstream backports to ussuri.

We need to backport patches from #6 and #7 as there are some changes in those backports not included from master branch as those were related to new feature.
So it will be quiet easy for us if we consider those #6 and #7 patches only.

Thank you,

Abhishek

Comment 20 Mike Abrams 2021-06-01 10:25:40 UTC
to verify:

=== use copy-image method to copy to a store while a copy-image op is underway (should result in a 409 error)

(central) [stack@site-undercloud-0 ~]$ glance image-import 3a0ddfd6-c4e2-41d5-915e-4d8a39aadb5e --import-method copy-image --stores dcn1; glance image-import 3a0ddfd6-c4e2-41d5-915e-4d8a39aadb5e --import-method copy-image --stores dcn2 
(central) [stack@site-undercloud-0 ~]$ glance image-create-via-import --container-format ami --disk-format ami --name import_scenario --import-method web-download --uri https://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud-1510.qcow2
+-----------------------+--------------------------------------+
| Property              | Value                                |
+-----------------------+--------------------------------------+
| checksum              | None                                 |
| container_format      | ami                                  |
| created_at            | 2021-06-01T08:48:41Z                 |
| disk_format           | ami                                  |
| id                    | cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 |
| locations             | []                                   |
| min_disk              | 0                                    |
| min_ram               | 0                                    |
| name                  | import_scenario                      |
| os_glance_import_task | 9b4fe0c9-b8d6-4b5b-9c09-60238c5173b8 |
| os_hash_algo          | None                                 |
| os_hash_value         | None                                 |
| os_hidden             | False                                |
| owner                 | fbf93b9227c548938a91ed1b43507a69     |
| protected             | False                                |
| size                  | None                                 |
| status                | queued                               |
| tags                  | []                                   |
| updated_at            | 2021-06-01T08:48:41Z                 |
| virtual_size          | Not available                        |
| visibility            | shared                               |
+-----------------------+--------------------------------------+
(central) [stack@site-undercloud-0 ~]$
(central) [stack@site-undercloud-0 ~]$ glance image-import cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 --import-method copy-image --stores dcn1
+-------------------------------+----------------------------------------------------------------------------------+
| Property                      | Value                                                                            |
+-------------------------------+----------------------------------------------------------------------------------+
| checksum                      | dcef39a9f88ef73cf782d9633d7e3451                                                 |
| container_format              | bare                                                                             |
| created_at                    | 2021-06-01T08:48:41Z                                                             |
| direct_url                    | rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/cb8d05cd-a19f-42f6-b2c9-7c16f1 |
|                               | 4c4482/snap                                                                      |
| disk_format                   | raw                                                                              |
| id                            | cb8d05cd-a19f-42f6-b2c9-7c16f14c4482                                             |
| locations                     | [{"url": "rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/cb8d05cd-a19f-42f6-b |
|                               | 2c9-7c16f14c4482/snap", "metadata": {"store": "central"}}]                       |
| min_disk                      | 0                                                                                |
| min_ram                       | 0                                                                                |
| name                          | import_scenario                                                                  |
| os_glance_failed_import       |                                                                                  |
| os_glance_import_task         | 7995f4b8-8342-49bf-932a-c45a74a044a7                                             |
| os_glance_importing_to_stores |                                                                                  |
| os_hash_algo                  | sha512                                                                           |
| os_hash_value                 | 2ae24da25883b04af54f2e0edc65627772b2c4150108613e3d399718efab5b5353f5271e84ba5578 |
|                               | 9fcdd19ab5ff4bac1ef4934f5c0f204400c9f8bb541dbe44                                 |
| os_hidden                     | False                                                                            |
| owner                         | fbf93b9227c548938a91ed1b43507a69                                                 |
| protected                     | False                                                                            |
| size                          | 8589934592                                                                       |
| status                        | active                                                                           |
| stores                        | central                                                                          |
| tags                          | []                                                                               |
| updated_at                    | 2021-06-01T08:55:24Z                                                             |
| virtual_size                  | 8589934592                                                                       |
| visibility                    | shared                                                                           |
+-------------------------------+----------------------------------------------------------------------------------+
(central) [stack@site-undercloud-0 ~]$ 

=== try to upload the image that is actively being uploaded to a third store:

(central) [stack@site-undercloud-0 ~]$ glance image-import cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 --import-method copy-image --stores dcn2
HTTP 502 Proxy Error: Proxy Error: The proxy server received an invalid: response from an upstream server.: The proxy server could not handle the requestReason: Error reading from remote server
(central) [stack@site-undercloud-0 ~]$ glance image-import cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 --import-method copy-image --stores dcn2
HTTP 409 Conflict: Image has active task
(central) [stack@site-undercloud-0 ~]$ 

=== ...409 expected

=== wait until the copy op is done to the second store:
(central) [stack@site-undercloud-0 ~]$ while :; do glance image-show cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 | grep stores; sleep 5; done
| os_glance_importing_to_stores | dcn1                                                                             |
| stores                        | central         
...

=== once the original copy to store task is complete, should be able to copy to additional store without the 409 error; the whole process can be seen here:

(central) [stack@site-undercloud-0 ~]$ glance image-create-via-import --container-format ami --disk-format ami --name import_scenario --import-method web-download --uri https://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img
+-----------------------+--------------------------------------+
| Property              | Value                                |
+-----------------------+--------------------------------------+
| checksum              | None                                 |
| container_format      | ami                                  |
| created_at            | 2021-06-01T10:22:44Z                 |
| disk_format           | ami                                  |
| id                    | 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 |
| locations             | []                                   |
| min_disk              | 0                                    |
| min_ram               | 0                                    |
| name                  | import_scenario                      |
| os_glance_import_task | 93399714-52c2-4fc8-b37a-45383eb1c51f |
| os_hash_algo          | None                                 |
| os_hash_value         | None                                 |
| os_hidden             | False                                |
| owner                 | fbf93b9227c548938a91ed1b43507a69     |
| protected             | False                                |
| size                  | None                                 |
| status                | queued                               |
| tags                  | []                                   |
| updated_at            | 2021-06-01T10:22:44Z                 |
| virtual_size          | Not available                        |
| visibility            | shared                               |
+-----------------------+--------------------------------------+
(central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep status
| status                        | active                                                                           |
(central) [stack@site-undercloud-0 ~]$ glance image-import 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 --import-method copy-image --stores dcn1                                    
+-------------------------------+----------------------------------------------------------------------------------+
| Property                      | Value                                                                            |
+-------------------------------+----------------------------------------------------------------------------------+
| checksum                      | 318f4eb47a165b55db0be707e0ecbf63                                                 |
| container_format              | bare                                                                             |
| created_at                    | 2021-06-01T10:22:44Z                                                             |
| direct_url                    | rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8b64-82ae7f |
|                               | 0e2bd2/snap                                                                      |
| disk_format                   | raw                                                                              |
| id                            | 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2                                             |
| locations                     | [{"url": "rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8 |
|                               | b64-82ae7f0e2bd2/snap", "metadata": {"store": "central"}}]                       |
| min_disk                      | 0                                                                                |
| min_ram                       | 0                                                                                |
| name                          | import_scenario                                                                  |
| os_glance_failed_import       |                                                                                  |
| os_glance_import_task         | e4bff141-1633-4d86-b118-43511813b74d                                             |
| os_glance_importing_to_stores |                                                                                  |
| os_hash_algo                  | sha512                                                                           |
| os_hash_value                 | a3fd523ada34f4e4a8d2cd1eeb94a6c1b05aabd63b3e0ff2c6522f3fd2f221d79569e8c2fffa92c2 |
|                               | 7d4a6e8e832771cb035c6d162f7a0192ae3896b168ecba51                                 |
| os_hidden                     | False                                                                            |
| owner                         | fbf93b9227c548938a91ed1b43507a69                                                 |
| protected                     | False                                                                            |
| size                          | 117440512                                                                        |
| status                        | active                                                                           |
| stores                        | central                                                                          |
| tags                          | []                                                                               |
| updated_at                    | 2021-06-01T10:22:52Z                                                             |
| virtual_size                  | 117440512                                                                        |
| visibility                    | shared                                                                           |
+-------------------------------+----------------------------------------------------------------------------------+
(central) [stack@site-undercloud-0 ~]$ glance image-import 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 --import-method copy-image --stores dcn2
HTTP 409 Conflict: Image has active task
(central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores
| os_glance_importing_to_stores |                                                                                  |
| stores                        | central,dcn1                                                                     |
(central) [stack@site-undercloud-0 ~]$ glance image-import 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 --import-method copy-image --stores dcn2
+-------------------------------+----------------------------------------------------------------------------------+
| Property                      | Value                                                                            |
+-------------------------------+----------------------------------------------------------------------------------+
| checksum                      | 318f4eb47a165b55db0be707e0ecbf63                                                 |
| container_format              | bare                                                                             |
| created_at                    | 2021-06-01T10:22:44Z                                                             |
| direct_url                    | rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8b64-82ae7f |
|                               | 0e2bd2/snap                                                                      |
| disk_format                   | raw                                                                              |
| id                            | 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2                                             |
| locations                     | [{"url": "rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8 |
|                               | b64-82ae7f0e2bd2/snap", "metadata": {"store": "central"}}, {"url": "rbd://e0eff2 |
|                               | f5-f00b-4888-b558-33b99b9a48b2/images/678ebcbd-0c23-4239-8b64-82ae7f0e2bd2/snap" |
|                               | , "metadata": {"store": "dcn1"}}]                                                |
| min_disk                      | 0                                                                                |
| min_ram                       | 0                                                                                |
| name                          | import_scenario                                                                  |
| os_glance_failed_import       |                                                                                  |
| os_glance_import_task         | 8a915808-e6fc-4b7b-a830-eacf99a960f7                                             |
| os_glance_importing_to_stores |                                                                                  |
| os_hash_algo                  | sha512                                                                           |
| os_hash_value                 | a3fd523ada34f4e4a8d2cd1eeb94a6c1b05aabd63b3e0ff2c6522f3fd2f221d79569e8c2fffa92c2 |
|                               | 7d4a6e8e832771cb035c6d162f7a0192ae3896b168ecba51                                 |
| os_hidden                     | False                                                                            |
| owner                         | fbf93b9227c548938a91ed1b43507a69                                                 |
| protected                     | False                                                                            |
| size                          | 117440512                                                                        |
| status                        | active                                                                           |
| stores                        | central,dcn1                                                                     |
| tags                          | []                                                                               |
| updated_at                    | 2021-06-01T10:23:42Z                                                             |
| virtual_size                  | 117440512                                                                        |
| visibility                    | shared                                                                           |
+-------------------------------+----------------------------------------------------------------------------------+
(central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores
| os_glance_importing_to_stores | dcn2                                                                             |
| stores                        | central,dcn1                                                                     |
(central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores
| os_glance_importing_to_stores | dcn2                                                                             |
| stores                        | central,dcn1                                                                     |
(central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores
| os_glance_importing_to_stores |                                                                                  |
| stores                        | central,dcn1,dcn2                                                                |
(central) [stack@site-undercloud-0 ~]$ 


VERIFIED

Comment 23 errata-xmlrpc 2021-09-15 07:08:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483