Bug 1851051
| Summary: | copy-image can start multiple importing threads due to race condition | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Abhishek Kekane <akekane> | |
| Component: | openstack-glance | Assignee: | Abhishek Kekane <akekane> | |
| Status: | CLOSED ERRATA | QA Contact: | Mike Abrams <mabrams> | |
| Severity: | high | Docs Contact: | RHOS Documentation Team <rhos-docs> | |
| Priority: | high | |||
| Version: | 16.1 (Train) | CC: | athomas, cyril, eglynn, gcharot, gfidente, gregraka, lmarsh, pgrist | |
| Target Milestone: | beta | Keywords: | Triaged | |
| Target Release: | 16.2 (Train on RHEL 8.4) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-glance-19.0.4-2.20210216215005.5bbd356.el8ost | Doc Type: | Bug Fix | |
| Doc Text: |
Before this update, RBD performance was degraded when multiple instances were launched simultaneously. This was due to the Image service (glance) starting multiple threads to perform the same copying operation. This update resolves the issue.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1852496 (view as bug list) | Environment: | ||
| Last Closed: | 2021-09-15 07:08:44 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1866592 | |||
| Bug Blocks: | 1852496 | |||
|
Description
Abhishek Kekane
2020-06-25 13:55:31 UTC
Is there some kind of workaround available right now? Or should the admin just be careful? Should we release an update to the documentation? (In reply to Cyril Roelandt from comment #1) > Is there some kind of workaround available right now? Or should the admin > just be careful? Should we release an update to the documentation? At the moment admin should be just careful to avoid this scenario. We should mention this in documentation as well. @Laura: can this be added as a warning in our documentation? yes, working on the wording now And also https://review.opendev.org/744024 Disregard #8 @Abhishek: the patches in the "Links" box are still required, right? These ones: https://review.opendev.org/737868/ https://review.opendev.org/743426/ https://review.opendev.org/743427/ https://review.opendev.org/743593/ https://review.opendev.org/743595/ https://review.opendev.org/743596/ https://review.opendev.org/743597/ https://review.opendev.org/743839/ (In reply to Cyril Roelandt from comment #10) > @Abhishek: the patches in the "Links" box are still required, right? These > ones: > > https://review.opendev.org/737868/ > https://review.opendev.org/743426/ > https://review.opendev.org/743427/ > https://review.opendev.org/743593/ > https://review.opendev.org/743595/ > https://review.opendev.org/743596/ > https://review.opendev.org/743597/ > https://review.opendev.org/743839/ Hi Cyril, Patches mentioned in comment #6 and #7 includes these patches. Actually patches in links box are from master branch and patches in comment #6 and #7 are upstream backports to ussuri. We need to backport patches from #6 and #7 as there are some changes in those backports not included from master branch as those were related to new feature. So it will be quiet easy for us if we consider those #6 and #7 patches only. Thank you, Abhishek to verify: === use copy-image method to copy to a store while a copy-image op is underway (should result in a 409 error) (central) [stack@site-undercloud-0 ~]$ glance image-import 3a0ddfd6-c4e2-41d5-915e-4d8a39aadb5e --import-method copy-image --stores dcn1; glance image-import 3a0ddfd6-c4e2-41d5-915e-4d8a39aadb5e --import-method copy-image --stores dcn2 (central) [stack@site-undercloud-0 ~]$ glance image-create-via-import --container-format ami --disk-format ami --name import_scenario --import-method web-download --uri https://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud-1510.qcow2 +-----------------------+--------------------------------------+ | Property | Value | +-----------------------+--------------------------------------+ | checksum | None | | container_format | ami | | created_at | 2021-06-01T08:48:41Z | | disk_format | ami | | id | cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 | | locations | [] | | min_disk | 0 | | min_ram | 0 | | name | import_scenario | | os_glance_import_task | 9b4fe0c9-b8d6-4b5b-9c09-60238c5173b8 | | os_hash_algo | None | | os_hash_value | None | | os_hidden | False | | owner | fbf93b9227c548938a91ed1b43507a69 | | protected | False | | size | None | | status | queued | | tags | [] | | updated_at | 2021-06-01T08:48:41Z | | virtual_size | Not available | | visibility | shared | +-----------------------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ (central) [stack@site-undercloud-0 ~]$ glance image-import cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 --import-method copy-image --stores dcn1 +-------------------------------+----------------------------------------------------------------------------------+ | Property | Value | +-------------------------------+----------------------------------------------------------------------------------+ | checksum | dcef39a9f88ef73cf782d9633d7e3451 | | container_format | bare | | created_at | 2021-06-01T08:48:41Z | | direct_url | rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/cb8d05cd-a19f-42f6-b2c9-7c16f1 | | | 4c4482/snap | | disk_format | raw | | id | cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 | | locations | [{"url": "rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/cb8d05cd-a19f-42f6-b | | | 2c9-7c16f14c4482/snap", "metadata": {"store": "central"}}] | | min_disk | 0 | | min_ram | 0 | | name | import_scenario | | os_glance_failed_import | | | os_glance_import_task | 7995f4b8-8342-49bf-932a-c45a74a044a7 | | os_glance_importing_to_stores | | | os_hash_algo | sha512 | | os_hash_value | 2ae24da25883b04af54f2e0edc65627772b2c4150108613e3d399718efab5b5353f5271e84ba5578 | | | 9fcdd19ab5ff4bac1ef4934f5c0f204400c9f8bb541dbe44 | | os_hidden | False | | owner | fbf93b9227c548938a91ed1b43507a69 | | protected | False | | size | 8589934592 | | status | active | | stores | central | | tags | [] | | updated_at | 2021-06-01T08:55:24Z | | virtual_size | 8589934592 | | visibility | shared | +-------------------------------+----------------------------------------------------------------------------------+ (central) [stack@site-undercloud-0 ~]$ === try to upload the image that is actively being uploaded to a third store: (central) [stack@site-undercloud-0 ~]$ glance image-import cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 --import-method copy-image --stores dcn2 HTTP 502 Proxy Error: Proxy Error: The proxy server received an invalid: response from an upstream server.: The proxy server could not handle the requestReason: Error reading from remote server (central) [stack@site-undercloud-0 ~]$ glance image-import cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 --import-method copy-image --stores dcn2 HTTP 409 Conflict: Image has active task (central) [stack@site-undercloud-0 ~]$ === ...409 expected === wait until the copy op is done to the second store: (central) [stack@site-undercloud-0 ~]$ while :; do glance image-show cb8d05cd-a19f-42f6-b2c9-7c16f14c4482 | grep stores; sleep 5; done | os_glance_importing_to_stores | dcn1 | | stores | central ... === once the original copy to store task is complete, should be able to copy to additional store without the 409 error; the whole process can be seen here: (central) [stack@site-undercloud-0 ~]$ glance image-create-via-import --container-format ami --disk-format ami --name import_scenario --import-method web-download --uri https://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img +-----------------------+--------------------------------------+ | Property | Value | +-----------------------+--------------------------------------+ | checksum | None | | container_format | ami | | created_at | 2021-06-01T10:22:44Z | | disk_format | ami | | id | 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | | locations | [] | | min_disk | 0 | | min_ram | 0 | | name | import_scenario | | os_glance_import_task | 93399714-52c2-4fc8-b37a-45383eb1c51f | | os_hash_algo | None | | os_hash_value | None | | os_hidden | False | | owner | fbf93b9227c548938a91ed1b43507a69 | | protected | False | | size | None | | status | queued | | tags | [] | | updated_at | 2021-06-01T10:22:44Z | | virtual_size | Not available | | visibility | shared | +-----------------------+--------------------------------------+ (central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep status | status | active | (central) [stack@site-undercloud-0 ~]$ glance image-import 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 --import-method copy-image --stores dcn1 +-------------------------------+----------------------------------------------------------------------------------+ | Property | Value | +-------------------------------+----------------------------------------------------------------------------------+ | checksum | 318f4eb47a165b55db0be707e0ecbf63 | | container_format | bare | | created_at | 2021-06-01T10:22:44Z | | direct_url | rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8b64-82ae7f | | | 0e2bd2/snap | | disk_format | raw | | id | 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | | locations | [{"url": "rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8 | | | b64-82ae7f0e2bd2/snap", "metadata": {"store": "central"}}] | | min_disk | 0 | | min_ram | 0 | | name | import_scenario | | os_glance_failed_import | | | os_glance_import_task | e4bff141-1633-4d86-b118-43511813b74d | | os_glance_importing_to_stores | | | os_hash_algo | sha512 | | os_hash_value | a3fd523ada34f4e4a8d2cd1eeb94a6c1b05aabd63b3e0ff2c6522f3fd2f221d79569e8c2fffa92c2 | | | 7d4a6e8e832771cb035c6d162f7a0192ae3896b168ecba51 | | os_hidden | False | | owner | fbf93b9227c548938a91ed1b43507a69 | | protected | False | | size | 117440512 | | status | active | | stores | central | | tags | [] | | updated_at | 2021-06-01T10:22:52Z | | virtual_size | 117440512 | | visibility | shared | +-------------------------------+----------------------------------------------------------------------------------+ (central) [stack@site-undercloud-0 ~]$ glance image-import 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 --import-method copy-image --stores dcn2 HTTP 409 Conflict: Image has active task (central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores | os_glance_importing_to_stores | | | stores | central,dcn1 | (central) [stack@site-undercloud-0 ~]$ glance image-import 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 --import-method copy-image --stores dcn2 +-------------------------------+----------------------------------------------------------------------------------+ | Property | Value | +-------------------------------+----------------------------------------------------------------------------------+ | checksum | 318f4eb47a165b55db0be707e0ecbf63 | | container_format | bare | | created_at | 2021-06-01T10:22:44Z | | direct_url | rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8b64-82ae7f | | | 0e2bd2/snap | | disk_format | raw | | id | 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | | locations | [{"url": "rbd://62cc482d-756a-407c-868a-4700d73a0cba/images/678ebcbd-0c23-4239-8 | | | b64-82ae7f0e2bd2/snap", "metadata": {"store": "central"}}, {"url": "rbd://e0eff2 | | | f5-f00b-4888-b558-33b99b9a48b2/images/678ebcbd-0c23-4239-8b64-82ae7f0e2bd2/snap" | | | , "metadata": {"store": "dcn1"}}] | | min_disk | 0 | | min_ram | 0 | | name | import_scenario | | os_glance_failed_import | | | os_glance_import_task | 8a915808-e6fc-4b7b-a830-eacf99a960f7 | | os_glance_importing_to_stores | | | os_hash_algo | sha512 | | os_hash_value | a3fd523ada34f4e4a8d2cd1eeb94a6c1b05aabd63b3e0ff2c6522f3fd2f221d79569e8c2fffa92c2 | | | 7d4a6e8e832771cb035c6d162f7a0192ae3896b168ecba51 | | os_hidden | False | | owner | fbf93b9227c548938a91ed1b43507a69 | | protected | False | | size | 117440512 | | status | active | | stores | central,dcn1 | | tags | [] | | updated_at | 2021-06-01T10:23:42Z | | virtual_size | 117440512 | | visibility | shared | +-------------------------------+----------------------------------------------------------------------------------+ (central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores | os_glance_importing_to_stores | dcn2 | | stores | central,dcn1 | (central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores | os_glance_importing_to_stores | dcn2 | | stores | central,dcn1 | (central) [stack@site-undercloud-0 ~]$ glance image-show 678ebcbd-0c23-4239-8b64-82ae7f0e2bd2 | grep stores | os_glance_importing_to_stores | | | stores | central,dcn1,dcn2 | (central) [stack@site-undercloud-0 ~]$ VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483 |