Bug 1713724
Summary: | When a storage domain is updated to V5 during a DC upgrade, if there are volumes with metadata that has been reset then the upgrade fails | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Gordon Watson <gwatson> | ||||
Component: | vdsm | Assignee: | Nir Soffer <nsoffer> | ||||
Status: | CLOSED ERRATA | QA Contact: | Evelina Shames <eshames> | ||||
Severity: | urgent | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 4.3.1 | CC: | aefrat, emarcus, gveitmic, izuckerm, lsurette, mtessun, nsoffer, rdlugyhe, srevivo, tnisan, ycui | ||||
Target Milestone: | ovirt-4.4.0 | Flags: | izuckerm:
testing_plan_complete+
|
||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: |
Previously, converting a storage domain to the V5 format failed when, following an unsuccessful delete volume operation, partly-deleted volumes with cleared metadata remained in the storage domain. The current release fixes this issue. Converting a storage domain succeeds even when partly-deleted volumes with cleared metadata remain in the storage domain.
|
Story Points: | --- | ||||
Clone Of: | |||||||
: | 1714154 (view as bug list) | Environment: | |||||
Last Closed: | 2020-08-04 13:27:06 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1714154 | ||||||
Attachments: |
|
Description
Gordon Watson
2019-05-24 15:25:59 UTC
Gordon, can you attach: - lvm commands output from sosreport? - copy of the first MiB of the metadata LV from a domain with this issue. The attached patch should fix this issue, but it is not tested yet. I can test in on Sunday, but we can save time if you can test it during the weekend. ## Testing the fix 1. Create a DC/cluster with compatibility version 4.2 2. Create new iSCSI/FC V4 storage domain 3. Add some floating disks 4. Clear the metadata of one or more disks (see "How to clear volume metadata bellow") 5. Upgrade the cluster and DC to version 4.3 Expected result: - Storage domain upgraded to V5 successfully - Warning about volume with cleared metadata logged for the volume with this issue Should be tested with with 4.3.3, reproducing this issue, and then with a build including this fix, verifying that upgrading storage domain to V5 succeeds. ## How to clear volume metadata This is not possible since 4.2.5, and impossible to reproduce to older version, so we have to clear the metadata manually: 1. Find the volume metadata slot using: lvs -o vg_name,lv_name,tags -> MD_42 2. Write cleared metadata to the volume metadata area import os with open("/dev/vg-name/metadata, "rb+") as f: f.seek(42 * 512) f.write(b"NONE=" + (b"#" * 502) + b"\nEOF\n") os.fsync(f.fileno()) Trying to target to 4.3.4, since this issue breaks upgrades to 4.3, and does not have an easy workaround. Also test installing the temp fix vdsm-4.30.16-2.gitfb7cdef.el7.x86_64 before upgrading the CD to 4.3 and at looks well. SD/DC upgraded to V5/4.3 without issues and all hosts are up. Turns out we had the same issue 6.5 years ago, see bug 902838. Cleaning after bad gerrit script, adding unrelated patches. Cleaning again after bad gerrit hook. Looking in the metadata lv content from attachment 1573377 [details], we see 4 volumes
with cleared metadata out of 53 volumes.
volumes: 53
slot 10 is cleared
lv: 793df8d3-6853-4408-afa0-abe414f652ee
size: 35.00g
tags: IU_da733dfb-491b-4d59-a2df-b2008e36de16,MD_10,PU_00000000-0000-0000-0000-000000000000
slot 11 is cleared
lv: 772c45c4-35b9-4133-b603-4eee1c33208d
size: 35.00g
tags: IU_aaf92ffa-2344-4617-be60-e7ecf5baf2e8,MD_11,PU_00000000-0000-0000-0000-000000000000
slot 12 is cleared
lv: bdc029ac-c858-418c-ae90-072bc8f1ddaf
size: 35.00g
tags: IU_e7c4f50c-465e-476d-9360-762f2fea73d9,MD_12,PU_00000000-0000-0000-0000-000000000000
slot 14 is cleared
lv: a25e9312-71a3-4d06-8890-5034c56fd895
size: 35.00g
tags: IU_8683a8e2-2af2-4ae2-8c2d-8fd2aa7d4cdf,MD_14,PU_00000000-0000-0000-0000-000000000000
Created attachment 1574397 [details]
script for analyzing metadata area
sync2jira sync2jira WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops Verified (steps in comment #9) on engine-4.4.0-0.13.master.el7 WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHV RHEL Host (ovirt-host) 4.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:3246 |