Bug 1446492

(In reply to nijin ashok from comment #0)
> Description of problem:
> 
> If the customer's LVM was manually restored without --metadatacopies switch,
> then it will fail to activate in RHV 4.1 as we are having new check as a
> part of StorageDomain.getInfo flow ( https://gerrit.ovirt.org/#/c/64433/ )
> where vdsm reports PV which is having active LVM metadata to engine.

This is expected, RHV supports only RHV storage domain format, and this
storage domain is in the correct format.

> As we can't increase the number of metadata copies after creating PV,  then
> the only option will be to restore the metadata again with correct options
> which needs a complete downtime of the VMs.

Yes, this is what should be done, recreate the vg with the correct options.

> Also  vdsm is only expecting 1 PV with active metadata and others should be
> disabled with pvchange --metadataignore y or else it will fail with same
> error as above even if it's created with 2 metadata copies.

Right, this is also part of the format.

> Expected results:
> 
> Don't fail the storage domain if mda_used_count is 1 for the PV.

I don't think this expectation is feasible.

We never tested this format, and we cannot guarantee that anything
will work with such storage domain. The best thing we can do is to fail
to activate this storage domain.

The new checks are required for the removing pvs from storage domain
(new feature in 4.1). We can consider not failing to activate such storage
domain, and disable this feature, but I don't see how we can support
such system.

Liron, can engine treat the new metadata keys as optional, and disable
removing pvs (with a warning) if the info is not available?

(In reply to Nir Soffer from comment #6)
> Liron, can engine treat the new metadata keys as optional, and disable
> removing pvs (with a warning) if the info is not available?

A quick review of the engine's code seems to show that this is indeed the behavior. I think we're OK from the engine's side.

I reproduced this issue by modifying vdsm to not use the --metadatacopies and
--metadaignore options when creating a storage domain, so it creates an invalid
storage domain with the same configuration as badly restored storage domain.

This is how vdsm log looks now when we find such storage domain:

1. Starting getStorageDomainInfo requets

2017-05-08 15:24:56,054+0300 INFO  (jsonrpc/5) [dispatcher] Run and protect: getStorageDomainInfo(sdUUID=u'e529906a-f20f-4ad7-99e6-20242678a58e', options=None) (logUtils:51)

2. Warning about unsupported storage domain:

2930 2017-05-08 15:24:56,258+0300 WARN  (jsonrpc/5) [storage.StorageDomain] Cannot get metadata device, this storage domain is unsupported: Volume Group metadata isn't as expected: u"reason=     Expected one metadata pv in vg: e529906a-f20f-4ad7-99e6-20242678a58e, vg pvs: [PV(uuid='CTcOe0-uqQJ-c3lk-02SV-PYGf-cPvS-fPBjZ8', name='/dev/mapper/360014052e489b6f5ed34881ac5ed27fd', si     ze='53418655744', vg_name='e529906a-f20f-4ad7-99e6-20242678a58e', vg_uuid='CXMs11-xiGr-JzAU-5PtP-PW1j-bkjO-2jKlgc', pe_start='142606336', pe_count='398', pe_alloc_count='39', mda_count=     '1', dev_size='53687091200', mda_used_count='1', guid='360014052e489b6f5ed34881ac5ed27fd'), PV(uuid='hGmFCu-i6Hm-LylO-EFLX-I5p2-hxWW-GyoTfZ', name='/dev/mapper/360014052de8e2b8007944a1a     93a82c40', size='53418655744', vg_name='e529906a-f20f-4ad7-99e6-20242678a58e', vg_uuid='CXMs11-xiGr-JzAU-5PtP-PW1j-bkjO-2jKlgc', pe_start='142606336', pe_count='398', pe_alloc_count='0'     , mda_count='1', dev_size='53687091200', mda_used_count='1', guid='360014052de8e2b8007944a1a93a82c40'), PV(uuid='bWau5Z-C5Vb-Vt0G-Wjwp-2gYP-8X6m-Unocp3', name='/dev/mapper/36001405bfc49     549313946daa720cb1ba', size='53418655744', vg_name='e529906a-f20f-4ad7-99e6-20242678a58e', vg_uuid='CXMs11-xiGr-JzAU-5PtP-PW1j-bkjO-2jKlgc', pe_start='142606336', pe_count='398', pe_all     oc_count='0', mda_count='1', dev_size='53687091200', mda_used_count='1', guid='36001405bfc49549313946daa720cb1ba')]" (blockSD:1243)                                                      

3. Returning response without vgMetadataDevice key:

2931 2017-05-08 15:24:56,258+0300 INFO  (jsonrpc/5) [dispatcher] Run and protect: getStorageDomainInfo, Return response: {'info': {'uuid': u'e529906a-f20f-4ad7-99e6-20242678a58e', 'vguuid':      'CXMs11-xiGr-JzAU-5PtP-PW1j-bkjO-2jKlgc', 'metadataDevice': '360014052e489b6f5ed34881ac5ed27fd', 'state': 'OK', 'version': '4', 'role': 'Regular', 'type': 'ISCSI', 'class': 'Data', 'poo     l': [], 'name': 'bad-sd'}} (logUtils:54)

Testing this change:
1. Create an invalid storage domain with one used metadata area on every PV
2. Run StorageDomain getInfo:

# vdsm-client StorageDomain getInfo storagedomainID=e529906a-f20f-4ad7-99e6-20242678a58e
{
    "uuid": "e529906a-f20f-4ad7-99e6-20242678a58e", 
    "vguuid": "CXMs11-xiGr-JzAU-5PtP-PW1j-bkjO-2jKlgc", 
    "metadataDevice": "360014052e489b6f5ed34881ac5ed27fd", 
    "state": "OK", 
    "version": "4", 
    "role": "Regular", 
    "type": "ISCSI", 
    "class": "Data", 
    "pool": [
        "6c99f4e5-8588-46f5-a818-e11151c1d19c"
    ], 
    "name": "bad-sd"
}

The call should succeed, not returning the vgMetadataDevice key.

Here is the same request for a good sd:

# vdsm-client StorageDomain getInfo storagedomainID=aed577ea-d1ca-4ebe-af80-f852c7ce59bb
{
    "uuid": "aed577ea-d1ca-4ebe-af80-f852c7ce59bb", 
    "type": "ISCSI", 
    "vguuid": "7T9sFi-okfz-JZON-xDUK-n0vH-OpyH-L7IjKO", 
    "metadataDevice": "360014052761af2654a94a70a60a7ee3f", 
    "state": "OK", 
    "version": "4", 
    "role": "Master", 
    "vgMetadataDevice": "360014052761af2654a94a70a60a7ee3f", 
    "class": "Data", 
    "pool": [
        "6c99f4e5-8588-46f5-a818-e11151c1d19c"
    ], 
    "name": "dumbo-iscsi-01"
}

WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found clone flags: ['rhevm-4.1.z', 'rhevm-4.2-ga'], ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found clone flags: ['rhevm-4.1.z', 'rhevm-4.2-ga'], ]

For more info please contact: rhv-devops

Tal, this bug should be ready for QA, but the bot is complaning about the flags,
see comment 16. Can you help with this?

Verified with the following code:
---------------------------------------
ovirt-engine-4.2.0-0.0.master.20170523140304.git04be891.el7.centos.noarch
vdsm-4.20.0-886.gitf9accf8.el7.centos.x86_64

Verified with the following scenario:
---------------------------------------
Modified the lvm.py file on the vdsm in order to create an invalid storage domain with one used metadata area on every PV as follows:

------------------------------------------
diff -uab lvm.py.orig lvm.py.bad
--- lvm.py.orig	2017-05-28 16:13:26.855334099 +0300
+++ lvm.py.bad	2017-05-28 18:06:45.883392405 +0300
@@ -722,9 +722,7 @@
     if options:
         cmd.extend(options)
     if metadataSize != 0:
-        cmd.extend(("--metadatasize", "%sm" % metadataSize,
-                    "--metadatacopies", "2",
-                    "--metadataignore", "y"))
+        cmd.extend(("--metadatasize", "%sm" % metadataSize))
     cmd.extend(devices)
     rc, out, err = _lvminfo.cmd(cmd, devices)
     return rc, out, err
@@ -984,12 +982,6 @@
     _checkpvsblksize(pvs)
 
     _initpvs(pvs, metadataSize, force)
-    # Activate the 1st PV metadata areas
-    cmd = ["pvchange", "--metadataignore", "n"]
-    cmd.append(pvs[0])
-    rc, out, err = _lvminfo.cmd(cmd, tuple(pvs))
-    if rc != 0:
-        raise se.PhysDevInitializationError(pvs[0])
 
     options = ["--physicalextentsize", "%dm" % VG_EXTENT_SIZE_MB]
     if initialTag:
------------------------------------------

2. Created a new block domain
3. Verified the warning in the vdsm.log as follows:

2017-05-28 18:07:44,672+0300 WARN  (jsonrpc/5) [storage.StorageDomain] Cannot get VG metadata device, this storage domain is unsupported: Volume Group metadata isn't as expected: "reason=Expected one metadata pv in vg: 8985b8d5-404c-4de8-b2d9-d8466b06ab77, vg pvs: [PV(uuid='4RJO7I-xzm0-VTKs-Lday-S1cl-UdrC-5PKa6V', name='/dev/mapper/3514f0c5a51600676', size='53418655744', vg_name='8985b8d5-404c-4de8-b2d9-d8466b06ab77', vg_uuid='Mnz1Dx-Vkeh-ibX0-BcHU-T2uu-1uht-9TucUe', pe_start='135266304', pe_count='398', pe_alloc_count='71', mda_count='1', dev_size='53687091200', mda_used_count='1', guid='3514f0c5a51600676')]" (blockSD:1243)

4. Verified creation of a new VM and disk on the invalid domain


Moving to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1489

BZ<2>Jira Resync