Bug 844184 - 3.1.z - [vdsm] domain is not found if meta-data is altered and produces StorageDomainDoesNotExist which prevents pool reconnection
3.1.z - [vdsm] domain is not found if meta-data is altered and produces Stora...
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: vdsm (Show other bugs)
6.3
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Yeela Kaplan
Haim
storage
:
Depends On:
Blocks: 835824
  Show dependency treegraph
 
Reported: 2012-07-29 12:14 EDT by Haim
Modified: 2016-02-02 17:39 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-19 05:36:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine and vdsm logs (8.91 MB, application/x-tar)
2012-07-29 12:16 EDT, Haim
no flags Details

  None (edit)
Description Haim 2012-07-29 12:14:19 EDT
Description of problem:

host connected to storage pool with 1 storage domain which is master domain.
alter domain meta-data cause meta-data seal broken.
getStorageDomaininfo or getStorageDomainsList fails with the following:

Thread-111::ERROR::2012-07-29 22:29:41,049::sdc::145::Storage.StorageDomainCache::(_findDomain) Error while looking for domain `a3ceac0d-a435-4306-bd61-2465ef9525f9`
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 140, in _findDomain
    return mod.findDomain(sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 1093, in findDomain
    return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/blockSD.py", line 308, in __init__
    lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
  File "/usr/share/vdsm/storage/lvm.py", line 909, in checkVGBlockSizes
    _checkpvsblksize(pvs, vgBlkSize)
  File "/usr/share/vdsm/storage/lvm.py", line 887, in _checkpvsblksize
    pvBlkSize = _getpvblksize(pv)
  File "/usr/share/vdsm/storage/lvm.py", line 882, in _getpvblksize
    dev = devicemapper.getDmId(os.path.basename(pv))
  File "/usr/share/vdsm/storage/devicemapper.py", line 37, in getDmId
    raise OSError(errno.ENODEV, "Could not find dm device named `%s`" % deviceMultipathName)
OSError: [Errno 19] Could not find dm device named `unknown device`
Thread-111::ERROR::2012-07-29 22:29:41,055::hsm::2349::Storage.HSM::(getStorageDomainsList) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2335, in getStorageDomainsList
    dom = sdCache.produce(sdUUID=sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 110, in produce
    dom.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce
    dom = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('a3ceac0d-a435-4306-bd61-2465ef9525f9',)


domain is indeed valid:

  4f9154d7-0964-46db-8d88-15ca812eb086   1   7   0 wz--n- 74.62g 69.75g MDT_CLASS=Data,MDT_VERSION=3,MDT_TYPE=ISCSI,MDT_LOCKRENEWALINTERVALSEC=5,MDT_LOGBLKSIZE=512,MDT_LEASERETRIES=3,MDT_SDUUID=4f9154d7-0964-46db-8d88-15ca812eb086,MDT_PHYBLKSIZE=512,MDT_VGUUID=QP6oLc-hk52-YyJ9-SKQc-qTAk-WgEz-md9CEQ,MDT_PV0=pv:3514f0c5d66200271&44&uuid:Fa0wiu-1rcR-m5Qb-fzSL-erO8-jS6F-PH4pv2&44&pestart:0&44&pecount:597&44&mapoffset:0,MDT_DESCRIPTION=DEMO_STORAGE,MDT_LOCKPOLICY=,RHAT_storage_domain,MDT_LEASETIMESEC=60,MDT_IOOPTIMEOUTSEC=10,MDT_ROLE=Master,MDT_POOL_DESCRIPTION=DEMO_POOL,MDT_POOL_UUID=77887fd1-3408-44ec-8de4-153948d89cdc,MDT_POOL_SPM_ID=2,MDT_POOL_SPM_LVER=2,MDT__SHA_CKSUM=83e39ea4c36bf072fa1155baad1c30163840b44b,MDT_POOL_DOMAINS=4f9154d7-0964-46db-8d88-15ca812eb086:Active&44&4ac437ad-560b-4d93-a22a-289b149c22e2:Attached,MDT_MASTER_VERSION=-1

[root@nott-vds2 ~]# vdsClient -s 0 getStorageDomainInfo 4f9154d7-0964-46db-8d88-15ca812eb086
Storage domain does not exist: ('4f9154d7-0964-46db-8d88-15ca812eb086',)

if its indeed a problem with cache, we should either deal with it with vdsm kill itself, or re-read the cache.
Comment 1 Haim 2012-07-29 12:16:29 EDT
Created attachment 601057 [details]
engine and vdsm logs
Comment 3 Laszlo Hornyak 2012-10-11 06:03:23 EDT
Haim, how do you alter the storage domain metadata?
Comment 4 Haim 2012-10-11 07:29:39 EDT
(In reply to comment #3)
> Haim, how do you alter the storage domain metadata?

block device (iSCSI\FC):

- each storage domain is considered as a VG (in LVM), thus, metadata is written as a tag inside the VG, so in order to change it, we will use the vgchange <vgUUID> --deltag ',MDT__SHA_CKSUM=266fedb0a788c18c5f1e580f744be098756b154e' --addtag ',MDT__SHA_CKSUM=test'

in order to get a list of all the tags in storage domain, run the following command:

vgs 402471a6-154f-43a3-b9fe-5308024c33a6 -o+tags

402471a6-154f-43a3-b9fe-5308024c33a6   2   7   0 wz--n- 99.75g 92.88g RHAT_storage_domain,MDT_IOOPTIMEOUTSEC=,MDT_ROLE=Regular,MDT_POOL_SPM_ID=-1,MDT_VERSION=3,MDT_TYPE=ISCSI,MDT_VGUUID=GDXWuw-v5Lm-LasB-5JBR-ddlH-pBRd-3VmZhl,MDT_LOGBLKSIZE=512,MDT_LEASETIMESEC=,MDT__SHA_CKSUM=266fedb0a788c18c5f1e580f744be098756b154e,MDT_POOL_SPM_LVER=-1,MDT_PV0=pv:1Dafna-041349352&44&uuid:ZdF3DC-n7rP-mMcI-1ei7-8BdE-Vq34-apeXAU&44&pestart:0&44&pecount:399&44&mapoffset:0,MDT_MASTER_VERSION=0,MDT_LOCKPOLICY=,MDT_LOCKRENEWALINTERVALSEC=,MDT_LEASERETRIES=,MDT_POOL_UUID=11d18980-5c97-40ca-b7ff-6d1fa0f01cc8,MDT_PHYBLKSIZE=512,MDT_SDUUID=402471a6-154f-43a3-b9fe-5308024c33a6,MDT_CLASS=Data,MDT_DESCRIPTION=Dafna-01,MDT_PV1=pv:1Dafna-011349352&44&uuid:bKfCLy-8NPZ-PEys-7C83-YuMc-zbCG-23dAB0&44&pestart:0&44&pecount:399&44&mapoffset:399
Comment 5 Ayal Baron 2012-12-19 05:36:09 EST
Discussed with Haim and Kaul, closing wontfix.  The use case is not interesting.

Note You need to log in before you can comment on or make changes to this bug.