853748 – [vdsm] unable to detach block storage domain in partial state

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 853748 - [vdsm] unable to detach block storage domain in partial state

Summary: [vdsm] unable to detach block storage domain in partial state

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	vdsm
Sub Component:
Version:	6.3
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Dan Kenigsberg
QA Contact:	Haim
Docs Contact:
URL:
Whiteboard:	storage
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-09-02 15:39 UTC by vvyazmin@redhat.com
Modified:	2014-01-13 00:54 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2012-09-03 06:37:10 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
## Logs vdsm, rhevm (2.10 MB, application/x-gzip) 2012-09-02 15:39 UTC, vvyazmin@redhat.com	no flags	Details
View All

Description vvyazmin@redhat.com 2012-09-02 15:39:45 UTC

Created attachment 609124 [details]
## Logs vdsm, rhevm

Description of problem:
unable to deactivate block storage domain in partial state


Version-Release number of selected component (if applicable):
Verified on RHEVM 3.1 - SI16

RHEVM: rhevm-3.1.0-14.el6ev.noarch
VDSM: vdsm-4.9.6-31.0.el6_3.x86_64
LIBVIRT: libvirt-0.9.10-21.el6_3.4.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.295.el6_3.1.x86_64
SANLOCK: sanlock-2.3-3.el6_3.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create DC iSCSI with 2 hosts.
2. Create first Storage Domain (Master) SD-01
3. Create second SD-02 from 4 LUN's (example: LUN-01, LUN-02, LUN-03, LUN-04)
4. Create third SD-03 from 4 LUN's, but one of LUN's used in second SD (example: LUN-05, LUN-06, LUN-03, LUN-07) – accept to warning
5. “Maintenance” and than “Detach” third SD-03
6. “Maintenance” and than “Detach” second SD-02
  
Actual results:
Failed “Detach” second SD-02
On “Detach” second SD-02 action SPM move from host to host

Expected results:
Succeed “Detach” second SD-02

Additional info:

[root@cougar08 ~]# vgs
  Couldn't find device with uuid JsoOxl-4axo-oHAl-B18k-Q7kl-RTdj-KsV2kf.
  VG                                   #PV #LV #SN Attr   VSize   VFree 
  d0727c0c-4542-4319-9132-d0f0017f82be   4   6   0 wz--n-  18.50g 14.62g
  da68e840-61b2-4e7a-b4b3-92810b24a4af   4   6   0 wz-pn-  18.50g 14.62g
  f386dedb-24f9-465c-8718-099d026fbf8c   1   9   0 wz--n-  49.62g 36.75g
  vg0                                    1   3   0 wz--n- 465.27g     0 
[root@cougar08 ~]# vgs  d0727c0c-4542-4319-9132-d0f0017f82be da68e840-61b2-4e7a-b4b3-92810b24a4af -o+pv_name
  Couldn't find device with uuid JsoOxl-4axo-oHAl-B18k-Q7kl-RTdj-KsV2kf.
  VG                                   #PV #LV #SN Attr   VSize  VFree  PV                           
  d0727c0c-4542-4319-9132-d0f0017f82be   4   6   0 wz--n- 18.50g 14.62g /dev/mapper/3514f0c56958002bb
  d0727c0c-4542-4319-9132-d0f0017f82be   4   6   0 wz--n- 18.50g 14.62g /dev/mapper/3514f0c56958002c9
  d0727c0c-4542-4319-9132-d0f0017f82be   4   6   0 wz--n- 18.50g 14.62g /dev/mapper/3514f0c56958002bc
  d0727c0c-4542-4319-9132-d0f0017f82be   4   6   0 wz--n- 18.50g 14.62g /dev/mapper/3514f0c56958002c2
  da68e840-61b2-4e7a-b4b3-92810b24a4af   4   6   0 wz-pn- 18.50g 14.62g /dev/mapper/3514f0c56958002ba
  da68e840-61b2-4e7a-b4b3-92810b24a4af   4   6   0 wz-pn- 18.50g 14.62g /dev/mapper/3514f0c56958002c1
  da68e840-61b2-4e7a-b4b3-92810b24a4af   4   6   0 wz-pn- 18.50g 14.62g unknown device               
  da68e840-61b2-4e7a-b4b3-92810b24a4af   4   6   0 wz-pn- 18.50g 14.62g /dev/mapper/3514f0c56958002bd


Thread-10740::ERROR::2012-09-02 18:23:04,275::sdc::145::Storage.StorageDomainCache::(_findDomain) Error while looking for domain `da68e840-61b2-4e7a-b4b3-92810b24a4af`
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 140, in _findDomain
    return mod.findDomain(sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 1095, in findDomain
    return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/blockSD.py", line 308, in __init__
    lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
  File "/usr/share/vdsm/storage/lvm.py", line 909, in checkVGBlockSizes
    _checkpvsblksize(pvs, vgBlkSize)
  File "/usr/share/vdsm/storage/lvm.py", line 887, in _checkpvsblksize
    pvBlkSize = _getpvblksize(pv)
  File "/usr/share/vdsm/storage/lvm.py", line 882, in _getpvblksize
    dev = devicemapper.getDmId(os.path.basename(pv))
  File "/usr/share/vdsm/storage/devicemapper.py", line 37, in getDmId
    raise OSError(errno.ENODEV, "Could not find dm device named `%s`" % deviceMultipathName)
OSError: [Errno 19] Could not find dm device named `unknown device`
Thread-10740::ERROR::2012-09-02 18:23:04,282::task::853::TaskManager.Task::(_setError) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 861, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 638, in detachStorageDomain
    pool.detachSD(sdUUID)
  File "/usr/share/vdsm/storage/securable.py", line 63, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 986, in detachSD
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 110, in produce
    dom.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce
    dom = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('da68e840-61b2-4e7a-b4b3-92810b24a4af',)
Thread-10740::DEBUG::2012-09-02 18:23:04,283::task::872::TaskManager.Task::(_run) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::Task._run: 8598e46d-1f54-46b9-9592-bec943a22833 ('da68e840-61b2-4e7a-b4b3-92810b24a4af', '8f310cee-22b4-482d-b5b2-6f5d3131dd67', '00000000-0000-0000-0000-000000000000', 1) {} failed - stopping task
Thread-10740::DEBUG::2012-09-02 18:23:04,283::task::1199::TaskManager.Task::(stop) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::stopping in state preparing (force False)
Thread-10740::DEBUG::2012-09-02 18:23:04,284::task::978::TaskManager.Task::(_decref) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::ref 1 aborting True
Thread-10740::INFO::2012-09-02 18:23:04,284::task::1157::TaskManager.Task::(prepare) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
Thread-10740::DEBUG::2012-09-02 18:23:04,284::task::1162::TaskManager.Task::(prepare) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::Prepare: aborted: Storage domain does not exist
Thread-10740::DEBUG::2012-09-02 18:23:04,284::task::978::TaskManager.Task::(_decref) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::ref 0 aborting True
Thread-10740::DEBUG::2012-09-02 18:23:04,285::task::913::TaskManager.Task::(_doAbort) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::Task._doAbort: force False
Thread-10740::DEBUG::2012-09-02 18:23:04,285::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-10740::DEBUG::2012-09-02 18:23:04,285::task::588::TaskManager.Task::(_updateState) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::moving from state preparing -> state aborting
Thread-10740::DEBUG::2012-09-02 18:23:04,286::task::537::TaskManager.Task::(__state_aborting) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::_aborting: recover policy none
Thread-10740::DEBUG::2012-09-02 18:23:04,286::task::588::TaskManager.Task::(_updateState) Task=`8598e46d-1f54-46b9-9592-bec943a22833`::moving from state aborting -> state failed
Thread-10740::DEBUG::2012-09-02 18:23:04,286::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da68e840-61b2-4e7a-b4b3-92810b24a4af': < ResourceRef 'Storage.da68e840-61b2-4e7a-b4b3-92810b24a4af', isValid: 'True' obj: 'None'>, 'Storage.8f310cee-22b4-482d-b5b2-6f5d3131dd67': < ResourceRef 'Storage.8f310cee-22b4-482d-b5b2-6f5d3131dd67', isValid: 'True' obj: 'None'>}
Thread-10740::DEBUG::2012-09-02 18:23:04,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-10740::DEBUG::2012-09-02 18:23:04,287::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da68e840-61b2-4e7a-b4b3-92810b24a4af'
Thread-10740::DEBUG::2012-09-02 18:23:04,288::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da68e840-61b2-4e7a-b4b3-92810b24a4af' (0 active users)
Thread-10740::DEBUG::2012-09-02 18:23:04,288::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da68e840-61b2-4e7a-b4b3-92810b24a4af' is free, finding out if anyone is waiting for it.
Thread-10740::DEBUG::2012-09-02 18:23:04,288::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da68e840-61b2-4e7a-b4b3-92810b24a4af', Clearing records.
Thread-10740::DEBUG::2012-09-02 18:23:04,289::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.8f310cee-22b4-482d-b5b2-6f5d3131dd67'
Thread-10740::DEBUG::2012-09-02 18:23:04,289::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.8f310cee-22b4-482d-b5b2-6f5d3131dd67' (0 active users)
Thread-10740::DEBUG::2012-09-02 18:23:04,289::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.8f310cee-22b4-482d-b5b2-6f5d3131dd67' is free, finding out if anyone is waiting for it.
Thread-10740::DEBUG::2012-09-02 18:23:04,290::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.8f310cee-22b4-482d-b5b2-6f5d3131dd67', Clearing records.
Thread-10740::ERROR::2012-09-02 18:23:04,290::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('da68e840-61b2-4e7a-b4b3-92810b24a4af',)", 'code': 358}}

Comment 2 Ayal Baron 2012-09-03 06:37:10 UTC

To detach a faulty domain one must use forcedDetachStorageDomain

Note You need to log in before you can comment on or make changes to this bug.