Bug 722649

Summary: VDSM: CannotFormatAttachedStorageDomain error message when trying to remove data domain ( format=True )
Product: Red Hat Enterprise Linux 6 Reporter: Jaroslav Henner <jhenner>
Component: vdsmAssignee: Saggi Mizrahi <smizrahi>
Status: CLOSED ERRATA QA Contact: Tomas Dosek <tdosek>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1CC: abaron, bazulay, iheim, oramraz, tdosek, ykaul
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: vdsm-4.9-93 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 07:31:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
logs none

Description Jaroslav Henner 2011-07-16 07:02:19 UTC
Description of problem:
Sometimes, I'm given following error message when trying to remove and format the data domain with host A.  With host B it works fine. No host is SPM when trying this and error message appears.

Version-Release number of selected component (if applicable):
vdsm-4.9-81.el6.x86_64

How reproducible:
20%

Steps to Reproduce:
1. Have a data domain.
2. Remove it

  
Actual results:
vdsm.log

Thread-486::DEBUG::2011-07-16 08:48:53,883::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: moving from state init -> state preparing
Thread-486::DEBUG::2011-07-16 08:48:53,883::resourceManager::154::ResourceManager.Request::(__init__) ResName=`Storage.bb564256-e768-4dde-9e24-ba3fa9738b93`ReqID=`f8660b7b-eb3d-455e-a4a2-1cbc50553397`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1209' at 'public_formatStorageDomain'
Thread-486::DEBUG::2011-07-16 08:48:53,883::resourceManager::467::ResourceManager::(registerResource) Trying to register resource 'Storage.bb564256-e768-4dde-9e24-ba3fa9738b93' for lock type 'exclusive'
Thread-486::DEBUG::2011-07-16 08:48:53,884::resourceManager::508::ResourceManager::(registerResource) Resource 'Storage.bb564256-e768-4dde-9e24-ba3fa9738b93' is free. Now locking as 'exclusive' (1 active user)
Thread-486::DEBUG::2011-07-16 08:48:53,884::resourceManager::191::ResourceManager.Request::(grant) ResName=`Storage.bb564256-e768-4dde-9e24-ba3fa9738b93`ReqID=`f8660b7b-eb3d-455e-a4a2-1cbc50553397`::Granted request
Thread-486::DEBUG::2011-07-16 08:48:53,884::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: _resourcesAcquired: Storage.bb564256-e768-4dde-9e24-ba3fa9738b93 (exclusive)
Thread-486::DEBUG::2011-07-16 08:48:53,885::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: ref 1 aborting False
Thread-486::ERROR::2011-07-16 08:48:53,885::task::865::TaskManager.Task::(_setError) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1221, in public_formatStorageDomain
    raise se.CannotFormatAttachedStorageDomain(sdUUID)
CannotFormatAttachedStorageDomain: Cannot format attached storage domain: ('bb564256-e768-4dde-9e24-ba3fa9738b93',)
Thread-486::DEBUG::2011-07-16 08:48:53,886::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: Task._run: 8f9e7925-6558-4e22-af72-482b2c6b93ce ('bb564256-e768-4dde-9e24-ba3fa9738b93',) {} failed - stopping task
Thread-486::DEBUG::2011-07-16 08:48:53,886::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: stopping in state preparing (force False)
Thread-486::DEBUG::2011-07-16 08:48:53,887::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: ref 1 aborting True
Thread-486::INFO::2011-07-16 08:48:53,887::task::1177::TaskManager.Task::(prepare) aborting: Task is aborted: 'Cannot format attached storage domain' - code 391
Thread-486::DEBUG::2011-07-16 08:48:53,887::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: Prepare: aborted: Cannot format attached storage domain
Thread-486::DEBUG::2011-07-16 08:48:53,888::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: ref 0 aborting True
Thread-486::DEBUG::2011-07-16 08:48:53,888::task::492::TaskManager.Task::(_debug) Task 8f9e7925-6558-4e22-af72-482b2c6b93ce: Task._doAbort: force False
Thread-486::DEBUG::2011-07-16 08:48:53,888::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}


Expected results:
Storage domain gone and formatted.

Additional info:

Comment 2 Dan Kenigsberg 2011-07-16 10:41:21 UTC
How did you get to remove a data domain which is still attached to its pool?

Please give details on your setup (how many hosts? how many storage domains?), the perceived state of your storage domains - and relevant logs.

Comment 3 Jaroslav Henner 2011-07-16 13:40:36 UTC
(In reply to comment #2)
> How did you get to remove a data domain which is still attached to its pool?
It was not. It was in detached state. However I removed the pool just before I removed the SD.

> Please give details on your setup (how many hosts?
two

> how many storage domains?
I had one storage pool, one export domain there, which I removed fine. Also one data domain -- the one that caused the problems. There was a template imported from the export domain to the data domain and couple of machines from VM pool from that template. I removed all of that and after that I was removing the storage domain and went to this issue.

> the perceived state of your storage domains - and relevant logs.
Which log would you like? You have vdsm log up there. How can I know what's relevant?

Comment 4 Dan Kenigsberg 2011-07-16 20:41:58 UTC
Please attach vdsm.log files (bugzilla would thank you if you use compression) from both hosts, from the time that you started dismantling the storage pool.

Comment 5 Jaroslav Henner 2011-07-16 23:08:54 UTC
Created attachment 513503 [details]
logs

23:00:54,517 There was an error when attaching the export domain to the pool.
23:22:27,940 There was an error when removing the data domain.

Note there is Bug 722699 - Failed to attach Storage due to an error on the Data Center master Storage Domain. Now it seems that it very probably is related.

Comment 6 Saggi Mizrahi 2011-07-20 11:17:25 UTC
Looks like a clone of https://bugzilla.redhat.com/show_bug.cgi?id=719330 to me

Comment 7 Saggi Mizrahi 2011-07-20 11:51:46 UTC
Sorry my bad, not a clone. Here's a patch:
http://gerrit.usersys.redhat.com/#change,728

Comment 9 Tomas Dosek 2011-08-25 11:35:44 UTC
Verified - vdsm-4.9-95 - Cannot format storage domain no longer appears under above specified scenario.

Comment 10 errata-xmlrpc 2011-12-06 07:31:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1782.html