Description of problem: I am seeing multiple errors when removing a storage domain from RHEV 2013-10-21 00:18:38,285 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) Error code CannotFormatStorageDomainInConnectedPool and error message VDSGenericException: VDSErrorException: Failed to FormatStorageDomainVDS, error = Cannot format storage domain in connected pool: ('22317723-54be-4807-a477-14ff7c872b32',) 2013-10-21 00:18:38,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) Command org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand return value 2013-10-21 00:18:38,285 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) Command FormatStorageDomainVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to FormatStorageDomainVDS, error = Cannot format storage domain in connected pool: ('22317723-54be-4807-a477-14ff7c872b32',) 2013-10-21 00:18:38,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) FINISH, FormatStorageDomainVDSCommand, log id: 41a39bca 2013-10-21 00:18:38,286 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-4) Command org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to FormatStorageDomainVDS, error = Cannot format storage domain in connected pool: ('22317723-54be-4807-a477-14ff7c872b32',) (Failed with error CannotFormatStorageDomainInConnectedPool and code 392) 2013-10-21 00:18:38,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: 213cf37a, Job ID: 39d92ab1-9a7d-4767-89ad-8ca20437e5da, Call Stack: null, Custom Event ID: -1, Message: Failed to remove Storage Domain DataDomainRest2. (User: admin@internal) 2013-10-21 00:18:38,313 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-4) Operation Failed: [Cannot format storage domain in connected pool] REST command is 2013-10-21 00:19:17,427 - MainThread - storagedomains - DEBUG - DELETE request content is -- url:/api/storagedomains/5ef98023-bf0a-45d0-884d-0f7ef4c29b4a body:<storage_domain> <host id="21b2bc10-b09a-4383-ad05-3ccefd0639c1"/> <format>true</format> </storage_domain>
All, just to be more focused - The issue is that we can't remove the unattached storage domain (master) after removing the DC, It has nothing to do with automation as manual doesn't work as well. The automation test fails because of the cleanup - the test itself pass. to reproduce - 1. setup DC with one storage domain (master) 2. enter SD to maintenance 3. remove the DC 4. try to remove the SD fails due to - Thread-4486::ERROR::2013-10-27 15:43:20,166::task::850::TaskManager.Task::(_setError) Task=`9b129564-1fcf-4a93-bd7c-10d1625e337f`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 857, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2701, in formatStorageDomain raise se.CannotFormatAttachedStorageDomain(sdUUID)
format storage domain for a regular, properly detached domain works. It only fails when the ids stays open/locked by sanlock due to error in flow by engine during remove data center that causes disconnectStorageServer before disconnectStoragePool: [root@aqua-vds4 ~]# grep 'formatStorageDomain' /var/log/vdsm/vdsm.log Thread-169857::INFO::2013-10-27 16:09:53,319::logUtils::44:ispatcher:wrapper) Run and protect: formatStorageDomain(sdUUID='040c5c60-72a9-447a-a0df-b65be3c8311b', autoDetach=False, options=None) Thread-169857:EBUG::2013-10-27 16:09:55,545::resourceManager::197::ResourceManager.Request:__init__) ResName=`Storage.040c5c60-72a9-447a-a0df-b65be3c8311b`ReqID=`88ad1086-857c-4b24-b642-83164915b91b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2682' at 'formatStorageDomain' Thread-169857::INFO::2013-10-27 16:10:01,472::logUtils::47:ispatcher:wrapper) Run and protect: formatStorageDomain, Return response: None
This is most likely a duplicate of bug 1021557. Please check bug 1021557 comment 9.
I can't say if it is dup or not, fede, yours... in case it is please mark as dup' if not and we want to keep them both add comment. Thanks.
verified using is24.1 following steps in comment #4
is26. tested as described in comment #4. Master SD was successfully removed.
Closing - RHEV 3.3 Released