Bug 1302745 - [engine-setup] creation of iso domain path is not rolled back, a next attempt leaves an empty domain
[engine-setup] creation of iso domain path is not rolled back, a next attempt...
Product: ovirt-engine
Classification: oVirt
Component: Setup.Engine (Show other bugs)
Unspecified Unspecified
low Severity medium (vote)
: ovirt-4.0.0-rc
: ---
Assigned To: Yedidyah Bar David
Pavel Stehlik
Depends On:
  Show dependency treegraph
Reported: 2016-01-28 09:06 EST by Yedidyah Bar David
Modified: 2016-06-02 04:12 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-06-02 04:12:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ylavi: ovirt‑4.0.0?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?

Attachments (Terms of Use)

  None (edit)
Description Yedidyah Bar David 2016-01-28 09:06:41 EST
Description of problem:

If during engine-setup we create an iso domain, and later fail and rollback, we remove the files we created there, but do not remove directories. So these are left behind.

A next attempt to run engine-setup with the answer file created by the first one will try to create an iso domain in the same location. It checks if it exists, and if so, uses the directory's name found there as the uuid of the domain. Later, if such a uuid is defined, we do not populate it with actual files (including metadata), thus leaving it empty.

engine-setup will finish successfully, but when actually trying to attach it, this will fail, with:


2016-01-28 15:27:34,235 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-7-thread-26) [fd1aad0] Command 'AttachStorageDomainVDSCommand( AttachStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='00000001-0001-0001-0001-00000000023b', ignoreFailoverLimit='false', storageDomainId='3b1a2f5b-f807-4e09-86fa-a31c9e59305c'})' execution failed: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Error in storage domain action: (u'sdUUID=3b1a2f5b-f807-4e09-86fa-a31c9e59305c, spUUID=00000001-0001-0001-0001-00000000023b',), code = 350
2016-01-28 15:27:34,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-7-thread-26) [fd1aad0] FINISH, AttachStorageDomainVDSCommand, log id: 2d2f5ac6
2016-01-28 15:27:34,235 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-7-thread-26) [fd1aad0] Command 'org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Error in storage domain action: (u'sdUUID=3b1a2f5b-f807-4e09-86fa-a31c9e59305c, spUUID=00000001-0001-0001-0001-00000000023b',), code = 350 (Failed with error StorageDomainActionError and code 350)


jsonrpc.Executor/7::DEBUG::2016-01-28 15:27:33,168::fileSD::159::Storage.StorageDomainManifest::(__init__) Reading domain in path /rhev/data-center/mnt/didi-f19-engine.eng.lab.tlv.redhat.com:_var_lib_exports_iso/3b1a2f5b-f807-4e09-86fa-a31c9e59305c
jsonrpc.Executor/7::DEBUG::2016-01-28 15:27:33,169::__init__::318::IOProcessClient::(_run) Starting IOProcess...
jsonrpc.Executor/7::DEBUG::2016-01-28 15:27:33,198::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
jsonrpc.Executor/7::DEBUG::2016-01-28 15:27:33,202::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[]
jsonrpc.Executor/7::DEBUG::2016-01-28 15:27:33,203::persistentDict::252::Storage.PersistentDict::(refresh) Empty metadata
jsonrpc.Executor/7::ERROR::2016-01-28 15:27:33,203::task::867::Storage.TaskManager.Task::(_setError) Task=`130a4b02-138c-401b-b618-31f50760c333`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 874, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 49, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1210, in attachStorageDomain
  File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 884, in attachSD
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 330, in __init__
    manifest = self.manifestClass(domainPath)
  File "/usr/share/vdsm/storage/fileSD.py", line 168, in __init__
    sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata)
  File "/usr/share/vdsm/storage/sd.py", line 310, in __init__
    self._domainLock = self._makeDomainLock()
  File "/usr/share/vdsm/storage/sd.py", line 439, in _makeDomainLock
    domVersion = self.getVersion()
  File "/usr/share/vdsm/storage/sd.py", line 359, in getVersion
    return self.getMetaParam(DMDK_VERSION)
  File "/usr/share/vdsm/storage/sd.py", line 356, in getMetaParam
    return self._metadata[key]
  File "/usr/share/vdsm/storage/persistentDict.py", line 89, in __getitem__
    return dec(self._dict[key])
  File "/usr/share/vdsm/storage/persistentDict.py", line 201, in __getitem__
    raise KeyError(key)
KeyError: 'VERSION'

The web interface will show:

VDSM command failed: Error in storage domain action: (u'sdUUID=3b1a2f5b-f807-4e09-86fa-a31c9e59305c, spUUID=00000001-0001-0001-0001-00000000023b',)

Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin@internal)

Version-Release number of selected component (if applicable):

Current master. engine-setup behavior is way older, not sure about ui/engine/vdsm.

How reproducible:

Always I think

Steps to Reproduce:
1. engine-setup, choose to create an nfs iso domain
2. fail engine-setup after it prepared the iso domain
3. run engine-setup again with the answerfile generated by 1.

Actual results:

engine-setup finishes successfully, leaving an empty domain, attaching it fails

Expected results:

I guess engine-setup should try to also remove the empty directories it created during rollback. It's probably not very easy to do properly with current transaction rollback and current filetransaction.

Additional info:


If between (2.) and (3.) above, remove the iso domain

If after (3.), destroy the iso domain from the web interface and create a new one.
Comment 1 Sandro Bonazzola 2016-05-02 05:48:36 EDT
Moving from 4.0 alpha to 4.0 beta since 4.0 alpha has been already released and bug is not ON_QA.
Comment 2 Yaniv Lavi 2016-05-23 09:13:22 EDT
oVirt 4.0 beta has been released, moving to RC milestone.
Comment 3 Yaniv Lavi 2016-06-02 04:12:22 EDT
We will not longer allow users to define a ISO domain in the setup, therefore closing this bug.
Comment 4 Yaniv Lavi 2016-06-02 04:12:46 EDT
This change is from 4.0 and on.

Note You need to log in before you can comment on or make changes to this bug.