Created attachment 709025 [details] engine + vdsm logs Description of problem: When trying to create a new data storage domain (nfs) on a mount that has an existing storage domain on it, engine enters a flow for HSM AddExistingStorageDomain, even though there is no Pool (this is the first storage domain), and importing data domains is not supported. Version-Release number of selected component (if applicable): rhevm-3.2.0-10.10.beta1.el6ev.noarch How reproducible: 100% Steps to Reproduce: Datacenter with 1 host, no storage domains. 1. Create an NFS data storage domain. (new Pool is created) 2. Destroy storage domain from rhevm GUI (storage is not cleaned) 3. Attempt to create another storage domain on the same mount. e.g.: CREATE request content is -- url:https://gadi-rhevm.scl.lab.tlv.redhat.com/api/storagedomains body:<storage_domain> <name>nfs_0</name> <type>data</type> <storage> <type>nfs</type> <address>lion.qa.lab.tlv.redhat.com</address> <path>/fastpass/gickowic/nfs1</path> </storage> <host> <name>cheetah01.scl.lab.tlv.redhat.com</name> </host> <storage_format>v3</storage_format> </storage_domain> Actual results: Creation fails with error message: DEBUG - Response body for CREATE request is: <fault> <reason>Operation Failed</reason> <detail>[Importing data domain is prohibited.]</detail> </fault> This is after engine logs show it enters the following flow: 2013-03-12 16:11:06,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] START, ValidateStorageServerConnectionVDSCommand(HostName = ch eetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.red hat.com:/fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 390f0553 2013-03-12 16:11:07,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00 000000-0000-0000-0000-000000000000=0}, log id: 390f0553 2013-03-12 16:11:07,015 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] Running command: AddStorageServerConnectionCommand internal: false. Entities a ffected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2013-03-12 16:11:07,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.re dhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/fastpass/gick owic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1e0de259 2013-03-12 16:11:07,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000- 000000000000=0}, log id: 1e0de259 2013-03-12 16:11:07,860 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp-/127.0.0.1:8702-5) START, GetExistingStorageDomainListQuery(org.ovirt.engine.core.common.queries.GetExisting StorageDomainListParameters@7abc7253), log id: 52423d25 2013-03-12 16:11:07,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-5) START, HSMGetStorageDomainsListVDSCommand(HostName = cheetah01.scl.lab.tlv.redha t.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=Data, path=lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1), log id: 22e2d9b2 2013-03-12 16:11:16,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-5) FINISH, HSMGetStorageDomainsListVDSCommand, return: [05e9f330-fc2f-4ef4-8f1b-36c 6e6fc1126], log id: 22e2d9b2 2013-03-12 16:11:16,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-5) START, HSMGetStorageDomainInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat. com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storageDomainId=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126), log id: 5767429b 2013-03-12 16:11:16,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-5) FINISH, HSMGetStorageDomainInfoVDSCommand, return: org.ovirt.engine.core.common.u tils.Pair@de142d60, log id: 5767429b 2013-03-12 16:11:16,271 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp-/127.0.0.1:8702-5) FINISH, GetExistingStorageDomainListQuery, log id: 52423d25 2013-03-12 16:11:16,282 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. 2013-03-12 16:11:16,291 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for versio n 2013-03-12 16:11:16,291 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version 2013-03-12 16:11:16,301 WARN [org.ovirt.engine.core.bll.storage.AddExistingNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [78cf530] CanDoAction of action AddExistingNFSStorageDomain failed. Reasons:VAR__TYPE__S TORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_IMPORT_DATA_DOMAIN_PROHIBITED 2013-03-12 16:11:16,305 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-5) Operation Failed: [Importing data domain is prohibited.] Expected results: engine should not try to AddExistingNFSStorageDomain if requested domain type in API is <type>data</type> Additional info: Trying to create the same storage domain through GUI returns appropriate error message directly from vdsm, also visible in the log - error = Storage domain is not empty - requires cleaning: ('lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1'
Please note that the command AddExistingNFSStorageDomainCommand mentioned in bug's description was renamed to AddExistingFileStorageDomainCommand. Indeed the flow in UI is that import existing domain and add new domain are 2 different user actions, while in REST they are initiated by a single action of add domain - decided by the engine. The relevant code in REST is in BackendStorageDomainsResource in addDomain method. The logic should be aligned to have same behavior in UI and REST - by creating an option in REST to import an existing domain.
Maor, this should be handled as part of your work for importing a storage domain. In a REST API perspective, we should have a new API for importing, and a /create/ request should /not/ attempt to import (although this needs to be double-checked with the REST maintainers)
It should work now: Importing an existing Storage Domain: <storage_domain> <type>data</type> <storage> <type>nfs</type> <address>xx.xx.xx.xx</address> <path>/export/images/rnd/{some_path}</path> </storage> <host id="3dc5ba65-2f7a-414c-ad2b-635cdb1afbc7"/> </storage_domain> Storage was added. Got the following result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_domain href="/ovirt-engine/api/storagedomains/4bc4bd4f-852b-458e-8975-d774bfc92cb9" id="4bc4bd4f-852b-458e-8975-d774bfc92cb9"> <name>dsadsad222</name> .... <type>data</type> <status> <state>unknown</state> </status> <master>false</master> <storage> <address>xx.xx.xx.xx</address> <type>nfs</type> <path>/export/images/rnd/{some_path}</path> </storage> <available>15032385536</available> <used>587336777728</used> <committed>0</committed> <storage_format>v3</storage_format> </storage_domain> Moving to ON_QA for verification
hi Maor, This bug has been flagged for release notes. Please select the correct Doc Type and provide the doc text ASAP for this bug to make it in the 3.5 Beta Manager Release Notes. If this bug is not required for release notes, please set the require_release_note flag to -. Cheers, Julie
I checked both for UI and REST, and in both the adding operation fails. setup: 1 DC shared, 1 Host. 1. add nfs_0 domain, and nfs_1 domain 2. put in maintenance nfs_0 3. destroy nfs_1 4. try to import that destroyed domain from either the UI (Import domain) or the REST (CREATE without a name for the storage domain) result: domain is added but cannot be attached to the DC In my run ovirt.log shows: START, ConnectStorageServerVDSCommand(HostName = 10.34.62.206, HostId = c6b2a203-3b1f-4cf6-91b2-2c001529452c, storagePoolId = 00000000-0000-0000-0000-000000000000 [...] The meta data of the Storage Domain might still indicate that it is attached to a different Storage Pool. and indeed the vdsm.log shows: Unknown pool id, pool not connected: (u'00000000-0000-0000-0000-000000000000',) while the poolId is really 00000002-0002-0002-0002-00000000002b (checked on vt5). Maor: can you comment if I'm understanding this correctly?
Hi Carlos, there is a known issue with json regarding this, can u please try to reproduce this with Host which not using json
(In reply to Carlos Mestre González from comment #7) > setup: 1 DC shared, 1 Host. > > 1. add nfs_0 domain, and nfs_1 domain > 2. put in maintenance nfs_0 > 3. destroy nfs_1 --> typo, should have been destroy nfs_0 since it's in maintenance, > 4. try to import that destroyed domain from either the UI (Import domain) or the REST (CREATE without a name for the storage domain) (In reply to Maor from comment #8) > Hi Carlos, there is a known issue with json regarding this, can u please try > to reproduce this with Host which not using json I'm doing a xml call (?) Just to emphasize, the REST API call returns the proper response object, and behaves exactly as the Web UI: thee storage domain is imported but it cannot be attached to the existing DC. So I'm guessing this is a backend bug not related to REST.
sorry Maor, disabled json messaging and works as it suppose to. verified in vt5.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0158.html