Bug 1460501

Summary: Failed to extend block-based storage domain, ExtendSANStorageDomainCommand pass multiple PVs with the same name
Product: [oVirt] ovirt-engine Reporter: Eyal Shenitzky <eshenitz>
Component: BLL.StorageAssignee: Daniel Erez <derez>
Status: CLOSED CURRENTRELEASE QA Contact: Eyal Shenitzky <eshenitz>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: amureini, bugs, eshenitz, stirabos, tnisan
Target Milestone: ovirt-4.1.3Flags: rule-engine: ovirt-4.1+
Target Release: 4.1.3.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-06 13:42:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1454864    
Attachments:
Description Flags
engine and vdsm logs none

Description Eyal Shenitzky 2017-06-11 11:30:03 UTC
Created attachment 1286824 [details]
engine and vdsm logs

Description of problem:
iSCSI storage domain extention fails in case the initiator is connected to the LUN through multiple targets. This happens because engine sends to vdsm multiple PVs, with the same PV name, according to the targets number in CreateVG.

VDSM command ExtendStorageDomainVDS failed: Cannot extend Volume Group: "vgname=c00b0577-4be8-434d-ac6d-fad06daa7172, devname=['/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd']"

Engine log:

2017-06-11 13:58:20,481+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ExtendStorageDomainVDSCommand] (default task-23) [118515b7] Failed in 'ExtendStorageDomainVDS' method
2017-06-11 13:58:20,500+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-23) [118515b7] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM command ExtendStorageDomainVDS failed: Cannot extend Volume Group: "vgname=c00b0577-4be8-434d-ac6d-fad06daa7172, devname=['/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd']"
2017-06-11 13:58:20,500+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (default task-23) [118515b7] IrsBroker::Failed::ExtendStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ExtendStorageDomainVDS, error = Cannot extend Volume Group: "vgname=c00b0577-4be8-434d-ac6d-fad06daa7172, devname=['/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd']", code = 503
2017-06-11 13:58:20,522+03 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ExtendStorageDomainVDSCommand] (default task-23) [118515b7] FINISH, ExtendStorageDomainVDSCommand, log id: 40c6a546
2017-06-11 13:58:20,523+03 ERROR [org.ovirt.engine.core.bll.storage.domain.ExtendSANStorageDomainCommand] (default task-23) [118515b7] Command 'org.ovirt.engine.core.bll.storage.domain.ExtendSANStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to ExtendStorageDomainVDS, error = Cannot extend Volume Group: "vgname=c00b0577-4be8-434d-ac6d-fad06daa7172, devname=['/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd', '/dev/mapper/3514f0c5a516008dd']", code = 503 (Failed with error VolumeGroupExtendError and code 503)

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Create a bloc-based storage domain
2. Extend the storage-domain using free device

Actual results:
Storage domain extension failed

Expected results:
Storage domain extension should succeed

Additional info:
VDSM and engine log attached

Comment 1 Eyal Shenitzky 2017-06-11 12:01:46 UTC
Versions:
VDSM - 4.20.0-970.gitf0c24e5.el7.centos.x86_64
Engine - unregistered

Comment 2 Eyal Shenitzky 2017-06-11 12:02:35 UTC
Engine - 4.2.0-0.0.master.20170602194647.gitaf23eb5.el7.centos

Comment 3 Allon Mureinik 2017-06-12 08:59:02 UTC
Sounds like a similar issue to bug 1441706. If so, it's probably an easy fix.

Comment 4 Daniel Erez 2017-06-12 12:59:11 UTC
@Eyal - I've tried the described scenario (using the WebAdmin) with two luns exposed from multiple targets - seems to behave as expected (i.e. the used luns are grayed-out in the extend dialog). Can you please describe your exact luns/targets configuration? Did you able to select the same (used) lun? Did you try it only from rest-api?

Comment 5 Eyal Shenitzky 2017-06-13 05:12:37 UTC
I try it using the web admin and the Rest-API,
All the flow worked well, the engine sends a message that the storage-domain updated and afterward the error occurred.
We are using XIO as backend if it is relevant.

Comment 6 Eyal Shenitzky 2017-06-13 14:47:55 UTC
Steps to reproduce:

1. Create a block-based storage domain
2. Press the 'Manage Domain' button
3. Press 'Target > Luns' side tab
4. Select free device
5. Extend the storage domain

Comment 7 rhev-integ 2017-06-16 16:44:34 UTC
INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Open patch attached]

For more info please contact: infra

Comment 8 Eyal Shenitzky 2017-06-20 11:15:00 UTC
Verified with the following code:
---------------------------------------
VDSM - 4.19.19-1.el7ev.x86_64
RHEVM - 4.1.3.4-0.1.el7
 
Steps to reproduce:
------------------------
1. Create a block-based storage domain
2. Press the 'Manage Domain' button
3. Press 'Target > Luns' side tab
4. Select free device
5. Extend the storage domain

Storage Extended as expected

Moving to VERIFIED