VDSM now provides greater detail on the nature of errors that occurr when attempting to create a new storage domain and a host connected to the storage server on which the domain is being created is disconnected during the operation.
Description of problem:
When attempting to create a new iscsi storage domain createVG fails with Invalid physical device.
Between the time getDeviceList was requested and the vgCreate the host suffered a disconnection from the storage server, so the /sys/block/dm-*/dm directory was not present anymore.
Thread-281::INFO::2012-08-27 09:12:59,238::logUtils::39:ispatcher:wrapper) Run and protect: getDeviceList, Return response: {'devList': [{'status': 'used', 'vendorID': 'IET'
, 'capacity': '21474836480', 'fwrev': '0001', 'vgUUID': 'lFOk00-QNSe-Z20g-osyk-rzpV-1fBe-FytEvU', 'pathlist': [{'initiatorname': 'default', 'connection': 'orion2.qa.lab.tlv.redh
at.com', 'iqn': 'iqn.1994-05.com.redhat:awinter_0_20120826164454668563', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physd
ev': 'sdb', 'type': 'iSCSI', 'state': 'active', 'lun': '1'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': 'efiQe1-cgGI-MeQI-mwbv-RMKk-lD4A-YCdQmB', 'serial': 'SIET
_VIRTUAL-DISK', 'GUID': '1492565d9-28fb-4858', 'productID': 'VIRTUAL-DISK'}]}
Thread-881::INFO::2012-08-27 09:29:56,303::logUtils::37:ispatcher:wrapper) Run and protect: createVG(vgname='b1d90f0a-981f-4203-a9c2-f8f309409846', devlist=['1492565d9-28fb-
4858'], force=True, options=None)
Thread-881::ERROR::2012-08-27 09:29:56,304::task::853::TaskManager.Task:_setError) Task=`ca9e63ff-f252-4820-b557-57c2ace3c18b`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 861, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1713, in createVG
raise se.InvalidPhysDev(dev)
InvalidPhysDev: Invalid physical device: ('1492565d9-28fb-4858',)
Version-Release number of selected component (if applicable):
vdsm-4.9.6-28.0.el6_3.x86_64
How reproducible:
Steps to Reproduce:
1. connect host to storage server
2. manually (not through vdsm) disconnect the storage server
3. send createVG
Actual results:
Expected results:
It may be more informative to provide slightly more information in the logging error message, e.g. information about what part of the process fails - if the device is not in
/sys/block/dm-*/dm or multipath, or if it fails finding the specific GUID etc.
Additional info:
Created attachment 607419[details]
vdsm
1. VDSM log attached
2.Eduardo managed to reproduce the bug with haim:
>Reproduced with Haim in the same host.
>
> In between the getDeviceList and the createVG vdsm commands the host
> was (or suffered) a disconection of the storage server.
> The disconnect was not issued with vdsm.
> In this case the interval was about ~17 minutes.
>
> When the createVG was issued the /sys/block/dm-*/dm directory was not
> present anymore, therefore not listed by
> multipath.getMPDeviceIter().
> This and undeliying functions swallow all errors and we get only the
> cryptic: "InvalidPhysDev: Invalid physical device:" error.
>
> Since we use the GUID for pvcreate may be the whole use of the
> iterator in createVG can be avoided.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHBA-2014-0504.html
Description of problem: When attempting to create a new iscsi storage domain createVG fails with Invalid physical device. Between the time getDeviceList was requested and the vgCreate the host suffered a disconnection from the storage server, so the /sys/block/dm-*/dm directory was not present anymore. Thread-281::INFO::2012-08-27 09:12:59,238::logUtils::39:ispatcher:wrapper) Run and protect: getDeviceList, Return response: {'devList': [{'status': 'used', 'vendorID': 'IET' , 'capacity': '21474836480', 'fwrev': '0001', 'vgUUID': 'lFOk00-QNSe-Z20g-osyk-rzpV-1fBe-FytEvU', 'pathlist': [{'initiatorname': 'default', 'connection': 'orion2.qa.lab.tlv.redh at.com', 'iqn': 'iqn.1994-05.com.redhat:awinter_0_20120826164454668563', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physd ev': 'sdb', 'type': 'iSCSI', 'state': 'active', 'lun': '1'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': 'efiQe1-cgGI-MeQI-mwbv-RMKk-lD4A-YCdQmB', 'serial': 'SIET _VIRTUAL-DISK', 'GUID': '1492565d9-28fb-4858', 'productID': 'VIRTUAL-DISK'}]} Thread-881::INFO::2012-08-27 09:29:56,303::logUtils::37:ispatcher:wrapper) Run and protect: createVG(vgname='b1d90f0a-981f-4203-a9c2-f8f309409846', devlist=['1492565d9-28fb- 4858'], force=True, options=None) Thread-881::ERROR::2012-08-27 09:29:56,304::task::853::TaskManager.Task:_setError) Task=`ca9e63ff-f252-4820-b557-57c2ace3c18b`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 861, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1713, in createVG raise se.InvalidPhysDev(dev) InvalidPhysDev: Invalid physical device: ('1492565d9-28fb-4858',) Version-Release number of selected component (if applicable): vdsm-4.9.6-28.0.el6_3.x86_64 How reproducible: Steps to Reproduce: 1. connect host to storage server 2. manually (not through vdsm) disconnect the storage server 3. send createVG Actual results: Expected results: It may be more informative to provide slightly more information in the logging error message, e.g. information about what part of the process fails - if the device is not in /sys/block/dm-*/dm or multipath, or if it fails finding the specific GUID etc. Additional info: