Bug 844180 - formatStorageDomain fails due to devices not being discovered in time
formatStorageDomain fails due to devices not being discovered in time
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: vdsm (Show other bugs)
6.2
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Yeela Kaplan
Dafna Ron
storage
: ZStream
Depends On:
Blocks: 766281 820486
  Show dependency treegraph
 
Reported: 2012-07-29 11:37 EDT by Daniel Paikov
Modified: 2016-02-02 17:39 EST (History)
14 users (show)

See Also:
Fixed In Version: vdsm-4.9.6-41.0
Doc Type: Bug Fix
Doc Text:
Previously, deleting an iSCSI domain sometimes failed due to a race between discovering devices after connecting to the iSCSI target and trying to delete the domain. Now, formatStorageDomain performs a scan to make certain that all devices are detected, and iSCSI domains can be successfully deleted.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-04 14:03:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm.log (983.94 KB, application/x-compressed-tar)
2012-07-29 11:37 EDT, Daniel Paikov
no flags Details
logs (613.98 KB, application/x-gzip)
2012-10-28 06:00 EDT, Dafna Ron
no flags Details

  None (edit)
Description Daniel Paikov 2012-07-29 11:37:30 EDT
Created attachment 601047 [details]
vdsm.log

getDeviceList on an FCP host (also logged into some iSCSI targets) takes 83 seconds. This is too slow and the GUI times out before presenting a New Domain dialog.

real    1m23.188s
user    0m0.085s
sys     0m0.043s
Comment 2 Haim 2012-07-29 12:52:29 EDT
(In reply to comment #0)
> Created attachment 601047 [details]
> vdsm.log
> 
> getDeviceList on an FCP host (also logged into some iSCSI targets) takes 83
> seconds. This is too slow and the GUI times out before presenting a New
> Domain dialog.
> 
> real    1m23.188s
> user    0m0.085s
> sys     0m0.043s

getDeviceList was initiated at Thread-12656::DEBUG::2012-07-29 18:32:36,046::BindingXMLRPC::164::vds::(wrapper), thread started to work, and last call was at Thread-12656::DEBUG::2012-07-29 18:32:36,071::misc::1090::SamplingMethod::(__call__), then it went to sleep for a minute and so till Thread-12656::DEBUG::2012-07-29 18:33:58,474::__init__::1164::Storage.Misc.excCmd::(_log). 

is it a locking issue (not appears to be related to storage or any lvm operations).
Comment 3 Ayal Baron 2012-07-29 16:31:52 EDT
(In reply to comment #2)
> (In reply to comment #0)
> > Created attachment 601047 [details]
> > vdsm.log
> > 
> > getDeviceList on an FCP host (also logged into some iSCSI targets) takes 83
> > seconds. This is too slow and the GUI times out before presenting a New
> > Domain dialog.
> > 
> > real    1m23.188s
> > user    0m0.085s
> > sys     0m0.043s
> 
> getDeviceList was initiated at Thread-12656::DEBUG::2012-07-29
> 18:32:36,046::BindingXMLRPC::164::vds::(wrapper), thread started to work,
> and last call was at Thread-12656::DEBUG::2012-07-29
> 18:32:36,071::misc::1090::SamplingMethod::(__call__), then it went to sleep
> for a minute and so till Thread-12656::DEBUG::2012-07-29
> 18:33:58,474::__init__::1164::Storage.Misc.excCmd::(_log). 
> 
> is it a locking issue (not appears to be related to storage or any lvm
> operations).

No, looks like either a supervdsm issue or forceIScsiScan is taking too long.
For some reason we're lacking logging there so we need to add some to forceIScsiScan and reproduce.
Comment 5 Haim 2012-07-30 12:10:44 EDT
(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #0)
> > > Created attachment 601047 [details]
> > > vdsm.log
> > > 
> > > getDeviceList on an FCP host (also logged into some iSCSI targets) takes 83
> > > seconds. This is too slow and the GUI times out before presenting a New
> > > Domain dialog.
> > > 
> > > real    1m23.188s
> > > user    0m0.085s
> > > sys     0m0.043s
> > 
> > getDeviceList was initiated at Thread-12656::DEBUG::2012-07-29
> > 18:32:36,046::BindingXMLRPC::164::vds::(wrapper), thread started to work,
> > and last call was at Thread-12656::DEBUG::2012-07-29
> > 18:32:36,071::misc::1090::SamplingMethod::(__call__), then it went to sleep
> > for a minute and so till Thread-12656::DEBUG::2012-07-29
> > 18:33:58,474::__init__::1164::Storage.Misc.excCmd::(_log). 
> > 
> > is it a locking issue (not appears to be related to storage or any lvm
> > operations).
> 
> No, looks like either a supervdsm issue or forceIScsiScan is taking too long.
> For some reason we're lacking logging there so we need to add some to
> forceIScsiScan and reproduce.

well, reducing severity, it appears that scsi-rescan takes too long (appears to be storage problem):

[root@orange-vdse ~]#  for p in qls /sys/class/scsi_host/host*/; do basename $p; time echo "- - -" > $p/scan; done
ls

real    0m0.000s
user    0m0.000s
sys     0m0.000s
host0

real    0m0.161s
user    0m0.000s
sys     0m0.000s
host1

real    0m0.000s
user    0m0.000s
sys     0m0.000s
host2

real    0m0.225s
user    0m0.000s
sys     0m0.035s
host3

real    1m22.011s
user    0m0.000s
sys     0m0.000s
host4

real    0m0.000s
user    0m0.000s
sys     0m0.000s

however, let's make sure we are adding logs for such cases, I would change BZ title.
Comment 6 Yeela Kaplan 2012-07-31 04:04:36 EDT
(In reply to comment #5)
> (In reply to comment #3)
> > (In reply to comment #2)
> > > (In reply to comment #0)
> > > > Created attachment 601047 [details]
> > > > vdsm.log
> > > > 
> > > > getDeviceList on an FCP host (also logged into some iSCSI targets) takes 83
> > > > seconds. This is too slow and the GUI times out before presenting a New
> > > > Domain dialog.
> > > > 
> > > > real    1m23.188s
> > > > user    0m0.085s
> > > > sys     0m0.043s
> > > 
> > > getDeviceList was initiated at Thread-12656::DEBUG::2012-07-29
> > > 18:32:36,046::BindingXMLRPC::164::vds::(wrapper), thread started to work,
> > > and last call was at Thread-12656::DEBUG::2012-07-29
> > > 18:32:36,071::misc::1090::SamplingMethod::(__call__), then it went to sleep
> > > for a minute and so till Thread-12656::DEBUG::2012-07-29
> > > 18:33:58,474::__init__::1164::Storage.Misc.excCmd::(_log). 
> > > 
> > > is it a locking issue (not appears to be related to storage or any lvm
> > > operations).
> > 
> > No, looks like either a supervdsm issue or forceIScsiScan is taking too long.
> > For some reason we're lacking logging there so we need to add some to
> > forceIScsiScan and reproduce.
> 
> well, reducing severity, it appears that scsi-rescan takes too long (appears
> to be storage problem):
> 
> [root@orange-vdse ~]#  for p in qls /sys/class/scsi_host/host*/; do basename
> $p; time echo "- - -" > $p/scan; done
> ls
> 
> real    0m0.000s
> user    0m0.000s
> sys     0m0.000s
> host0
> 
> real    0m0.161s
> user    0m0.000s
> sys     0m0.000s
> host1
> 
> real    0m0.000s
> user    0m0.000s
> sys     0m0.000s
> host2
> 
> real    0m0.225s
> user    0m0.000s
> sys     0m0.035s
> host3
> 
> real    1m22.011s
> user    0m0.000s
> sys     0m0.000s
> host4

Actually only one scsi host is problematic here, we should limit the amount of time we wait and return whatever we have after the timeout.
loggin is also important though.

> 
> real    0m0.000s
> user    0m0.000s
> sys     0m0.000s
> 
> however, let's make sure we are adding logs for such cases, I would change
> BZ title.
Comment 8 Yeela Kaplan 2012-07-31 11:30:08 EDT
http://gerrit.ovirt.org/#/c/6796/
Comment 10 Ayal Baron 2012-08-22 06:30:02 EDT
Need to document the new vdsm config options.
Yeela, please provide a short explanations on which parameters have been added and what their function is.
Comment 14 Dafna Ron 2012-10-28 06:00:01 EDT
since the iscsiscan was suppose to solve the multipath race bug I am assigning this bug back on devel. 

during remove of several domains one domain failed on format due to the multipath race bug. 

attaching logs
Comment 15 Dafna Ron 2012-10-28 06:00:32 EDT
Created attachment 634474 [details]
logs
Comment 16 Haim 2012-10-28 06:23:03 EDT
I would like to clarify things up a bit, we decided to move this bug back to assigned per our agreement where the patch solves this bug should workaround multipath race we are facing in the various scenarios, thus, we found out that it still fail in format storage domain scenario.

Ayal - i need some kind of a fix for the multipath race, this is a blocker for the version, there is no point of opening a new bug since we have several already, either we should call the forceScsiRescan on every connectStorageServer or drop the disconnect storage server on engine side.
in any case, its your call.
Comment 17 Ayal Baron 2012-10-29 22:00:14 EDT
The fix here is to scan in formatStorageDomain in case device is not visible.
Comment 18 Yeela Kaplan 2012-10-30 05:37:56 EDT
a patch not related to the original bug that is meant to solve the multipath race in formatStorageDomin:

http://gerrit.ovirt.org/#/c/8921/
Comment 19 Dafna Ron 2012-11-06 09:01:52 EST
verified on vdsm-4.9.6-41.0.el6_3.x86_64 and si24
Comment 20 Zac Dover 2012-11-08 01:19:31 EST
Ayal,

Please read the Doc Text field and let me know if I've correctly summarized this fix.

Thanks!

Zac
Comment 22 errata-xmlrpc 2012-12-04 14:03:58 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1508.html
Comment 23 Dan Kenigsberg 2012-12-07 15:54:02 EST
The patch of comment 8 attempted to warn on bad config values, but instead - would raise an exception. See http://gerrit.ovirt.org/#/c/9690/
Comment 24 Yeela Kaplan 2012-12-09 02:20:54 EST
Dan, you are most definitely right. Thank you for the patch.

Note You need to log in before you can comment on or make changes to this bug.