Bug 980139 - Need to add deps on kernel] vdsm iscsi failover taking too long during controller maintenance
Need to add deps on kernel] vdsm iscsi failover taking too long during contro...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.5.0
x86_64 Linux
high Severity high
: ovirt-3.6.0-rc3
: 3.6.0
Assigned To: Nir Soffer
Elad
: ZStream
Depends On: 1081264 1099932 1117499 1139038 1253789
Blocks: 880738 1273421
  Show dependency treegraph
 
Reported: 2013-07-01 10:09 EDT by Jon
Modified: 2016-03-09 14:17 EST (History)
40 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1081264 1099932 1273421 (view as bug list)
Environment:
Last Closed: 2016-03-09 14:17:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 413503 None None None Never
oVirt gerrit 47078 master MERGED spec: Require newer kernel version for RHEL Never
oVirt gerrit 47500 ovirt-3.6 MERGED spec: Require newer kernel version for RHEL Never
oVirt gerrit 47502 ovirt-3.5 MERGED spec: Require newer kernel version for RHEL Never

  None (edit)
Description Jon 2013-07-01 10:09:45 EDT
Description of problem:

iSCSI multipath failover delays result in performance issues and guest being marked as 'Down'.  Path failover is taking too long during controller failure or maintenance.

Version-Release number of selected component (if applicable):

Red Hat Enterprise Virtualization Hypervisor release 6.4 (20130501.0.el6_4)
2.6.32-358.6.1.el6.x86_64

How reproducible:
very

Steps to Reproduce:
1. Present iSCSI storage from ALUA configured storage
2. Perform storage controller reboot
3. multipath failover is delayed and results in guest stability issues

36005076802808538a00000000000009b dm-10 IBM,2145
size=2.0T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 1:0:0:3 sde 8:64   active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 2:0:0:3 sdm 8:192  active ready running


Actual results:

Storage domain <domain> experienced a high latency of 28.7959890366 seconds from host <rhev-host>. This may cause performance and functional issues. Please consult your Storage Administrator.


Expected results:

Fast failover time with no latency messages logged and no impact to guests.    


Additional info:

- Reduce the replacement_timeout since all iSCSI storage is default candidate for multipath: 
node.session.timeo.replacement_timeout = 120

Default is 2 minutes.  

replacement_timeout will control how long to wait for session re-establishment
before failing pending SCSI commands and commands that are being operated on by
the SCSI layer's error handler up to a higher level like multipath or to
an application if multipath is not being used.

Current work-around is to manually reduce this values within the iscsi target record db.
Comment 2 Mike Burns 2013-07-01 10:54:48 EDT
Ayal,

Is this something vdsm should handle? or something to send to dmm?
Comment 4 Ayal Baron 2013-07-24 05:33:21 EDT
This looks like a generic iscsi/dmm issue to me (default seems wrong when using dmm).  If this is wrong, please recommend what the value should be for virt use case and move back.
Comment 6 Ben Marzinski 2013-07-24 12:21:34 EDT
In RHEL6, multipath doesn't override iscsi's default parameters.  The instructions for setting up iscsi devices for multipath are at /usr/share/docs/iscsi-initiator-utils-<version>/README: Chapter 8.1

In RHEL7, multipath will automatically set the recovery_tmo iscsi sysfs parameter to the value defined for fast_io_fail_tmo in /etc/multipath.conf. The recovery_tmo parameter is normally set by the node.session.timeo.replacement_timeout configuration parameter, so this is equivalent to modifying the node.session.timeo.replacement_timeout for just the multipath devices.

It's possible to backport this, but it will most likely have to wait for RHEL-6.6
Comment 13 Fabian Deutsch 2014-05-15 03:59:50 EDT
If I understand comment 6 correctly, then this problem can probably be solved when we change the defaults at build-time.

For reference:
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/iscsi-replacements_timeout.html

Because of this I am moving this back to RHEV-H.
Comment 18 Federico Simoncelli 2014-08-11 07:30:41 EDT
I don't think this problem is limited to node only. It's a generic rhel/vdsm issue (changing component/whiteboard).

I verified that with device-mapper-multipath-0.4.9-78.el6.x86_64 the recovery timeout is now properly configured:

# iscsiadm -m session -P 2
...
 		*********
 		Timeouts:
 		*********
-		Recovery Timeout: 120
+		Recovery Timeout: 5


In the iscsi database (/var/lib/iscsi) we still have the default value:

node.session.timeo.replacement_timeout = 120

but then it's properly configured once multipath takes over.

We'll keep this bz on vdsm to track the spec requirement changes, e.g.:

Requires: device-mapper-multipath >= 0.4.9-78
Comment 19 Allon Mureinik 2014-08-19 09:36:59 EDT
(In reply to Federico Simoncelli from comment #18)
> Requires: device-mapper-multipath >= 0.4.9-78
This will only be available in RHEL 6.6 - lets wait for that in order to consume it.
Comment 20 Allon Mureinik 2014-09-02 07:56:48 EDT
(In reply to Allon Mureinik from comment #19)
> (In reply to Federico Simoncelli from comment #18)
> > Requires: device-mapper-multipath >= 0.4.9-78
> This will only be available in RHEL 6.6 - lets wait for that in order to
> consume it.
Pushing out to RHEV 3.6.0 based on this.
We can always pull it back in if device-mapper-multipath becomes available sooner.
Comment 21 Nir Soffer 2014-09-02 19:51:08 EDT
Testing fix for bug 880738 show that we must fix this first.

I blocked access to the storage server using iptables - this is probably the worst case, since removed device should return some error from the storage server. This test just drop all packets sent to the server.

With default node.session.timeo.replacement_timeout = 120, "multipath -ll" will block for 2 minutes before it fails.

With node.session.timeo.replacement_timeout = 10, "multipath -ll" failes after 15-25 seconds.
Comment 22 Nir Soffer 2014-09-02 19:57:09 EDT
Reproduced with:
device-mapper-multipath-0.4.9-66.el7.x86_64
iscsi-initiator-utils-6.2.0.873-21.el7.x86_64
kernel-3.10.0-123.el7.x86_64
Comment 23 Nir Soffer 2014-09-02 20:04:43 EDT
Ben, can we have a backport for 6.5/7.0?

If not, how about adding a udev rule to modify the timeout? The rule can do this:

    echo 5 > /sys/class/iscsi_session/xxx/recovery_tmo
Comment 24 Nir Soffer 2014-09-03 07:27:13 EDT
Attached patch handle only ISCSI devices, which can be handled when adding iscsi nodes in /var/lib/iscsi/nodes/. This keep the change localized to devices used by vdsm.

FC devices should be handled differently, probably using udev rule.
Comment 30 Yaniv Lavi (Dary) 2015-02-23 07:55:23 EST
Moving to new since we only have partial fix and we are waiting for platform to fully correct this.
Comment 35 Yaniv Kaul 2015-09-16 10:07:59 EDT
Nir, Allon, Yaniv D. - the bug made it to 7.2 (BZ 1139038) - anything blocking this bug?
Comment 36 Nir Soffer 2015-09-16 10:13:37 EDT
(In reply to Yaniv Kaul from comment #35)
> Nir, Allon, Yaniv D. - the bug made it to 7.2 (BZ 1139038) - anything
> blocking this bug?

We are waiting until this kernel is released. Vdsm will need to require
this kernel version.
Comment 37 Yaniv Kaul 2015-09-24 09:12:03 EDT
Nir - we are good to go - seems like the blocking bugs were resolved long time ago. Please continue with this one.
Comment 38 Yaniv Kaul 2015-09-24 09:15:06 EDT
(In reply to Yaniv Kaul from comment #37)
> Nir - we are good to go - seems like the blocking bugs were resolved long
> time ago. Please continue with this one.

From the bugs:
F21: 4.1.6
F22: 4.1.6
F23: 4.2.0-rc8
rawhide: 4.2.0-rc8

Downstream:
kernel-3.10.0-295.el7
Comment 39 Nir Soffer 2015-10-07 08:10:01 EDT
(In reply to Yaniv Kaul from comment #38)
> (In reply to Yaniv Kaul from comment #37)
> > Nir - we are good to go - seems like the blocking bugs were resolved long
> > time ago. Please continue with this one.
> 
> From the bugs:
> F21: 4.1.6
> F22: 4.1.6

We depend on these version now (see bug 1253790).

> F23: 4.2.0-rc8
> rawhide: 4.2.0-rc8

Not supported yet.

> Downstream:
> kernel-3.10.0-295.el7

We cannot depend on this in ovirt-3.6, which must run on el 7.1. We will depend on this kernel when el 7.2 is released.

I updated the vdsm patch to depend on the 7.1.z version:
kernel-3.10.0-229.17.1.el7

But this kernel is not available yet on Centos; we can see here
http://mirror.centos.org/centos/7/updates/x86_64/Packages/
that the latest kernel is kernel-3.10.0-229.14.1.el7.x86_64.rpm
which do not include this fix.

Currently we separate rhel and contos dependencies, so we cannot 
depend on package which is not available on centos yet.
Comment 40 Nir Soffer 2015-10-20 05:01:40 EDT
We have different requirement now for rhel and centos. This is now fixed for
rhel because the package is available, and not fixed on centos, because
nobody care to public the fixed package there.

We will require the fixed kernel on centos when it is available.
Comment 42 Allon Mureinik 2015-10-26 09:42:05 EDT
Nir, can you please add some doctext on the impact this has on the customer?
Comment 43 Aharon Canan 2015-11-09 06:19:16 EST
Target Milestone/Release - 
Can you please set it right?
Comment 44 Elad 2015-11-19 04:37:00 EST
Recovery timeout is now 15 seconds:

[root@green-vdsb ~]# iscsiadm -m session -P 2

               *********
                Timeouts:
                *********
                Recovery Timeout: 15
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15

[root@green-vdsb ~]# cat /sys/class/iscsi_session/session1/recovery_tmo 
15


vdsm requires kernel >= 3.10.0-229.17.1.el7:

[root@green-vdsb ~]# repoquery --requires vdsm |grep kernel
kernel >= 3.10.0-229.17.1.el7


Verified using 
vdsm-4.17.10.1-0.el7ev.noarch
Comment 46 errata-xmlrpc 2016-03-09 14:17:59 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0362.html

Note You need to log in before you can comment on or make changes to this bug.