Bug 1455667

Summary:	Increase SSHInactivityTimeoutSeconds for Upgrade host action
Product:	[oVirt] ovirt-engine	Reporter:	Ryan Barry <rbarry>
Component:	BLL.Infra	Assignee:	Martin Perina <mperina>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jiri Belka <jbelka>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.1.3	CC:	bugs, cshao, dguo, dougsland, huzhao, jbelka, jiawu, lsvaty, mperina, oourfali, qiyuan, rbarry, sbonazzo, stirabos, weiwang, yaniwang, ycui, yturgema, yzhao
Target Milestone:	ovirt-4.1.3	Flags:	rule-engine: ovirt-4.1+ rule-engine: exception+ lsvaty: testing_ack+
Target Release:	4.1.3.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: During NGN installation/upgrade there are performed extensive I/O operation without any output on SSH connection, which may cause the time out of SSH connection and fail the whole process. The time out is defined by default to 300 seconds in SSHInactivityTimeoutSeconds option and can be changed by engine-config. But we don't want to change the default for all SSH related operation as it may increase time outs where not necessary. So we decided to double value of SSHInactivityTimeoutSeconds for host installation/upgrade flow to prevent time outs on long running operations. Consequence: Fix: Result:	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-07-06 13:19:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1450831

Description Ryan Barry 2017-05-25 18:01:30 UTC

Description of problem:
When updating oVirt Node, we are bumping up against the timeout, and upgrades sometimes fail. Especially on systems with slower disks, or as problems are fixed in Node which increases the upgrade time, failing to upgrade from engine is a more likely scenario

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Install oVirt Node NGN 4.1.1
2. Upgrade to oVirt Node 4.1.2 on a system with slow disks

Actual results:
Upgrade fails sometimes at 300s. Succeeds every time at 600s.

Expected results:
Upgrade succeeds.

Additional info:

Comment 1 Red Hat Bugzilla Rules Engine 2017-05-25 18:03:17 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 2 Martin Perina 2017-05-31 08:13:07 UTC

(In reply to Ryan Barry from comment #0)
> Description of problem:
> When updating oVirt Node, we are bumping up against the timeout, and
> upgrades sometimes fail. Especially on systems with slower disks, or as
> problems are fixed in Node which increases the upgrade time, failing to
> upgrade from engine is a more likely scenario
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1. Install oVirt Node NGN 4.1.1
> 2. Upgrade to oVirt Node 4.1.2 on a system with slow disks
> 
> Actual results:
> Upgrade fails sometimes at 300s. Succeeds every time at 600s.

The issue here is that the same timeout is used for all SSH connection timeout, so if we increase timeout to 10 minutes it may also in some cases increase timeout to detect failure during SSH Soft Fencing, which could make HA VMs be restarted later.

So I'd prefer only to increase SSHInactivityTimeoutSeconds for Upgrade action and use 2 * SSHInactivityTimeoutSeconds as a value for SSH timeout during upgrade

Comment 3 Jiri Belka 2017-06-21 09:31:43 UTC

ok, ovirt-engine-4.1.3.4-0.1.el7.noarch

tested while upgrading to rhvh-4.1-0.20170609.0+1