Bug 1638140

Summary:	Long deployments terminated due to closed watch
Product:	OpenShift Container Platform	Reporter:	Robert Bost <rbost>
Component:	openshift-controller-manager	Assignee:	Tomáš Nožička <tnozicka>
Status:	CLOSED ERRATA	QA Contact:	zhou ying <yinzhou>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.9.0	CC:	aos-bugs, glamb, grodrigu, mdame, openshift-bugs-escalate, tnozicka, yinzhou
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Deployment takes longer than some of the infrastructure or apiserver related timeouts. Consequence: Long running deployments fail. Fix: Deployer fixed to tolerate long running deployments by re-establishing the watch. Result: It works.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-01-10 09:04:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Robert Bost 2018-10-10 21:10:31 UTC

Description of problem:

Long-running deployments are killed due to:

  error: update acceptor rejected <deployment name>: watch closed before Until timeout.

These deployments are performing database migrations or loading large amounts of data that can take hours. 


Version-Release number of selected component (if applicable):
atomic-openshift-3.9.25-1.git.0.6bc473e.el7.x86_64

Additional info:
Possibly related -> bz1618663
Customer environment has F5 LB with 5 min timeout, however, closed watch can take hours to occur.

Comment 14 zhou ying 2018-12-04 08:01:48 UTC

Confirmed with latest openshift v3.11.51, the issue has fixed.

Comment 18 errata-xmlrpc 2019-01-10 09:04:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0024