Bug 1638140 - Long deployments terminated due to closed watch
Summary: Long deployments terminated due to closed watch
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-controller-manager
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.11.z
Assignee: Tomáš Nožička
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-10-10 21:10 UTC by Robert Bost
Modified: 2019-01-10 09:04 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Deployment takes longer than some of the infrastructure or apiserver related timeouts. Consequence: Long running deployments fail. Fix: Deployer fixed to tolerate long running deployments by re-establishing the watch. Result: It works.
Clone Of:
Environment:
Last Closed: 2019-01-10 09:04:01 UTC
Target Upstream Version:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0024 0 None None None 2019-01-10 09:04:07 UTC

Description Robert Bost 2018-10-10 21:10:31 UTC 
Description of problem:

Long-running deployments are killed due to:

  error: update acceptor rejected <deployment name>: watch closed before Until timeout.

These deployments are performing database migrations or loading large amounts of data that can take hours. 


Version-Release number of selected component (if applicable):
atomic-openshift-3.9.25-1.git.0.6bc473e.el7.x86_64

Additional info:
Possibly related -> bz1618663
Customer environment has F5 LB with 5 min timeout, however, closed watch can take hours to occur.
Comment 14 zhou ying 2018-12-04 08:01:48 UTC 
Confirmed with latest openshift v3.11.51, the issue has fixed.
Comment 18 errata-xmlrpc 2019-01-10 09:04:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0024
Note You need to log in before you can comment on or make changes to this bug.