Bug 1956921 - Deployments take a very long time and replication controller state gets out of sync
Summary: Deployments take a very long time and replication controller state gets out o...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Maciej Szulik
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-04 16:34 UTC by Luke Stanton
Modified: 2021-05-05 14:29 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-05 14:29:03 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Luke Stanton 2021-05-04 16:34:27 UTC
Description of problem:

When creating a new deployment or scaling/refreshing an existing one, deployments take a very long time and may timeout. Symptoms also include pods being stuck in a 'Pending' or 'Terminating' state for a long time and replication controllers not reflecting the current/desired number of pods correctly, for example:

~~~~~~~~~~~~~
$ oc get pods
NAME             READY   STATUS        RESTARTS   AGE
httpd-5-j24x4    0/1     Terminating   0          15m
httpd-8-deploy   0/1     Error         0          12m
httpd-8-rlpzs    1/1     Running       0          12m
httpd-8-sgdcs    1/1     Running       0          5m
$ oc get rc
NAME      DESIRED   CURRENT   READY   AGE
httpd-7   1         1         1       21m
httpd-8   0         0         0       12m
$ oc get dc
NAME    REVISION   DESIRED   CURRENT   TRIGGERED BY
httpd   8          1         0         config,image(httpd:2.4)
~~~~~~~~~~~~~

This appears to be accompanied with slow etcd performance and consistent api errors related to pod listings.

Version-Release number of selected component (if applicable):

OCP 3.11.404


How reproducible:

Consistently


Steps to Reproduce:
1. Create a new deployment or scale/refresh an existing one


Actual results:

Deployments take an excessively long time to process and either remain in an incorrect state or fail.


Expected results:

Deployments would work as expected.


Note You need to log in before you can comment on or make changes to this bug.