Bug 1981770 - Problematic Deployment creates infinite number Replicasets causing etcd to reach quota limit
Summary: Problematic Deployment creates infinite number Replicasets causing etcd to re...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.7
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: 4.8.z
Assignee: Filip Krepinsky
QA Contact: zhou ying
URL:
Whiteboard:
: 1982717 (view as bug list)
Depends On: 1976775
Blocks: 1981775
TreeView+ depends on / blocked
 
Reported: 2021-07-13 11:24 UTC by Filip Krepinsky
Modified: 2021-10-21 17:50 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: caused if deployment is created with unresolved image (no image stream) and no image.openshift.io/triggers annotation. By creating image stream and subsequently patching our deployment, new image resolution will occur on new replica sets. This will result in inconsistent state between deployment controller and apiserver's imagepolicy plugin. Consequence: deployment controller creates replica sets in infinite loop Fix: responsibilities of apiserver's imagepolicy plugin were lowered Result: inconsistent image resolution should not occur in the deployments and thus it should not cause creation of infinite replica sets anymore
Clone Of: 1976775
Environment:
Last Closed: 2021-08-10 11:27:39 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift apiserver-library-go pull 54 0 None closed Bug 1981770: fix Deployment creates a huge number of ReplicaSets 2021-07-21 14:10:37 UTC
Github openshift kubernetes pull 864 0 None open Bug 1981770: UPSTREAM: <drop>: bump(apiserver-library-go) 2021-07-21 14:08:38 UTC
Red Hat Product Errata RHSA-2021:2983 0 None None None 2021-08-10 11:28:01 UTC

Comment 1 Filip Krepinsky 2021-07-21 11:58:06 UTC
*** Bug 1982717 has been marked as a duplicate of this bug. ***

Comment 4 zhou ying 2021-08-03 07:33:10 UTC
Can't reproduce with latest payload:

[root@localhost roottest]# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-07-31-065602   True        False         44m     Cluster version is 4.8.0-0.nightly-2021-07-31-065602

[root@localhost roottest]# oc create -f /tmp/depl.yaml 
deployment.apps/i-spawn-replicas created
[root@localhost roottest]# oc get deployment i-spawn-replicas -o json | jq '.spec'
{
  "progressDeadlineSeconds": 600,
  "replicas": 0,
  "revisionHistoryLimit": 10,
  "selector": {
    "matchLabels": {
      "app": "i-spawn-replicas"
    }
  },
  "strategy": {
    "rollingUpdate": {
      "maxSurge": "25%",
      "maxUnavailable": "25%"
    },
    "type": "RollingUpdate"
  },
  "template": {
    "metadata": {
      "creationTimestamp": null,
      "labels": {
        "app": "i-spawn-replicas"
      },
      "name": "i-spawn-replicas"
    },
    "spec": {
      "containers": [
        {
          "image": "imagestreamname",
          "imagePullPolicy": "Always",
          "name": "container1",
          "resources": {},
          "terminationMessagePath": "/dev/termination-log",
          "terminationMessagePolicy": "File"
        }
      ],
      "dnsPolicy": "ClusterFirst",
      "restartPolicy": "Always",
      "schedulerName": "default-scheduler",
      "securityContext": {},
      "terminationGracePeriodSeconds": 30
    }
  }
}
[root@localhost roottest]# oc get deploy
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
i-spawn-replicas   0/0     0            0           7m3s
[root@localhost roottest]# oc get rs
NAME                          DESIRED   CURRENT   READY   AGE
i-spawn-replicas-84778cc586   0         0         0       7m9s
[root@localhost roottest]# oc get rs i-spawn-replicas-84778cc586 -o json | jq '.spec'
{
  "replicas": 0,
  "selector": {
    "matchLabels": {
      "app": "i-spawn-replicas",
      "pod-template-hash": "84778cc586"
    }
  },
  "template": {
    "metadata": {
      "creationTimestamp": null,
      "labels": {
        "app": "i-spawn-replicas",
        "pod-template-hash": "84778cc586"
      },
      "name": "i-spawn-replicas"
    },
    "spec": {
      "containers": [
        {
          "image": "imagestreamname",
          "imagePullPolicy": "Always",
          "name": "container1",
          "resources": {},
          "terminationMessagePath": "/dev/termination-log",
          "terminationMessagePolicy": "File"
        }
      ],
      "dnsPolicy": "ClusterFirst",
      "restartPolicy": "Always",
      "schedulerName": "default-scheduler",
      "securityContext": {},
      "terminationGracePeriodSeconds": 30
    }
  }
}
[root@localhost roottest]# vi /tmp/is.yaml
[root@localhost roottest]# oc create -f /tmp/is.yaml 
imagestream.image.openshift.io/imagestreamname created
[root@localhost roottest]# oc get is
NAME              IMAGE REPOSITORY                                                         TAGS   UPDATED
imagestreamname   image-registry.openshift-image-registry.svc:5000/zhouy/imagestreamname          
[root@localhost roottest]#  oc get imagestream.image.openshift.io/imagestreamname -o json | jq '.spec'
{
  "lookupPolicy": {
    "local": true
  }
}
[root@localhost roottest]# oc get rs
NAME                          DESIRED   CURRENT   READY   AGE
i-spawn-replicas-84778cc586   0         0         0       8m37s
[root@localhost roottest]# vi /tmp/patch.yaml
[root@localhost roottest]# oc patch deployment i-spawn-replicas --type merge --patch "$(cat /tmp/patch.yaml)"
deployment.apps/i-spawn-replicas patched
[root@localhost roottest]# oc get rs |wc -l
3
[root@localhost roottest]# oc get rs |wc -l
3
[root@localhost roottest]# oc get rs |wc -l
3
[root@localhost roottest]# oc get rs |wc -l
3

Comment 6 errata-xmlrpc 2021-08-10 11:27:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.4 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2983


Note You need to log in before you can comment on or make changes to this bug.