Bug 1888622 - The resources are deleted when clusterlogging in Unmanaged status after OCP upgrade
Summary: The resources are deleted when clusterlogging in Unmanaged status after OCP ...
Keywords:
Status: CLOSED DUPLICATE of bug 1883777
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.7.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-15 11:40 UTC by Anping Li
Modified: 2022-04-18 19:40 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-29 11:57:42 UTC
Target Upstream Version:
Embargoed:
jcantril: needinfo-


Attachments (Terms of Use)

Description Anping Li 2020-10-15 11:40:10 UTC
Description of problem:
When clusterlogging/instance in Unmanaged status. After OCP upgrade, some resources(deployment,cronjob secret, damonset) are deleted. The elasticsearch weren't deleted. But it couldn't be started as secret are deleted.


Version-Release number of selected component (if applicable):
ocp 4.4.26 -> 4.5.14
clusterserviceversion.operators.coreos.com/clusterlogging.4.4.0-202009161309.p0
clusterserviceversion.operators.coreos.com/elasticsearch-operator.4.4.0-202009161309.p0

Steps :
1. Deploy Clusterlogging v4.4 in OCP v4.4
2. Set clusterlogging/instance to Unmanaged
3. Upgrade OCP to 4.5
                          
 
Actual Result:
Resources(deployment,cronjob secret, damonset, configmap and etc) are deleted. Elasticsearch deployment weren't deleted, but it couldn't be as secret was delelted.

$ oc get secret
NAME                                       TYPE                                  DATA   AGE
builder-dockercfg-vfdnj                    kubernetes.io/dockercfg               1      10h
builder-token-hxzpt                        kubernetes.io/service-account-token   4      10h
builder-token-w8p6c                        kubernetes.io/service-account-token   4      10h
cluster-logging-operator-dockercfg-tlnkt   kubernetes.io/dockercfg               1      10h
cluster-logging-operator-token-f8zgd       kubernetes.io/service-account-token   4      10h
cluster-logging-operator-token-xjbpx       kubernetes.io/service-account-token   4      10h
default-dockercfg-ftzwz                    kubernetes.io/dockercfg               1      10h
default-token-2wm4g                        kubernetes.io/service-account-token   4      10h
default-token-7l9mm                        kubernetes.io/service-account-token   4      10h
deployer-dockercfg-q4c9v                   kubernetes.io/dockercfg               1      10h
deployer-token-68szw                       kubernetes.io/service-account-token   4      10h
deployer-token-sdkgw                       kubernetes.io/service-account-token   4      10h
elasticsearch-dockercfg-8qckp              kubernetes.io/dockercfg               1      61m
elasticsearch-metrics                      kubernetes.io/tls                     2      61m
elasticsearch-token-7j6gg                  kubernetes.io/service-account-token   4      61m
elasticsearch-token-ncztn                  kubernetes.io/service-account-token   4      61m
$ oc get cm
NAME                            DATA   AGE
cluster-logging-operator-lock   0      38m
elasticsearch                   3      61m

$ oc get deployment
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
cluster-logging-operator       1/1     1            1           10h
elasticsearch-cdm-yuroqdyb-1   0/1     1            0           61m
elasticsearch-cdm-yuroqdyb-2   0/1     1            0           61m

$ oc get cronjob
No resources found in openshift-logging namespace.


$ oc get roles
NAME                                              CREATED AT
clusterlogging.4.4.0-202009161309.p0-6464df8dfc   2020-10-15T10:00:13Z
[anli@preserve-docker-slave 117935]$ 

$ oc get pods
NAME                                           READY   STATUS              RESTARTS   AGE
cluster-logging-operator-7787b7b847-rfdb5      1/1     Running             0          36m
elasticsearch-cdm-yuroqdyb-1-f59dfc79d-52lcx   0/2     ContainerCreating   0          40m
elasticsearch-cdm-yuroqdyb-2-85c747b89-f8ph7   0/2     ContainerCreating   0          36m

Events:
  Type     Reason                  Age                   From                                                       Message
  ----     ------                  ----                  ----                                                       -------
  Warning  FailedScheduling        38m                   default-scheduler                                          0/5 nodes are available: 1 node(s) had volume node affinity conflict, 1 node(s) were unschedulable, 3 node(s) didn't match node selector.
  Warning  FailedScheduling        38m                   default-scheduler                                          0/5 nodes are available: 1 node(s) had volume node affinity conflict, 1 node(s) were unschedulable, 3 node(s) didn't match node selector.
  Warning  FailedScheduling        38m                   default-scheduler                                          0/5 nodes are available: 1 node(s) had volume node affinity conflict, 2 node(s) didn't match node selector, 2 node(s) were unschedulable.
  Normal   Scheduled               35m                   default-scheduler                                          Successfully assigned openshift-logging/elasticsearch-cdm-yuroqdyb-1-f59dfc79d-52lcx to qe45anli2-10140433-w-a-0.c.openshift-qe.internal
  Normal   SuccessfulAttachVolume  35m                   attachdetach-controller                                    AttachVolume.Attach succeeded for volume "pvc-5363b8df-0d05-438e-98a3-4b4fa0e1cb70"
  Warning  FailedMount             31m                   kubelet, qe45anli2-10140433-w-a-0.c.openshift-qe.internal  Unable to attach or mount volumes: unmounted volumes=[certificates], unattached volumes=[elasticsearch-config certificates elasticsearch-token-ncztn elasticsearch-metrics elasticsearch-storage]: timed out waiting for the condition
  Warning  FailedMount             17m (x4 over 24m)     kubelet, qe45anli2-10140433-w-a-0.c.openshift-qe.internal  Unable to attach or mount volumes: unmounted volumes=[certificates], unattached volumes=[elasticsearch-metrics elasticsearch-storage elasticsearch-config certificates elasticsearch-token-ncztn]: timed out waiting for the condition
  Warning  FailedMount             15m (x4 over 33m)     kubelet, qe45anli2-10140433-w-a-0.c.openshift-qe.internal  Unable to attach or mount volumes: unmounted volumes=[certificates], unattached volumes=[elasticsearch-storage elasticsearch-config certificates elasticsearch-token-ncztn elasticsearch-metrics]: timed out waiting for the condition
  Warning  FailedMount             13m                   kubelet, qe45anli2-10140433-w-a-0.c.openshift-qe.internal  Unable to attach or mount volumes: unmounted volumes=[certificates], unattached volumes=[elasticsearch-token-ncztn elasticsearch-metrics elasticsearch-storage elasticsearch-config certificates]: timed out waiting for the condition
  Warning  FailedMount             4m58s (x23 over 35m)  kubelet, qe45anli2-10140433-w-a-0.c.openshift-qe.internal  MountVolume.SetUp failed for volume "certificates" : secret "elasticsearch" not found


Expected Result:

No resource are deleted. The Logging still works well.

Comment 2 Jeff Cantrill 2020-10-20 20:21:57 UTC
Believe this to be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1883777 which is also is slated to be cherrypicked to 4.4

Comment 3 Jeff Cantrill 2020-10-23 15:19:59 UTC
Setting UpcomingSprint as unable to resolve before EOD

Comment 4 Jeff Cantrill 2020-10-26 18:29:46 UTC
Please confirm #c2 when you verify  https://bugzilla.redhat.com/show_bug.cgi?id=1883777

Comment 5 Anping Li 2020-10-29 11:54:05 UTC
Jeff, Yes, this should be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1883777. I upgrade about 6 clusters with clusterlogging/elasticsearch in Unmanagement status. With the fix, no resources are deleted. 

Shall we close this bug and clone https://bugzilla.redhat.com/show_bug.cgi?id=1883777 to 4.4?

Comment 6 Anping Li 2020-10-29 11:57:42 UTC

*** This bug has been marked as a duplicate of bug 1883777 ***


Note You need to log in before you can comment on or make changes to this bug.