Bug 1908991

Summary:	The profile parser pod deployment and associated profiles should get removed after upgrade the compliance operator
Product:	OpenShift Container Platform	Reporter:	xiyuan
Component:	Compliance Operator	Assignee:	Jakub Hrozek <jhrozek>
Status:	CLOSED ERRATA	QA Contact:	Prashant Dhamdhere <pdhamdhe>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.6.z	CC:	jhrozek, josorior, knewcome, mrogers, nkinder, pdhamdhe, xiyuan
Target Milestone:	---	Keywords:	UpcomingSprint
Target Release:	4.6.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1908990	Environment:
Last Closed:	2021-01-19 13:53:52 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1908990
Bug Blocks:

Description xiyuan 2020-12-18 06:12:13 UTC

+++ This bug was initially created as a clone of Bug #1908990 +++

Description of problem:
The profile parser pod deployment and associated profiles should get removed after upgrade the compliance operator to latest version

Version-Release number of selected component (if applicable):
4.6.9

How reproducible:
Always

Steps to Reproduce:
1. on stage cluster, install a compliance operator:
$oc create -f - <<EOF    
apiVersion: v1                          
kind: Namespace
metadata:
  name: up                        
  annotations:                    
    openshift.io/node-selector: ""
EOF                            
 
$ oc create -f - <<EOF    
apiVersion: operators.coreos.com/v1      
kind: OperatorGroup
metadata:
   name: openshift-compliance-abcd
   namespace: up                  
spec:                            
   targetNamespaces:                          
   - up
EOF
 
$ oc create -f - <<EOF    
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
   name: openshift-compliance-operator
   namespace: up                  
spec:                            
   channel: "4.6"                          
   Approval: Manual
   name: compliance-operator
   source: redhat-operators
   sourceNamespace: openshift-marketplace
EOF
 
$ oc get ip
NAME            CSV                           APPROVAL    APPROVED
install-xbqh7   compliance-operator.v0.1.17   Automatic   true
 
 
$ oc get csv -w
NAME                                           DISPLAY                  VERSION                 REPLACES   PHASE
compliance-operator.v0.1.17                    Compliance Operator      0.1.17                             Succeeded
elasticsearch-operator.4.6.0-202012050130.p0   Elasticsearch Operator   4.6.0-202012050130.p0              Succeeded

2. patch source in subscriptions which will have stage index image with latest version of operator:
$ oc patch subscriptions openshift-compliance-operator -p '{"spec":{"source":"qe-app-registry"}}' --type='merge'

3. check the ip and csv to makre sure operator upgrade finished:
$ oc get ip
NAME            CSV                           APPROVAL    APPROVED
install-cx59m   compliance-operator.v0.1.22   Automatic   true
install-xbqh7   compliance-operator.v0.1.17   Automatic   true
$ oc get csv
NAME                                           DISPLAY                  VERSION                 REPLACES                      PHASE
compliance-operator.v0.1.22                    Compliance Operator      0.1.22                  compliance-operator.v0.1.17   Succeeded
elasticsearch-operator.4.6.0-202012050130.p0   Elasticsearch Operator   4.6.0-202012050130.p0                                 Succeeded

Actual results:
The profile parser pod deployment and associated profiles does not get removed after upgrade the compliance operator to latest version

############Before upgrade, there are deployments for profiles:
oc get deployment
NAME                 READY  UP-TO-DATE  AVAILABLE  AGE
compliance-operator  1/1    1           1          5m
ocp4-pp              1/1    1           1          4m
rhcos4-pp            1/1    1           1          4m


############After upgrade, there are two new deployments for profile parser:
$ oc get deployment
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
compliance-operator   1/1     1            1           15m
ocp4-pp               1/1     1            1           14m
ocp4-up-pp            1/1     1            1           3m43s
rhcos4-pp             1/1     1            1           14m
rhcos4-up-pp          1/1     1            1           3m43s

$ oc get pod
NAME                                   READY   STATUS    RESTARTS   AGE
compliance-operator-84c495768b-zsgqb   1/1     Running   0          4m36s
ocp4-pp-5bbc8cb8db-4xdkg               1/1     Running   0          14m
ocp4-up-pp-bb856cb95-8n6sd             1/1     Running   0          3m37s
rhcos4-pp-6f5fdfbdb-wqwxv              1/1     Running   0          14m
rhcos4-up-pp-f84f7559-m7vfh            1/1     Running   0          3m37s

$ oc get profile.compliance
NAME              AGE
ocp4-cis          18m
ocp4-cis-node     18m
ocp4-e8           18m
ocp4-moderate     18m
ocp4-ncp          18m
rhcos4-e8         17m
rhcos4-moderate   17m
rhcos4-ncp        17m

$ oc delete deployment ocp4-pp rhcos4-pp
deployment.apps "ocp4-pp" deleted
deployment.apps "rhcos4-pp" deleted
$ oc get pod
NAME                                   READY   STATUS    RESTARTS   AGE
compliance-operator-84c495768b-zsgqb   1/1     Running   0          9m31s
ocp4-up-pp-bb856cb95-8n6sd             1/1     Running   0          8m32s
rhcos4-up-pp-f84f7559-m7vfh            1/1     Running   0          8m32s

Expected results
The profile parser pod deployment and associated profiles should get removed after upgrade the compliance operator to latest version


Additional info
Delete  deployment manually doesn't work. The ocp profiles still exist. 
$ oc get profile.compliance
NAME              AGE
ocp4-cis          18m
ocp4-cis-node     18m
ocp4-e8           18m
ocp4-moderate     18m
ocp4-ncp          18m
rhcos4-e8         18m
rhcos4-moderate   18m
rhcos4-ncp        18m

Comment 1 Juan Antonio Osorio 2020-12-18 08:19:59 UTC

Removing objects when upgrading the operator is not expected behavior, they merely get picked up when an upgrade happens.

For scans this is not an issue, as any retriggering of a scan rebuilds all the objects. For the profileparsers, the existence of those deployments indicates that the profiles don't need to be rebuilt as they're pinned to a specific content image.

Updating content alongside the operator is being done in a separate PR: https://github.com/openshift/compliance-operator/pull/532

Comment 6 Prashant Dhamdhere 2021-01-08 11:43:07 UTC

This looks good to me. The compliance operator version is upgraded successfully and it updates to default profilebundles 
on the operator startup. Also, the old compliance operator version profile parser pod deployments and associated profiles 
like ncp get removed after the upgrade.

Verified on:
OCP 4.6.0-0.nightly-2021-01-05-203053
Compliance Operator v0.1.24


$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-01-05-203053   True        False         4h3m    Cluster version is 4.6.0-0.nightly-2021-01-05-203053

$ oc create -f - <<EOF  
> apiVersion: v1
> kind: Namespace
> metadata:
>   name: openshift-compliance
>   annotations:
>     openshift.io/node-selector: ""
> EOF
namespace/openshift-compliance created

$ oc create -f - <<EOF    
> apiVersion: operators.coreos.com/v1
> kind: OperatorGroup
> metadata:
>    name: openshift-compliance-abcd
>    namespace: openshift-compliance
> spec:
>    targetNamespaces:
>    - openshift-compliance
> EOF
operatorgroup.operators.coreos.com/openshift-compliance-abcd created

$ oc create -f - <<EOF    
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
   name: openshift-compliance-operator
   namespace: openshift-compliance
spec:
   channel: "4.6"
   Approval: Automatic
   name: compliance-operator
   source: redhat-operators
   sourceNamespace: openshift-marketplace
EOF
subscription.operators.coreos.com/openshift-compliance-operator created

$ oc project openshift-compliance
Already on project "openshift-compliance" on server "https://api.pdhamdhe0846.qe.devcluster.openshift.com:6443".

$ oc get sub
NAME                            PACKAGE               SOURCE             CHANNEL
openshift-compliance-operator   compliance-operator   redhat-operators   4.6

$ oc get ip
NAME            CSV                           APPROVAL    APPROVED
install-vdjx9   compliance-operator.v0.1.17   Automatic   true

$ oc get csv
NAME                          DISPLAY               VERSION   REPLACES   PHASE
compliance-operator.v0.1.17   Compliance Operator   0.1.17               Succeeded

$ oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
compliance-operator-5f9c645948-qjdr4   1/1     Running   0          3m13s
ocp4-pp-6f5676676c-nrpc2               1/1     Running   0          2m29s
rhcos4-pp-6db9dcd547-wp6ml             1/1     Running   0          2m29s

$ oc patch subscriptions openshift-compliance-operator -p '{"spec":{"source":"qe-app-registry"}}' --type='merge'
subscription.operators.coreos.com/openshift-compliance-operator patched

$ oc get sub
NAME                            PACKAGE               SOURCE            CHANNEL
openshift-compliance-operator   compliance-operator   qe-app-registry   4.6

$ oc get ip
NAME            CSV                           APPROVAL    APPROVED
install-kz772   compliance-operator.v0.1.24   Automatic   true
install-vdjx9   compliance-operator.v0.1.17   Automatic   true


$ oc get csv -w
NAME                          DISPLAY               VERSION   REPLACES                      PHASE
compliance-operator.v0.1.17   Compliance Operator   0.1.17                                  Replacing
compliance-operator.v0.1.24   Compliance Operator   0.1.24    compliance-operator.v0.1.17   Installing
compliance-operator.v0.1.24   Compliance Operator   0.1.24    compliance-operator.v0.1.17   Succeeded
compliance-operator.v0.1.17   Compliance Operator   0.1.17                                  Deleting
compliance-operator.v0.1.17   Compliance Operator   0.1.17                                  Deleting


$ oc get csv
NAME                          DISPLAY               VERSION   REPLACES                      PHASE
compliance-operator.v0.1.24   Compliance Operator   0.1.24    compliance-operator.v0.1.17   Succeeded


$ oc get pods -w
NAME                                   READY   STATUS    RESTARTS   AGE
compliance-operator-67c6f76f54-qtpxj   1/1     Running   0          44s
ocp4-pp-6f5676676c-nrpc2               1/1     Running   0          4m39s
rhcos4-pp-6db9dcd547-wp6ml             1/1     Running   0          4m39s
rhcos4-pp-6db9dcd547-wp6ml             1/1     Terminating   0          4m47s
ocp4-pp-6f5676676c-nrpc2               1/1     Terminating   0          4m47s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   0/1     Pending       0          0s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   0/1     Pending       0          0s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   0/1     Init:0/2      0          0s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   0/1     Init:0/2      0          0s
ocp4-openshift-compliance-pp-5bc4b87f99-9cstd     0/1     Pending       0          0s
ocp4-openshift-compliance-pp-5bc4b87f99-9cstd     0/1     Pending       0          0s
ocp4-openshift-compliance-pp-5bc4b87f99-9cstd     0/1     Init:0/2      0          0s
ocp4-openshift-compliance-pp-5bc4b87f99-9cstd     0/1     Init:0/2      0          0s
rhcos4-pp-6db9dcd547-wp6ml                        0/1     Terminating   0          4m48s
ocp4-pp-6f5676676c-nrpc2                          0/1     Terminating   0          4m48s
ocp4-pp-6f5676676c-nrpc2                          0/1     Terminating   0          4m49s
ocp4-pp-6f5676676c-nrpc2                          0/1     Terminating   0          4m49s
ocp4-openshift-compliance-pp-5bc4b87f99-9cstd     0/1     Init:0/2      0          2s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   0/1     Init:0/2      0          2s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   0/1     Init:1/2      0          7s

$ oc get pods
NAME                                              READY   STATUS     RESTARTS   AGE
compliance-operator-67c6f76f54-qtpxj              1/1     Running    0          108s
ocp4-openshift-compliance-pp-5bc4b87f99-9cstd     1/1     Running    0          56s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   0/1     Init:1/2   0          56s

$ oc get pods
NAME                                              READY   STATUS    RESTARTS   AGE
compliance-operator-67c6f76f54-qtpxj              1/1     Running   0          2m3s
ocp4-openshift-compliance-pp-5bc4b87f99-9cstd     1/1     Running   0          71s
rhcos4-openshift-compliance-pp-78d9c5d499-gjtrg   1/1     Running   0          71s

$ oc get deployment
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
compliance-operator              1/1     1            1           7m29s
ocp4-openshift-compliance-pp     1/1     1            1           118s
rhcos4-openshift-compliance-pp   1/1     1            1           118s

$ oc get profile.compliance
NAME              AGE
ocp4-cis          5m37s
ocp4-cis-node     5m37s
ocp4-e8           5m37s
ocp4-moderate     5m37s
rhcos4-e8         5m33s
rhcos4-moderate   5m32s

Comment 8 errata-xmlrpc 2021-01-19 13:53:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6 compliance-operator security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0190